openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-23 14:59:50 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
483d41c545
third_party_littlefs/lfs.c

5093 lines
144 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Updated copyright Due to employee contract Per ARM license remains under Apache 2.0
2017-10-13 01:27:33 +00:00
* Copyright (c) 2017 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
static int lfs_cache_read(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (pcache && block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
if (block == rcache->block && off >= rcache->off &&
off < rcache->off + lfs->cfg->read_size) {
// is already in rcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->read_size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->read_size == 0 && size >= lfs->cfg->read_size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
// bypass cache?
lfs_size_t diff = size - (size % lfs->cfg->read_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->read(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
data += diff;
off += diff;
size -= diff;
continue;
}
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
rcache->off = off - (off % lfs->cfg->read_size);
int err = lfs->cfg->read(lfs->cfg, rcache->block,
rcache->off, rcache->buffer, lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_cmp(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
if (c != data[i]) {
return false;
}
}
return true;
}
static int lfs_cache_crc(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
lfs_crc(crc, &c, 1);
}
return 0;
}
static int lfs_cache_flush(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache) {
if (pcache->block != 0xffffffff) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
pcache->block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_prog(lfs_t *lfs, lfs_cache_t *pcache,
lfs_cache_t *rcache, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->prog_size == 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
int err = lfs_cache_flush(lfs, pcache, rcache);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(pcache->block == 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (off % lfs->cfg->prog_size == 0 &&
size >= lfs->cfg->prog_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// bypass pcache?
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = size - (size % lfs->cfg->prog_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->prog(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL,
block, off, data, diff);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
pcache->off = off - (off % lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// General lfs block device operations ///
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_read(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return lfs_cache_read(lfs, &lfs->rcache, &lfs->pcache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_prog(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
return lfs_cache_prog(lfs, &lfs->pcache, NULL,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_cmp(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
return lfs_cache_cmp(lfs, &lfs->rcache, NULL, block, off, buffer, size);
}
static int lfs_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
return lfs_cache_crc(lfs, &lfs->rcache, NULL, block, off, size, crc);
}
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->erase(lfs->cfg, block);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_sync(lfs_t *lfs) {
Fixed dirty rcache during directory commit An interesting side-effect of adding internal checks to the littlefs for block errors, is that the littlefs starts to cover up its own flaws. Probably out of embarrassment. In this case, the relocation logic for directories left the littlefs rcache dirty with invalid data. The littlefs detected the error, treated it as a corrupted write, and just moved the "corrupted" block to a new block, which as a side-effect flushes the rcache. Since committing a dir will end up flushing the rcache to check for errors anyways, we can just drop the rcache in lfs_bd_sync.
2017-06-25 21:21:14 +00:00
lfs->rcache.block = 0xffffffff;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_flush(lfs, &lfs->pcache, NULL);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->sync(lfs->cfg);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
/// Internal operations predeclared here ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
int (*cb)(lfs_t*, void*, lfs_block_t), void *data);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_dir_t *pdir);
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_dir_t *parent, lfs_entry_t *entry);
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
static int lfs_moved(lfs_t *lfs, lfs_dir_t *fromdir, uint16_t fromid);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]);
int lfs_deorphan(lfs_t *lfs);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
/// Block allocator ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static int lfs_alloc_lookahead(lfs_t *lfs, void *p, lfs_block_t block) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs_block_t off = ((block - lfs->free.off)
Fixed issue with negative modulo with unaligned lookaheads When the lookahead buffer wraps around in an unaligned filesystem, it's possible for blocks at the beginning of the disk to have a negative distance from the lookahead, but still reside in the lookahead buffer. Switching to signed modulo doesn't quite work due to how negative modulo is implemented in C, so the simple solution is to shift the region to be positive.
2017-09-17 19:52:25 +00:00
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
if (off < lfs->free.size) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer[off / 32] |= 1U << (off % 32);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
return 0;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
while (true) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// found a free block
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
*block = (lfs->free.off + off) % lfs->cfg->block_count;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
// check if we have looked at all blocks since last ack
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
if (lfs->free.ack == 0) {
LFS_WARN("No more free space %d", lfs->free.i + lfs->free.off);
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
return LFS_ERR_NOSPC;
}
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->free.ack);
lfs->free.i = 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// find mask of free blocks from tree
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_fs_traverse(lfs, lfs_alloc_lookahead, NULL);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.ack = lfs->cfg->block_count;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
/// Endian swapping functions ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static void lfs_dir_fromle32(struct lfs_disk_dir *d) {
// d->rev = lfs_fromle32(d->rev);
// d->size = lfs_fromle32(d->size);
// d->tail[0] = lfs_fromle32(d->tail[0]);
// d->tail[1] = lfs_fromle32(d->tail[1]);
//}
//
//static void lfs_dir_tole32(struct lfs_disk_dir *d) {
// d->rev = lfs_tole32(d->rev);
// d->size = lfs_tole32(d->size);
// d->tail[0] = lfs_tole32(d->tail[0]);
// d->tail[1] = lfs_tole32(d->tail[1]);
//}
//
//static void lfs_entry_fromle32(struct lfs_disk_entry *d) {
// d->u.dir[0] = lfs_fromle32(d->u.dir[0]);
// d->u.dir[1] = lfs_fromle32(d->u.dir[1]);
//}
//
//static void lfs_entry_tole32(struct lfs_disk_entry *d) {
// d->u.dir[0] = lfs_tole32(d->u.dir[0]);
// d->u.dir[1] = lfs_tole32(d->u.dir[1]);
//}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
///*static*/ void lfs_superblock_fromle32(struct lfs_disk_superblock *d) {
// d->root[0] = lfs_fromle32(d->root[0]);
// d->root[1] = lfs_fromle32(d->root[1]);
// d->block_size = lfs_fromle32(d->block_size);
// d->block_count = lfs_fromle32(d->block_count);
// d->version = lfs_fromle32(d->version);
// d->inline_size = lfs_fromle32(d->inline_size);
// d->attrs_size = lfs_fromle32(d->attrs_size);
// d->name_size = lfs_fromle32(d->name_size);
//}
//
///*static*/ void lfs_superblock_tole32(struct lfs_disk_superblock *d) {
// d->root[0] = lfs_tole32(d->root[0]);
// d->root[1] = lfs_tole32(d->root[1]);
// d->block_size = lfs_tole32(d->block_size);
// d->block_count = lfs_tole32(d->block_count);
// d->version = lfs_tole32(d->version);
// d->inline_size = lfs_tole32(d->inline_size);
// d->attrs_size = lfs_tole32(d->attrs_size);
// d->name_size = lfs_tole32(d->name_size);
//}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
/// Other struct functions ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static inline lfs_size_t lfs_entry_elen(const lfs_entry_t *entry) {
// return (lfs_size_t)(entry->d.elen) |
// ((lfs_size_t)(entry->d.alen & 0xc0) << 2);
//}
//
//static inline lfs_size_t lfs_entry_alen(const lfs_entry_t *entry) {
// return entry->d.alen & 0x3f;
//}
//
//static inline lfs_size_t lfs_entry_nlen(const lfs_entry_t *entry) {
// return entry->d.nlen;
//}
//
//static inline lfs_size_t lfs_entry_size(const lfs_entry_t *entry) {
// return 4 + lfs_entry_elen(entry) +
// lfs_entry_alen(entry) +
// lfs_entry_nlen(entry);
//}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Metadata pair and directory operations ///
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline void lfs_pairswap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
static inline bool lfs_pairisnull(const lfs_block_t pair[2]) {
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
return pair[0] == 0xffffffff || pair[1] == 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline int lfs_paircmp(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static inline bool lfs_pairsync(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
/// Entry tag operations ///
static inline lfs_tag_t lfs_mktag(
uint16_t type, uint16_t id, lfs_size_t size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return (type << 22) | (id << 12) | size;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline bool lfs_tag_valid(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return !(tag & 0x80000000);
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline uint16_t lfs_tag_type(lfs_tag_t tag) {
return (tag & 0x7fc00000) >> 22;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline uint8_t lfs_tag_supertype(lfs_tag_t tag) {
return (tag & 0x70000000) >> 22;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline uint8_t lfs_tag_subtype(lfs_tag_t tag) {
return (tag & 0x7c000000) >> 22;
}
static inline uint8_t lfs_tag_struct(lfs_tag_t tag) {
return (tag & 0x03c00000) >> 22;
}
static inline uint16_t lfs_tag_id(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return (tag & 0x001ff000) >> 12;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
static inline lfs_size_t lfs_tag_size(lfs_tag_t tag) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return tag & 0x00000fff;
}
struct lfs_commit {
lfs_block_t block;
lfs_off_t off;
lfs_off_t begin;
lfs_off_t end;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t ptag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t crc;
struct {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
uint16_t begin;
uint16_t end;
} filter;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO predelcare
static int lfs_commit_move_(lfs_t *lfs, struct lfs_commit *commit,
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
uint16_t fromid, uint16_t toid,
lfs_dir_t *dir, lfs_entrylist_t *list);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static int lfs_commit_compactcheck(lfs_t *lfs, void *p, lfs_entry_t entry) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// struct lfs_commit *commit = p;
// if (lfs_tag_id(entry.tag) != commit->compact.id) {
// return 1;
// } else if (lfs_tag_type(entry.tag) == commit->compact.type) {
// return 2;
// }
//
// return 0;
//}
//
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
static int lfs_commit_commit(lfs_t *lfs,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
struct lfs_commit *commit, lfs_entry_t entry) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// filter out ids
if (lfs_tag_id(entry.tag) != 0x1ff && (
lfs_tag_id(entry.tag) < commit->filter.begin ||
lfs_tag_id(entry.tag) >= commit->filter.end)) {
return 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// special cases
if ((lfs_tag_type(entry.tag) & 0x103) == LFS_FROM_MOVE) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
return lfs_commit_move_(lfs, commit,
lfs_tag_size(entry.tag), lfs_tag_id(entry.tag),
entry.u.dir, NULL);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
uint16_t id = lfs_tag_id(entry.tag) - commit->filter.begin;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
entry.tag = lfs_mktag(0, id, 0) | (entry.tag & 0xffe00fff);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// check if we fit
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_size_t size = lfs_tag_size(entry.tag);
if (commit->off + sizeof(lfs_tag_t)+size > commit->end) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOSPC;
}
// write out tag
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO rm me
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
//printf("tag w %#010x (%x:%x %03x %03x %03x)\n", entry.tag, commit->block, commit->off+sizeof(lfs_tag_t), lfs_tag_type(entry.tag), lfs_tag_id(entry.tag), lfs_tag_size(entry.tag));
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t tag = lfs_tole32((entry.tag & 0x7fffffff) ^ commit->ptag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
lfs_crc(&commit->crc, &tag, sizeof(tag));
int err = lfs_bd_prog(lfs, commit->block, commit->off, &tag, sizeof(tag));
if (err) {
return err;
}
commit->off += sizeof(tag);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (!(entry.tag & 0x80000000)) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// from memory
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_crc(&commit->crc, entry.u.buffer, size);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
err = lfs_bd_prog(lfs, commit->block, commit->off,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
entry.u.buffer, size);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
} else {
// from disk
for (lfs_off_t i = 0; i < size; i++) {
uint8_t dat;
int err = lfs_bd_read(lfs,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
entry.u.d.block, entry.u.d.off+i, &dat, 1);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
lfs_crc(&commit->crc, &dat, 1);
err = lfs_bd_prog(lfs, commit->block, commit->off+i, &dat, 1);
if (err) {
return err;
}
}
}
commit->off += size;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
commit->ptag = entry.tag & 0x7fffffff; // TODO do this once
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return 0;
}
static int lfs_commit_crc(lfs_t *lfs, struct lfs_commit *commit) {
// align to program units
lfs_off_t noff = lfs_alignup(
commit->off + 2*sizeof(uint32_t), lfs->cfg->prog_size);
// read erased state from next program unit
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t tag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
int err = lfs_bd_read(lfs, commit->block, noff, &tag, sizeof(tag));
if (err) {
return err;
}
// build crc tag
tag = (0x80000000 & ~lfs_fromle32(tag)) |
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_mktag(LFS_TYPE_CRC, 0x1ff,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
noff - (commit->off+sizeof(uint32_t)));
// write out crc
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
//printf("tag w %#010x (%x:%x %03x %03x %03x)\n", tag, commit->block, commit->off+sizeof(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t footer[2];
footer[0] = lfs_tole32(tag ^ commit->ptag);
lfs_crc(&commit->crc, &footer[0], sizeof(footer[0]));
footer[1] = lfs_tole32(commit->crc);
err = lfs_bd_prog(lfs, commit->block, commit->off,
footer, sizeof(footer));
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
commit->off += sizeof(tag)+lfs_tag_size(tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
commit->ptag = tag;
// flush buffers
err = lfs_bd_sync(lfs);
if (err) {
return err;
}
// successful commit, check checksum to make sure
uint32_t crc = 0xffffffff;
err = lfs_bd_crc(lfs, commit->block, commit->begin,
commit->off-lfs_tag_size(tag) - commit->begin, &crc);
if (err) {
return err;
}
if (crc != commit->crc) {
return LFS_ERR_CORRUPT;
}
return 0;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// committer for regions
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_commit_regions(lfs_t *lfs, void *p, struct lfs_commit *commit) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
for (lfs_entrylist_t *regions = p; regions; regions = regions->next) {
int err = lfs_commit_commit(lfs, commit, regions->e);
if (err) {
return err;
}
}
return 0;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
static int lfs_commit_list(lfs_t *lfs, struct lfs_commit *commit,
lfs_entrylist_t *list) {
for (; list; list = list->next) {
int err = lfs_commit_commit(lfs, commit, list->e);
if (err) {
return err;
}
}
return 0;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// committer for moves
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO rename?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
struct lfs_commit_move {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
lfs_dir_t *dir; // TODO need dir?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
struct {
uint16_t from;
uint16_t to;
} id;
struct lfs_commit *commit;
};
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// TODO redeclare
static int lfs_dir_traverse(lfs_t *lfs, lfs_dir_t *dir,
int (*cb)(lfs_t *lfs, void *data, lfs_entry_t entry),
void *data);
static int lfs_dir_get(lfs_t *lfs, lfs_dir_t *dir,
uint32_t mask, lfs_entry_t *entry);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
static int lfs_commit_movescan(lfs_t *lfs, void *p, lfs_entry_t entry) {
struct lfs_commit_move *move = p;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (lfs_tag_type(entry.tag) == LFS_TYPE_DELETE &&
lfs_tag_id(entry.tag) <= move->id.from) {
// something was deleted, we need to move around it
move->id.from += 1;
return 0;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (lfs_tag_type(entry.tag) == LFS_TYPE_MOVE) {
// TODO need this?
// ignore moves
return 0;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
if (lfs_tag_id(entry.tag) != move->id.from) {
// ignore non-matching ids
return 0;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// check if type has already been committed
int err = lfs_dir_get(lfs,
&(lfs_dir_t){
.pair[0]=move->commit->block,
.off=move->commit->off,
.etag=move->commit->ptag,
.stop_at_commit=true},
lfs_tag_type(entry.tag) & 0x100 ? 0x7ffff000 : 0x7c1ff000,
&(lfs_entry_t){
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
lfs_mktag(lfs_tag_type(entry.tag),
move->id.to - move->commit->filter.begin, 0)}); // TODO can all these filter adjustments be consolidated?
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err != LFS_ERR_NOENT) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// already committed
return 0;
}
// update id and commit, as we are currently unique
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
entry.tag = lfs_mktag(0, move->id.to, 0) | (entry.tag & 0xffe00fff);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
return lfs_commit_commit(lfs, move->commit, entry);
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// TODO change this to be special RAM-side type in linked list of commits?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
static int lfs_commit_move(lfs_t *lfs, void *p, struct lfs_commit *commit) {
struct lfs_commit_move *move = p;
move->commit = commit;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (move->id.to < commit->filter.begin ||
move->id.to >= commit->filter.end) {
// skip if not in filter
return 0;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int err = lfs_dir_traverse(lfs, move->dir, lfs_commit_movescan, move);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err < 0) {
return err;
}
return 0;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
static int lfs_commit_move_(lfs_t *lfs, struct lfs_commit *commit,
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
uint16_t fromid, uint16_t toid,
lfs_dir_t *dir, lfs_entrylist_t *list) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
struct lfs_commit_move move = {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
.id.from = fromid,
.id.to = toid,
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
.commit = commit,
};
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
for (; list; list = list->next) {
int err = lfs_commit_movescan(lfs, &move, list->e);
if (err) {
return err;
}
}
int err = lfs_dir_traverse(lfs, dir, lfs_commit_movescan, &move);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
return err;
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_dir_t *dir,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
bool split, const lfs_block_t tail[2]) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// allocate pair of dir blocks (backwards, so we write to block 1 first)
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[(i+1)%2]);
if (err) {
return err;
}
}
// rather than clobbering one of the blocks we just pretend
// the revision may be valid
int err = lfs_bd_read(lfs, dir->pair[0], 0, &dir->rev, 4);
dir->rev = lfs_fromle32(dir->rev);
if (err) {
return err;
}
// set defaults
dir->off = sizeof(dir->rev);
dir->etag = 0;
dir->count = 0;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
dir->tail[0] = tail[0];
dir->tail[1] = tail[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->erased = false;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->split = split;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// don't write out yet, let caller take care of that
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_fetchwith(lfs_t *lfs,
lfs_dir_t *dir, const lfs_block_t pair[2],
int (*cb)(lfs_t *lfs, void *data, lfs_entry_t entry), void *data) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->pair[0] = pair[0];
dir->pair[1] = pair[1];
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
dir->stop_at_commit = false;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// find the block with the most recent revision
uint32_t rev[2];
for (int i = 0; i < 2; i++) {
int err = lfs_bd_read(lfs, dir->pair[i], 0, &rev[i], sizeof(rev[i]));
rev[i] = lfs_fromle32(rev[i]);
if (err) {
return err;
}
}
if (lfs_scmp(rev[1], rev[0]) > 0) {
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
// load blocks and check crc
for (int i = 0; i < 2; i++) {
lfs_off_t off = sizeof(dir->rev);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t ptag = 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
uint32_t crc = 0xffffffff;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->tail[0] = 0xffffffff;
dir->tail[1] = 0xffffffff;
dir->count = 0;
dir->split = false;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->moveid = -1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->rev = lfs_tole32(rev[0]);
lfs_crc(&crc, &dir->rev, sizeof(dir->rev));
dir->rev = lfs_fromle32(dir->rev);
while (true) {
// extract next tag
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_t tag;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
int err = lfs_bd_read(lfs, dir->pair[0], off, &tag, sizeof(tag));
if (err) {
return err;
}
lfs_crc(&crc, &tag, sizeof(tag));
tag = lfs_fromle32(tag) ^ ptag;
// next commit not yet programmed
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(ptag) == LFS_TYPE_CRC && !lfs_tag_valid(tag)) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->erased = true;
return 0;
}
// check we're in valid range
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (off + sizeof(tag)+lfs_tag_size(tag) > lfs->cfg->block_size) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
break;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
//printf("tag r %#010x (%x:%x %03x %03x %03x)\n", tag, dir->pair[0], off+sizeof(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(tag) == LFS_TYPE_CRC) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// check the crc entry
uint32_t dcrc;
int err = lfs_bd_read(lfs, dir->pair[0],
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off+sizeof(tag), &dcrc, sizeof(dcrc));
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
if (crc != lfs_fromle32(dcrc)) {
if (off == sizeof(dir->rev)) {
// try other block
break;
} else {
// consider what we have good enough
dir->erased = false;
return 0;
}
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->off = off + sizeof(tag)+lfs_tag_size(tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->etag = tag;
crc = 0xffffffff;
} else {
err = lfs_bd_crc(lfs, dir->pair[0],
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off+sizeof(tag), lfs_tag_size(tag), &crc);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(tag) == LFS_TYPE_SOFTTAIL ||
lfs_tag_type(tag) == LFS_TYPE_HARDTAIL) {
dir->split = lfs_tag_type(tag) == LFS_TYPE_HARDTAIL;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
err = lfs_bd_read(lfs, dir->pair[0], off+sizeof(tag),
dir->tail, sizeof(dir->tail));
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else if (lfs_tag_type(tag) == LFS_TYPE_MOVE) {
// TODO handle moves correctly?
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->moveid = lfs_tag_id(tag);
} else {
if (lfs_tag_id(tag) < 0x1ff &&
lfs_tag_id(tag) >= dir->count) {
dir->count = lfs_tag_id(tag)+1;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(tag) == LFS_TYPE_DELETE) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->count -= 1;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (dir->moveid != -1) {
//printf("RENAME DEL %d (%d)\n", lfs_tag_id(tag), dir->moveid);
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (lfs_tag_id(tag) == dir->moveid) {
dir->moveid = -1;
} else if (lfs_tag_id(tag) < dir->moveid) {
dir->moveid -= 1;
}
}
if (cb) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = cb(lfs, data, (lfs_entry_t){
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
(tag | 0x80000000),
.u.d.block=dir->pair[0],
.u.d.off=off+sizeof(tag)});
if (err) {
return err;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
}
}
ptag = tag;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
off += sizeof(tag)+lfs_tag_size(tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
// failed, try the other crc?
lfs_pairswap(dir->pair);
lfs_pairswap(rev);
}
LFS_ERROR("Corrupted dir pair at %d %d", dir->pair[0], dir->pair[1]);
return LFS_ERR_CORRUPT;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_fetch(lfs_t *lfs,
lfs_dir_t *dir, const lfs_block_t pair[2]) {
return lfs_dir_fetchwith(lfs, dir, pair, NULL, NULL);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_traverse(lfs_t *lfs, lfs_dir_t *dir,
int (*cb)(lfs_t *lfs, void *data, lfs_entry_t entry), void *data) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// iterate over dir block backwards (for faster lookups)
lfs_block_t block = dir->pair[0];
lfs_off_t off = dir->off;
lfs_tag_t tag = dir->etag;
while (off != sizeof(uint32_t)) {
// TODO rm me
//printf("tag r %#010x (%x:%x %03x %03x %03x)\n", tag, block, off-lfs_tag_size(tag), lfs_tag_type(tag), lfs_tag_id(tag), lfs_tag_size(tag));
// TODO hmm
if (dir->stop_at_commit && lfs_tag_type(tag) == LFS_TYPE_CRC) {
break;
}
int err = cb(lfs, data, (lfs_entry_t){
(0x80000000 | tag),
.u.d.block=block,
.u.d.off=off-lfs_tag_size(tag)});
if (err) {
return err;
}
LFS_ASSERT(off > sizeof(tag)+lfs_tag_size(tag));
off -= sizeof(tag)+lfs_tag_size(tag);
lfs_tag_t ntag;
err = lfs_bd_read(lfs, block, off, &ntag, sizeof(ntag));
if (err) {
return err;
}
tag ^= lfs_fromle32(ntag);
}
return 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//struct lfs_dir_commitmove {
// // traversal things
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// lfs_dir_t *dir;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int (*cb)(lfs_t *lfs, void *data, struct lfs_commit *commit);
// void *data;
//
// // ids to iterate through
// uint16_t begin;
// uint16_t end;
// uint16_t ack;
//};
//
//static int lfs_dir_commitmove_commit(lfs_t *lfs, void *p,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// lfs_entry_t entry) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// return lfs_commit_commit(lfs, p, entry);
//}
//
//int lfs_dir_commitmove(lfs_t *lfs, void *p, struct lfs_commit *commit) {
// struct lfs_dir_commitmove *move = p;
// for (int i = move->begin; i < move->end; i++) {
// // tell the committer to check for duplicates
// uint16_t old = commit->compact.id;
// if (commit->compact.id < 0) {
// commit->compact.id = i;
// }
//
// // commit pending commits
// int err = move->cb(lfs, move->data, commit);
// if (err) {
// commit->compact.id = old;
// return err;
// }
//
// // iterate over on-disk regions
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// err = lfs_dir_traverse(lfs, move->dir,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// lfs_dir_commitmove_commit, commit);
// if (err) {
// commit->compact.id = old;
// return err;
// }
//
// move->ack = i;
// commit->compact.id = old;
// }
//
// return 0;
//}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_compact(lfs_t *lfs, lfs_dir_t *dir,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
int (*cb)(lfs_t *lfs, void *data, struct lfs_commit *commit),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
void *data, lfs_dir_t *source, uint16_t begin, uint16_t end) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// save some state in case block is bad
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
// increment revision count
dir->rev += 1;
while (true) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// last complete id
int16_t ack = -1;
dir->count = end - begin;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (true) {
// erase block to write to
int err = lfs_bd_erase(lfs, dir->pair[1]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// write out header
uint32_t crc = 0xffffffff;
uint32_t rev = lfs_tole32(dir->rev);
lfs_crc(&crc, &rev, sizeof(rev));
err = lfs_bd_prog(lfs, dir->pair[1], 0, &rev, sizeof(rev));
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// setup compaction
struct lfs_commit commit = {
.block = dir->pair[1],
.off = sizeof(dir->rev),
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// space is complicated, we need room for tail, crc,
// and we keep cap at around half a block
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
.begin = 0,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
.end = lfs_min(
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
lfs_alignup(lfs->cfg->block_size / 2,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
lfs->cfg->prog_size),
lfs->cfg->block_size - 5*sizeof(uint32_t)),
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
.crc = crc,
.ptag = 0,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// filter out ids
.filter.begin = begin,
.filter.end = end,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
};
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// commit regions which can't fend for themselves
err = cb(lfs, data, &commit);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// move over other commits, leaving it up to lfs_commit_move to
// filter out duplicates, and the commit filtering to reassign ids
for (uint16_t id = begin; id < end; id++) {
err = lfs_commit_move(lfs,
&(struct lfs_commit_move){.dir=source, .id={id, id}},
&commit);
if (err) {
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
ack = id;
}
commit.end = lfs->cfg->block_size - 2*sizeof(uint32_t);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (!lfs_pairisnull(dir->tail)) {
// TODO le32
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_commit_commit(lfs, &commit, (lfs_entry_t){
lfs_mktag(LFS_TYPE_SOFTTAIL + dir->split*0x10,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
0x1ff, sizeof(dir->tail)),
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
.u.buffer=dir->tail});
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// successful compaction, swap dir pair to indicate most recent
lfs_pairswap(dir->pair);
dir->off = commit.off;
dir->etag = commit.ptag;
dir->erased = true;
}
break;
split:
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// commit no longer fits, need to split dir,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
// drop caches and create tail
lfs->pcache.block = 0xffffffff;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t tail;
int err = lfs_dir_alloc(lfs, &tail, dir->split, dir->tail);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_compact(lfs, &tail, cb, data, dir, ack+1, end);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
if (err) {
return err;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
end = ack+1;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->split = true;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
continue;
relocate:
//commit was corrupted
LFS_DEBUG("Bad block at %d", dir->pair[1]);
// drop caches and prepare to relocate block
relocated = true;
lfs->pcache.block = 0xffffffff;
// can't relocate superblock, filesystem is now frozen
if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
LFS_WARN("Superblock %d has become unwritable", oldpair[1]);
return LFS_ERR_CORRUPT;
}
// relocate half of pair
err = lfs_alloc(lfs, &dir->pair[1]);
if (err) {
return err;
}
continue;
}
if (relocated) {
// update references if we relocated
LFS_DEBUG("Relocating %d %d to %d %d",
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
int err = lfs_relocate(lfs, oldpair, dir->pair);
if (err) {
return err;
}
}
// shift over any directories that are affected
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
if (lfs_paircmp(d->pair, dir->pair) == 0) {
d->pair[0] = dir->pair[0];
d->pair[1] = dir->pair[1];
}
}
return 0;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
static int lfs_dir_compact_(lfs_t *lfs, lfs_dir_t *dir, lfs_entrylist_t *list,
lfs_dir_t *source, uint16_t begin, uint16_t end) {
// save some state in case block is bad
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
lfs_dir_t tail;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// increment revision count
dir->rev += 1;
while (true) {
// last complete id
int16_t ack = -1;
dir->count = end - begin;
if (true) {
// erase block to write to
int err = lfs_bd_erase(lfs, dir->pair[1]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// write out header
uint32_t crc = 0xffffffff;
uint32_t rev = lfs_tole32(dir->rev);
lfs_crc(&crc, &rev, sizeof(rev));
err = lfs_bd_prog(lfs, dir->pair[1], 0, &rev, sizeof(rev));
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// setup compaction
struct lfs_commit commit = {
.block = dir->pair[1],
.off = sizeof(dir->rev),
// space is complicated, we need room for tail, crc,
// and we keep cap at around half a block
.begin = 0,
.end = lfs_min(
lfs_alignup(lfs->cfg->block_size / 2,
lfs->cfg->prog_size),
lfs->cfg->block_size - 5*sizeof(uint32_t)),
.crc = crc,
.ptag = 0,
// filter out ids
.filter.begin = begin,
.filter.end = end,
};
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// // commit regions which can't fend for themselves
// err = lfs_commit_list(lfs, &commit, list);
// if (err) {
// if (err == LFS_ERR_NOSPC) {
// goto split;
// } else if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
// return err;
// }
//
// // move over other commits, leaving it up to lfs_commit_move to
// // filter out duplicates, and the commit filtering to reassign ids
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
for (uint16_t id = begin; id < end; id++) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
err = lfs_commit_move_(lfs, &commit, id, id, source, list);
// err = lfs_commit_commit(lfs, &commit, (lfs_entry_t){
// lfs_mktag(LFS_FROM_MOVE, id, id), .u.dir=source});
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
if (err) {
if (err == LFS_ERR_NOSPC) {
goto split;
} else if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
ack = id;
}
commit.end = lfs->cfg->block_size - 2*sizeof(uint32_t);
if (!lfs_pairisnull(dir->tail)) {
// TODO le32
err = lfs_commit_commit(lfs, &commit, (lfs_entry_t){
lfs_mktag(LFS_TYPE_SOFTTAIL + dir->split*0x10,
0x1ff, sizeof(dir->tail)),
.u.buffer=dir->tail});
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// successful compaction, swap dir pair to indicate most recent
lfs_pairswap(dir->pair);
dir->off = commit.off;
dir->etag = commit.ptag;
dir->erased = true;
}
break;
split:
// commit no longer fits, need to split dir,
// drop caches and create tail
lfs->pcache.block = 0xffffffff;
lfs_dir_t tail;
int err = lfs_dir_alloc(lfs, &tail, dir->split, dir->tail);
if (err) {
return err;
}
err = lfs_dir_compact_(lfs, &tail, list, dir, ack+1, end);
if (err) {
return err;
}
end = ack+1;
dir->tail[0] = tail.pair[0];
dir->tail[1] = tail.pair[1];
dir->split = true;
continue;
relocate:
//commit was corrupted
LFS_DEBUG("Bad block at %d", dir->pair[1]);
// drop caches and prepare to relocate block
relocated = true;
lfs->pcache.block = 0xffffffff;
// can't relocate superblock, filesystem is now frozen
if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
LFS_WARN("Superblock %d has become unwritable", oldpair[1]);
return LFS_ERR_CORRUPT;
}
// relocate half of pair
err = lfs_alloc(lfs, &dir->pair[1]);
if (err) {
return err;
}
continue;
}
if (relocated) {
// update references if we relocated
LFS_DEBUG("Relocating %d %d to %d %d",
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
int err = lfs_relocate(lfs, oldpair, dir->pair);
if (err) {
return err;
}
}
// shift over any directories that are affected
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
if (lfs_paircmp(d->pair, dir->pair) == 0) {
d->pair[0] = dir->pair[0];
d->pair[1] = dir->pair[1];
}
}
return 0;
}
static int lfs_dir_commit_(lfs_t *lfs, lfs_dir_t *dir, lfs_entrylist_t *list) {
if (!dir->erased) {
// not erased, must compact
return lfs_dir_compact_(lfs, dir, list, dir, 0, dir->count);
}
struct lfs_commit commit = {
.block = dir->pair[0],
.begin = dir->off,
.off = dir->off,
.end = lfs->cfg->block_size - 2*sizeof(uint32_t),
.crc = 0xffffffff,
.ptag = dir->etag,
.filter.begin = 0,
.filter.end = 0x1ff,
};
int err = lfs_commit_list(lfs, &commit, list);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
lfs->pcache.block = 0xffffffff;
return lfs_dir_compact_(lfs, dir, list, dir, 0, dir->count);
}
return err;
}
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
lfs->pcache.block = 0xffffffff;
return lfs_dir_compact_(lfs, dir, list, dir, 0, dir->count);
}
return err;
}
// successful commit, lets update dir
dir->off = commit.off;
dir->etag = commit.ptag;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_commitwith(lfs_t *lfs, lfs_dir_t *dir,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
int (*cb)(lfs_t *lfs, void *data, struct lfs_commit *commit),
void *data) {
if (!dir->erased) {
// not erased, must compact
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_compact(lfs, dir, cb, data, dir, 0, dir->count);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
struct lfs_commit commit = {
.block = dir->pair[0],
.begin = dir->off,
.off = dir->off,
.end = lfs->cfg->block_size - 2*sizeof(uint32_t),
.crc = 0xffffffff,
.ptag = dir->etag,
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
.filter.begin = 0,
.filter.end = 0x1ff,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
};
int err = cb(lfs, data, &commit);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs->pcache.block = 0xffffffff;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_compact(lfs, dir, cb, data, dir, 0, dir->count);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
return err;
}
err = lfs_commit_crc(lfs, &commit);
if (err) {
if (err == LFS_ERR_NOSPC || err == LFS_ERR_CORRUPT) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs->pcache.block = 0xffffffff;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_compact(lfs, dir, cb, data, dir, 0, dir->count);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
return err;
}
// successful commit, lets update dir
dir->off = commit.off;
dir->etag = commit.ptag;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
const lfs_entrylist_t *regions) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_commitwith(lfs, dir, lfs_commit_regions, (void*)regions);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_append(lfs_t *lfs, lfs_dir_t *dir, uint16_t *id) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
*id = dir->count;
dir->count += 1;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_delete(lfs_t *lfs, lfs_dir_t *dir, uint16_t id) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir->count -= 1;
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// check if we should drop the directory block
if (dir->count == 0) {
lfs_dir_t pdir;
int res = lfs_pred(lfs, dir->pair, &pdir);
if (res < 0) {
return res;
}
if (res && pdir.split) {
pdir.split = dir->split;
pdir.tail[0] = dir->tail[0];
pdir.tail[1] = dir->tail[1];
int err = lfs_dir_commit(lfs, &pdir, &(lfs_entrylist_t){
{lfs_mktag(LFS_TYPE_SOFTTAIL + pdir.split*0x10,
0x1ff, sizeof(pdir.tail)),
.u.buffer=pdir.tail}});
return err;
}
}
int err = lfs_dir_commit_(lfs, dir, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_DELETE, id, 0)}});
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
if (err) {
return err;
}
// shift over any files that are affected
// TODO move this to dir_commit?
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if (lfs_paircmp(f->pair, dir->pair) == 0) {
if (f->id == id) {
f->pair[0] = 0xffffffff;
f->pair[1] = 0xffffffff;
} else if (f->id > id) {
f->id -= 1;
}
}
}
// TODO ?
// for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
// if (lfs_paircmp(d->pair, dir->pair) == 0) {
// if (d->id > id) {
// d->id -= 1;
// d->pos -= 1;
// }
// }
// }
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
struct lfs_dir_getter {
uint32_t mask;
lfs_tag_t tag;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_entry_t *entry;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
};
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_getter(lfs_t *lfs, void *p, lfs_entry_t entry) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
struct lfs_dir_getter *get = p;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if ((entry.tag & get->mask) == (get->tag & get->mask)) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
*get->entry = entry;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return true;
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
} else if (lfs_tag_type(entry.tag) == LFS_TYPE_DELETE) {
if (lfs_tag_id(entry.tag) <= lfs_tag_id(get->tag)) {
get->tag += lfs_mktag(0, 1, 0);
}
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_get(lfs_t *lfs, lfs_dir_t *dir,
uint32_t mask, lfs_entry_t *entry) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
uint16_t id = lfs_tag_id(entry->tag);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int res = lfs_dir_traverse(lfs, dir, lfs_dir_getter,
&(struct lfs_dir_getter){mask, entry->tag, entry});
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_NOENT;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO hmm, stop at commit? maybe we need to handle this elsewhere?
// Should commit get be its own thing? commit traverse?
if (id == dir->moveid && !dir->stop_at_commit) {
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int moved = lfs_moved(lfs, dir, dir->moveid);
if (moved < 0) {
return moved;
}
if (moved) {
return LFS_ERR_NOENT;
}
}
entry->tag = lfs_mktag(0, id, 0) | (entry->tag & 0xffe00fff);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_getbuffer(lfs_t *lfs, lfs_dir_t *dir,
uint32_t mask, lfs_entry_t *entry) {
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
void *buffer = entry->u.buffer;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_size_t size = lfs_tag_size(entry->tag);
int err = lfs_dir_get(lfs, dir, mask, entry);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
lfs_size_t diff = lfs_min(size, lfs_tag_size(entry->tag));
memset((uint8_t*)buffer + diff, 0, size - diff);
err = lfs_bd_read(lfs, entry->u.d.block, entry->u.d.off, buffer, diff);
if (err) {
return err;
}
if (lfs_tag_size(entry->tag) > size) {
return LFS_ERR_RANGE;
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_getentry(lfs_t *lfs, lfs_dir_t *dir,
uint32_t mask, lfs_tag_t tag, lfs_entry_t *entry) {
entry->tag = tag | sizeof(entry->u);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
entry->u.buffer = &entry->u;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_getbuffer(lfs, dir, mask, entry);
if (err && err != LFS_ERR_RANGE) {
return err;
}
return 0;
}
static int lfs_dir_getinfo(lfs_t *lfs, lfs_dir_t *dir,
int16_t id, struct lfs_info *info) {
if (id < 0) {
// special case for root
strcpy(info->name, "/");
info->type = LFS_TYPE_DIR;
return 0;
}
lfs_entry_t entry;
int err = lfs_dir_getentry(lfs, dir, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, id, 0), &entry);
if (err) {
return err;
}
info->type = lfs_tag_subtype(entry.tag);
if (lfs_tag_type(entry.tag) == (LFS_TYPE_REG | LFS_STRUCT_CTZ)) {
info->size = entry.u.ctz.size;
} else if (lfs_tag_type(entry.tag) == (LFS_TYPE_REG | LFS_STRUCT_INLINE)) {
info->size = lfs_tag_size(entry.tag);
}
err = lfs_dir_getbuffer(lfs, dir, 0x7ffff000, &(lfs_entry_t){
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs_mktag(LFS_TYPE_NAME, id, lfs->name_size+1),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.buffer=info->name});
if (err) {
return err;
}
return 0;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
struct lfs_dir_finder {
const char *name;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
uint16_t len;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
int16_t id;
};
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
static int lfs_dir_finder(lfs_t *lfs, void *p, lfs_entry_t entry) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
struct lfs_dir_finder *find = p;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_type(entry.tag) == LFS_TYPE_NAME &&
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs_tag_size(entry.tag) == find->len) {
int res = lfs_bd_cmp(lfs, entry.u.d.block, entry.u.d.off,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find->name, find->len);
if (res < 0) {
return res;
}
if (res) {
// found a match
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
find->id = lfs_tag_id(entry.tag);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
} else if (lfs_tag_type(entry.tag) == LFS_TYPE_DELETE) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (lfs_tag_id(entry.tag) == find->id) {
find->id = -1;
} else if (lfs_tag_id(entry.tag) < find->id) {
find->id -= 1;
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO drop others, make this only return id, also make get take in only entry to populate (with embedded tag)
static int lfs_dir_find(lfs_t *lfs, lfs_dir_t *dir,
const char **path, int16_t *id) {
lfs_entry_t entry = {
.u.pair[0] = lfs->root[0],
.u.pair[1] = lfs->root[1],
};
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
struct lfs_dir_finder find = {
.name = *path,
};
while (true) {
nextname:
// skip slashes
find.name += strspn(find.name, "/");
find.len = strcspn(find.name, "/");
// special case for root dir
if (find.name[0] == '\0') {
// TODO set up root?
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*id = -1;
return 0;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
// skip '.' and root '..'
if ((find.len == 1 && memcmp(find.name, ".", 1) == 0) ||
(find.len == 2 && memcmp(find.name, "..", 2) == 0)) {
find.name += find.len;
goto nextname;
}
// skip if matched by '..' in name
const char *suffix = find.name + find.len;
lfs_size_t sufflen;
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
find.name = suffix + sufflen;
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
// update what we've found
*path = find.name;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// find path
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
while (true) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
//printf("checking %d %d for %s\n", entry.u.pair[0], entry.u.pair[1], *path);
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find.id = -1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetchwith(lfs, dir, entry.u.pair,
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
lfs_dir_finder, &find);
if (err) {
return err;
}
if (find.id >= 0) {
// found it
break;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (!dir->split) {
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
return LFS_ERR_NOENT;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
entry.u.pair[0] = dir->tail[0];
entry.u.pair[1] = dir->tail[1];
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (find.id == dir->moveid) {
int moved = lfs_moved(lfs, dir, dir->moveid);
if (moved < 0) {
return moved;
}
if (moved) {
return LFS_ERR_NOENT;
}
}
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*id = find.id;
Added rudimentary framework for journaling metadata pairs This is a big change stemming from the fact that resizable entries were surprisingly complicated to implement and came in with a sizable code cost. The theory is that the journalling has a comparable cost to resizable entries. Both need to handle overflowing blocks, and managing offsets is comparable to managing attribute IDs. But by jumping all the way to full journaling, we can statically wear-level the metadata written to metadata pairs. The idea of journaling littlefs's metadata has been mentioned several times in discussions and fits well into how littlefs works. You could even view the existing metadata log as a log of size 2. The downside of this approach is that changing the metadata in this way would break compatibility from the existing layout on disk. Something that resizable entries does not do. That being said, adopting journaling at the metadata layer offers a big improvement to littlefs's performance and wear-leveling, with very little cost (maybe even none or negative after resizable entries?).
2018-05-19 23:25:47 +00:00
find.name += find.len;
find.name += strspn(find.name, "/");
if (find.name[0] == '\0') {
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// TODO optimize grab for inline files and like?
// TODO would this mean more code?
// grab the entry data
int err = lfs_dir_getentry(lfs, dir, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, find.id, 0), &entry);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (err) {
return err;
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// continue on if we hit a directory
// TODO update with what's on master?
if (lfs_tag_subtype(entry.tag) != LFS_TYPE_DIR) {
return LFS_ERR_NOTDIR;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
}
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static int lfs_dir_findbuffer(lfs_t *lfs, lfs_dir_t *dir,
// const char **path, lfs_entry_t *entry) {
// void *buffer = entry->u.buffer;
// lfs_size_t size = lfs_tag_size(entry->tag);
// int err = lfs_dir_find(lfs, dir, path, entry);
// if (err) {
// return err;
// }
//
// lfs_size_t diff = lfs_min(size, lfs_tag_size(entry->tag));
// memset((uint8_t*)buffer + diff, 0, size - diff);
// // TODO hmm
// if (lfs_tag_valid(entry->tag)) {
// memcpy(buffer, entry->u.buffer, diff);
// } else {
// err = lfs_bd_read(lfs, entry->u.d.block, entry->u.d.off, buffer, diff);
// if (err) {
// return err;
// }
// }
//
// if (lfs_tag_size(entry->tag) > size) {
// return LFS_ERR_RANGE;
// }
//
// return 0;
//}
//
//static int lfs_dir_findentry(lfs_t *lfs, lfs_dir_t *dir,
// const char **path, lfs_entry_t *entry) {
// entry->tag = sizeof(entry->u);
// entry->u.buffer = &entry->u;
// return lfs_dir_findbuffer(lfs, dir, path, entry);
//}
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//////////////////////////////////////////////////////////
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static int lfs_dir_alloc(lfs_t *lfs, lfs_dir_t *dir) {
// // allocate pair of dir blocks
// for (int i = 0; i < 2; i++) {
// int err = lfs_alloc(lfs, &dir->pair[i]);
// if (err) {
// return err;
// }
// }
//
// // rather than clobbering one of the blocks we just pretend
// // the revision may be valid
// int err = lfs_bd_read(lfs, dir->pair[0], 0, &dir->d.rev, 4);
// dir->d.rev = lfs_fromle32(dir->d.rev);
// if (err) {
// return err;
// }
//
// // set defaults
// dir->d.rev += 1;
// dir->d.size = sizeof(dir->d)+4;
// dir->d.tail[0] = 0xffffffff;
// dir->d.tail[1] = 0xffffffff;
// dir->off = sizeof(dir->d);
//
// // don't write out yet, let caller take care of that
// return 0;
//}
//
//static int lfs_dir_fetch(lfs_t *lfs,
// lfs_dir_t *dir, const lfs_block_t pair[2]) {
// // copy out pair, otherwise may be aliasing dir
// const lfs_block_t tpair[2] = {pair[0], pair[1]};
// bool valid = false;
//
// // check both blocks for the most recent revision
// for (int i = 0; i < 2; i++) {
// struct lfs_disk_dir test;
// int err = lfs_bd_read(lfs, tpair[i], 0, &test, sizeof(test));
// lfs_dir_fromle32(&test);
// if (err) {
// return err;
// }
//
// if (valid && lfs_scmp(test.rev, dir->d.rev) < 0) {
// continue;
// }
//
// if ((0x7fffffff & test.size) < sizeof(test)+4 ||
// (0x7fffffff & test.size) > lfs->cfg->block_size) {
// continue;
// }
//
// uint32_t crc = 0xffffffff;
// lfs_dir_tole32(&test);
// lfs_crc(&crc, &test, sizeof(test));
// lfs_dir_fromle32(&test);
// err = lfs_bd_crc(lfs, tpair[i], sizeof(test),
// (0x7fffffff & test.size) - sizeof(test), &crc);
// if (err) {
// return err;
// }
//
// if (crc != 0) {
// continue;
// }
//
// valid = true;
//
// // setup dir in case it's valid
// dir->pair[0] = tpair[(i+0) % 2];
// dir->pair[1] = tpair[(i+1) % 2];
// dir->off = sizeof(dir->d);
// dir->d = test;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// if (!valid) {
// LFS_ERROR("Corrupted dir pair at %d %d", tpair[0], tpair[1]);
// return LFS_ERR_CORRUPT;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// return 0;
//}
//
//struct lfs_region {
// enum {
// LFS_FROM_MEM,
// LFS_FROM_REGION,
// LFS_FROM_ATTRS,
// } type;
//
// lfs_off_t oldoff;
// lfs_size_t oldsize;
// const void *buffer;
// lfs_size_t newsize;
//};
//
//struct lfs_region_attrs {
// const struct lfs_attr *attrs;
// int count;
//};
//
//struct lfs_region_region {
// lfs_block_t block;
// lfs_off_t off;
// struct lfs_region *regions;
// int count;
//};
//
//static int lfs_commit_region(lfs_t *lfs, uint32_t *crc,
// lfs_block_t oldblock, lfs_off_t oldoff,
// lfs_block_t newblock, lfs_off_t newoff,
// lfs_off_t regionoff, lfs_size_t regionsize,
// const struct lfs_region *regions, int count) {
// int i = 0;
// lfs_size_t newend = newoff + regionsize;
// while (newoff < newend) {
// // commit from different types of regions
// if (i < count && regions[i].oldoff == oldoff - regionoff) {
// switch (regions[i].type) {
// case LFS_FROM_MEM: {
// lfs_crc(crc, regions[i].buffer, regions[i].newsize);
// int err = lfs_bd_prog(lfs, newblock, newoff,
// regions[i].buffer, regions[i].newsize);
// if (err) {
// return err;
// }
// newoff += regions[i].newsize;
// oldoff += regions[i].oldsize;
// break;
// }
// case LFS_FROM_REGION: {
// const struct lfs_region_region *disk = regions[i].buffer;
// int err = lfs_commit_region(lfs, crc,
// disk->block, disk->off,
// newblock, newoff,
// disk->off, regions[i].newsize,
// disk->regions, disk->count);
// if (err) {
// return err;
// }
// newoff += regions[i].newsize;
// oldoff -= regions[i].oldsize;
// break;
// }
// case LFS_FROM_ATTRS: {
// const struct lfs_region_attrs *attrs = regions[i].buffer;
//
// // order doesn't matter, so we write new attrs first. this
// // is still O(n^2) but only O(n) disk access
// for (int j = 0; j < attrs->count; j++) {
// if (attrs->attrs[j].size == 0) {
// continue;
// }
//
// lfs_entry_attr_t attr;
// attr.d.type = attrs->attrs[j].type;
// attr.d.len = attrs->attrs[j].size;
//
// lfs_crc(crc, &attr.d, sizeof(attr.d));
// int err = lfs_bd_prog(lfs, newblock, newoff,
// &attr.d, sizeof(attr.d));
// if (err) {
// return err;
// }
//
// lfs_crc(crc,
// attrs->attrs[j].buffer, attrs->attrs[j].size);
// err = lfs_bd_prog(lfs, newblock, newoff+sizeof(attr.d),
// attrs->attrs[j].buffer, attrs->attrs[j].size);
// if (err) {
// return err;
// }
//
// newoff += 2+attrs->attrs[j].size;
// }
//
// // copy over attributes without updates
// lfs_off_t oldend = oldoff + regions[i].oldsize;
// while (oldoff < oldend) {
// lfs_entry_attr_t attr;
// int err = lfs_bd_read(lfs, oldblock, oldoff,
// &attr.d, sizeof(attr.d));
// if (err) {
// return err;
// }
//
// bool updating = false;
// for (int j = 0; j < attrs->count; j++) {
// if (attr.d.type == attrs->attrs[j].type) {
// updating = true;
// }
// }
//
// if (!updating) {
// err = lfs_commit_region(lfs, crc,
// oldblock, oldoff,
// newblock, newoff,
// 0, 2+attr.d.len,
// NULL, 0);
// if (err) {
// return err;
// }
//
// newoff += 2+attr.d.len;
// }
//
// oldoff += 2+attr.d.len;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// break;
// }
// }
//
// i += 1;
// } else {
// // copy data from old block if not covered by entry
// uint8_t data;
// int err = lfs_bd_read(lfs, oldblock, oldoff, &data, 1);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
// return err;
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// lfs_crc(crc, &data, 1);
// err = lfs_bd_prog(lfs, newblock, newoff, &data, 1);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
// return err;
// }
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//
// oldoff += 1;
// newoff += 1;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// }
//
// // sanity check our commit math
// LFS_ASSERT(newoff == newend);
// return 0;
//}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
// const struct lfs_region *regions, int count) {
// // state for copying over
// const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
// bool relocated = false;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // increment revision count
// dir->d.rev += 1;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // keep pairs in order such that pair[0] is most recent
// lfs_pairswap(dir->pair);
// for (int i = 0; i < count; i++) {
// dir->d.size += regions[i].newsize;
// dir->d.size -= regions[i].oldsize;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// while (true) {
// if (true) {
// int err = lfs_bd_erase(lfs, dir->pair[0]);
// if (err) {
// if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
// return err;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // commit header
// uint32_t crc = 0xffffffff;
// lfs_dir_tole32(&dir->d);
// lfs_crc(&crc, &dir->d, sizeof(dir->d));
// err = lfs_bd_prog(lfs, dir->pair[0], 0, &dir->d, sizeof(dir->d));
// lfs_dir_fromle32(&dir->d);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// return err;
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // commit entry
// err = lfs_commit_region(lfs, &crc,
// dir->pair[1], sizeof(dir->d),
// dir->pair[0], sizeof(dir->d),
// 0, (0x7fffffff & dir->d.size)-sizeof(dir->d)-4,
// regions, count);
// if (err) {
// if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
// return err;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // commit crc
// crc = lfs_tole32(crc);
// err = lfs_bd_prog(lfs, dir->pair[0],
// (0x7fffffff & dir->d.size)-4, &crc, 4);
// crc = lfs_fromle32(crc);
// if (err) {
// if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
// return err;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// err = lfs_bd_sync(lfs);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// if (err == LFS_ERR_CORRUPT) {
// goto relocate;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// return err;
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // successful commit, check checksum to make sure
// uint32_t ncrc = 0xffffffff;
// err = lfs_bd_crc(lfs, dir->pair[0], 0,
// (0x7fffffff & dir->d.size)-4, &ncrc);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
// return err;
// }
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//
// if (ncrc != crc) {
// goto relocate;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// break;
//
//relocate:
// //commit was corrupted
// LFS_DEBUG("Bad block at %d", dir->pair[0]);
//
// // drop caches and prepare to relocate block
// relocated = true;
// lfs->pcache.block = 0xffffffff;
//
// // can't relocate superblock, filesystem is now frozen
// if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
// LFS_WARN("Superblock %d has become unwritable", oldpair[0]);
// return LFS_ERR_CORRUPT;
// }
//
// // relocate half of pair
// int err = lfs_alloc(lfs, &dir->pair[0]);
// if (err) {
// return err;
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// if (relocated) {
// // update references if we relocated
// LFS_DEBUG("Relocating %d %d to %d %d",
// oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
// int err = lfs_relocate(lfs, oldpair, dir->pair);
// if (err) {
// return err;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// }
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // shift over any directories that are affected
// for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
// if (lfs_paircmp(d->pair, dir->pair) == 0) {
// d->pair[0] = dir->pair[0];
// d->pair[1] = dir->pair[1];
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// return 0;
//}
//
//static int lfs_dir_get(lfs_t *lfs, const lfs_dir_t *dir,
// lfs_off_t off, void *buffer, lfs_size_t size) {
// return lfs_bd_read(lfs, dir->pair[0], off, buffer, size);
//}
//
//static int lfs_dir_set(lfs_t *lfs, lfs_dir_t *dir, lfs_entry_t *entry,
// struct lfs_region *regions, int count) {
// return -999;
//// lfs_ssize_t diff = 0;
//// for (int i = 0; i < count; i++) {
//// diff += regions[i].newsize;
//// diff -= regions[i].oldsize;
//// }
////
//// lfs_size_t oldsize = entry->size;
//// if (entry->off == 0) {
//// entry->off = (0x7fffffff & dir->d.size) - 4;
//// }
////
//// if ((0x7fffffff & dir->d.size) + diff > lfs->cfg->block_size) {
//// lfs_dir_t olddir = *dir;
//// lfs_off_t oldoff = entry->off;
////
//// if (oldsize) {
//// // mark as moving
//// uint8_t type;
//// int err = lfs_dir_get(lfs, &olddir, oldoff, &type, 1);
//// if (err) {
//// return err;
//// }
////
//// type |= LFS_STRUCT_MOVED;
//// err = lfs_dir_commit(lfs, &olddir, (struct lfs_region[]){
//// {LFS_FROM_MEM, oldoff, 1, &type, 1}}, 1);
//// if (err) {
//// return err;
//// }
//// }
////
//// lfs_dir_t pdir = olddir;
////
//// // find available block or create a new one
//// while ((0x7fffffff & dir->d.size) + oldsize + diff
//// > lfs->cfg->block_size) {
//// // we need to allocate a new dir block
//// if (!(0x80000000 & dir->d.size)) {
//// pdir = *dir;
//// int err = lfs_dir_alloc(lfs, dir);
//// if (err) {
//// return err;
//// }
////
//// dir->d.tail[0] = pdir.d.tail[0];
//// dir->d.tail[1] = pdir.d.tail[1];
////
//// break;
//// }
////
//// int err = lfs_dir_fetch(lfs, dir, dir->d.tail);
//// if (err) {
//// return err;
//// }
//// }
////
//// // writing out new entry
//// entry->off = dir->d.size - 4;
//// entry->size += diff;
//// int err = lfs_dir_commit(lfs, dir, (struct lfs_region[]){
//// {LFS_FROM_REGION, entry->off, 0, &(struct lfs_region_region){
//// olddir.pair[0], oldoff,
//// regions, count}, entry->size}}, 1);
//// if (err) {
//// return err;
//// }
////
//// // update pred dir, unless pred == old we can coalesce
//// if (!oldsize || lfs_paircmp(pdir.pair, olddir.pair) != 0) {
//// pdir.d.size |= 0x80000000;
//// pdir.d.tail[0] = dir->pair[0];
//// pdir.d.tail[1] = dir->pair[1];
////
//// err = lfs_dir_commit(lfs, &pdir, NULL, 0);
//// if (err) {
//// return err;
//// }
//// } else if (oldsize) {
//// olddir.d.size |= 0x80000000;
//// olddir.d.tail[0] = dir->pair[0];
//// olddir.d.tail[1] = dir->pair[1];
//// }
////
//// // remove old entry
//// if (oldsize) {
//// lfs_entry_t oldentry;
//// oldentry.off = oldoff;
//// err = lfs_dir_set(lfs, &olddir, &oldentry, (struct lfs_region[]){
//// {LFS_FROM_MEM, 0, oldsize, NULL, 0}}, 1);
//// if (err) {
//// return err;
//// }
//// }
////
//// goto shift;
//// }
////
//// if ((0x7fffffff & dir->d.size) + diff == sizeof(dir->d)+4) {
//// lfs_dir_t pdir;
//// int res = lfs_pred(lfs, dir->pair, &pdir);
//// if (res < 0) {
//// return res;
//// }
////
//// if (pdir.d.size & 0x80000000) {
//// pdir.d.size &= dir->d.size | 0x7fffffff;
//// pdir.d.tail[0] = dir->d.tail[0];
//// pdir.d.tail[1] = dir->d.tail[1];
//// int err = lfs_dir_commit(lfs, &pdir, NULL, 0);
//// if (err) {
//// return err;
//// }
//// goto shift;
//// }
//// }
////
//// for (int i = 0; i < count; i++) {
//// regions[i].oldoff += entry->off;
//// }
////
//// int err = lfs_dir_commit(lfs, dir, regions, count);
//// if (err) {
//// return err;
//// }
////
//// entry->size += diff;
////
////shift:
//// // shift over any files/directories that are affected
//// for (lfs_file_t *f = lfs->files; f; f = f->next) {
//// if (lfs_paircmp(f->pair, dir->pair) == 0) {
//// if (f->pairoff == entry->off && entry->size == 0) {
//// f->pair[0] = 0xffffffff;
//// f->pair[1] = 0xffffffff;
//// } else if (f->pairoff > entry->off) {
//// f->pairoff += diff;
//// }
//// }
//// }
////
//// for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
//// if (lfs_paircmp(d->pair, dir->pair) == 0) {
//// if (d->off > entry->off) {
//// d->off += diff;
//// d->pos += diff;
//// }
//// }
//// }
////
//// return 0;
//}
//
//static int lfs_dir_next(lfs_t *lfs, lfs_dir_t *dir, lfs_entry_t *entry) {
// while (dir->off >= (0x7fffffff & dir->d.size)-4) {
// if (!(0x80000000 & dir->d.size)) {
// entry->off = dir->off;
// return LFS_ERR_NOENT;
// }
//
// int err = lfs_dir_fetch(lfs, dir, dir->d.tail);
// if (err) {
// return err;
// }
//
// dir->off = sizeof(dir->d);
// dir->pos += sizeof(dir->d) + 4;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// int err = lfs_dir_get(lfs, dir, dir->off, &entry->d, sizeof(entry->d));
// lfs_entry_fromle32(&entry->d);
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// if (err) {
// return err;
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// entry->off = dir->off;
// entry->size = lfs_entry_size(entry);
// dir->off += entry->size;
// dir->pos += entry->size;
// return 0;
//}
//
//static int lfs_dir_find(lfs_t *lfs, lfs_dir_t *dir,
// lfs_entry_t *entry, const char **path) {
// const char *pathname = *path;
// lfs_size_t pathlen;
//
// while (true) {
// nextname:
// // skip slashes
// pathname += strspn(pathname, "/");
// pathlen = strcspn(pathname, "/");
//
// // special case for root dir
// if (pathname[0] == '\0') {
// *entry = (lfs_entry_t){
// .d.type = LFS_STRUCT_DIR | LFS_TYPE_DIR,
// .d.u.dir[0] = lfs->root[0],
// .d.u.dir[1] = lfs->root[1],
// };
// return 0;
// }
//
// // skip '.' and root '..'
// if ((pathlen == 1 && memcmp(pathname, ".", 1) == 0) ||
// (pathlen == 2 && memcmp(pathname, "..", 2) == 0)) {
// pathname += pathlen;
// goto nextname;
// }
//
// // skip if matched by '..' in name
// const char *suffix = pathname + pathlen;
// lfs_size_t sufflen;
// int depth = 1;
// while (true) {
// suffix += strspn(suffix, "/");
// sufflen = strcspn(suffix, "/");
// if (sufflen == 0) {
// break;
// }
//
// if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
// depth -= 1;
// if (depth == 0) {
// pathname = suffix + sufflen;
// goto nextname;
// }
// } else {
// depth += 1;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//
// suffix += sufflen;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // update what we've found
// *path = pathname;
//
// // find path
// while (true) {
// int err = lfs_dir_next(lfs, dir, entry);
// if (err) {
// return err;
// }
//
// if (((0xf & entry->d.type) != LFS_TYPE_REG &&
// (0xf & entry->d.type) != LFS_TYPE_DIR) ||
// entry->d.nlen != pathlen) {
// continue;
// }
//
// int res = lfs_bd_cmp(lfs, dir->pair[0],
// entry->off + entry->size - pathlen,
// pathname, pathlen);
// if (res < 0) {
// return res;
// }
//
// // found match
// if (res) {
// break;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// }
// }
//
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// // check that entry has not been moved
// if (entry->d.type & LFS_STRUCT_MOVED) {
// int moved = lfs_moved(lfs, &entry->d.u);
// if (moved < 0 || moved) {
// return (moved < 0) ? moved : LFS_ERR_NOENT;
// }
//
// entry->d.type &= ~LFS_STRUCT_MOVED;
// }
//
// pathname += pathlen;
// pathname += strspn(pathname, "/");
// if (pathname[0] == '\0') {
// return 0;
// }
//
// // continue on if we hit a directory
// if ((0xf & entry->d.type) != LFS_TYPE_DIR) {
// return LFS_ERR_NOTDIR;
// }
//
// int err = lfs_dir_fetch(lfs, dir, entry->d.u.dir);
// if (err) {
// return err;
// }
// }
//}
//
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
/// Internal attribute operations ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
//static int lfs_dir_getinfo(lfs_t *lfs,
// lfs_dir_t *dir, const lfs_entry_t *entry, struct lfs_info *info) {
// memset(info, 0, sizeof(*info));
// info->type = 0xf & entry->d.type;
// if (entry->d.type == (LFS_STRUCT_CTZ | LFS_TYPE_REG)) {
// info->size = entry->d.u.file.size;
// } else if (entry->d.type == (LFS_STRUCT_INLINE | LFS_TYPE_REG)) {
// info->size = lfs_entry_elen(entry);
// }
//
// if (lfs_paircmp(entry->d.u.dir, lfs->root) == 0) {
// strcpy(info->name, "/");
// } else {
// int err = lfs_dir_get(lfs, dir,
// entry->off + entry->size - entry->d.nlen,
// info->name, entry->d.nlen);
// if (err) {
// return err;
// }
// }
//
// return 0;
//}
//
//static int lfs_dir_getattrs(lfs_t *lfs,
// lfs_dir_t *dir, const lfs_entry_t *entry,
// const struct lfs_attr *attrs, int count) {
// // set to zero in case we can't find the attributes or size mismatch
// for (int j = 0; j < count; j++) {
// memset(attrs[j].buffer, 0, attrs[j].size);
// }
//
// // search for attribute in attribute entry
// lfs_off_t off = entry->off + 4+lfs_entry_elen(entry);
// lfs_off_t end = off + lfs_entry_alen(entry);
// while (off < end) {
// lfs_entry_attr_t attr;
// int err = lfs_dir_get(lfs, dir, off, &attr.d, sizeof(attr.d));
// if (err) {
// return err;
// }
//
// for (int j = 0; j < count; j++) {
// if (attrs[j].type == attr.d.type) {
// if (attrs[j].size < attr.d.len) {
// return LFS_ERR_RANGE;
// }
//
// err = lfs_dir_get(lfs, dir, off+sizeof(attr.d),
// attrs[j].buffer, attr.d.len);
// if (err) {
// return err;
// }
// }
// }
//
// off += 2+attr.d.len;
// }
//
// return 0;
//}
//
//static lfs_ssize_t lfs_dir_checkattrs(lfs_t *lfs,
// lfs_dir_t *dir, lfs_entry_t *entry,
// const struct lfs_attr *attrs, int count) {
// // check that attributes fit
// // two separate passes so disk access is O(n)
// lfs_size_t nsize = 0;
// for (int j = 0; j < count; j++) {
// if (attrs[j].size > 0) {
// nsize += 2+attrs[j].size;
// }
// }
//
// lfs_off_t off = entry->off + 4+lfs_entry_elen(entry);
// lfs_off_t end = off + lfs_entry_alen(entry);
// while (off < end) {
// lfs_entry_attr_t attr;
// int err = lfs_dir_get(lfs, dir, off, &attr.d, sizeof(attr.d));
// if (err) {
// return err;
// }
//
// bool updated = false;
// for (int j = 0; j < count; j++) {
// if (attr.d.type == attrs[j].type) {
// updated = true;
// }
// }
//
// if (!updated) {
// nsize += 2+attr.d.len;
// }
//
// off += 2+attr.d.len;
// }
//
// if (nsize > lfs->attrs_size || (
// lfs_entry_size(entry) - lfs_entry_alen(entry) + nsize
// > lfs->cfg->block_size)) {
// return LFS_ERR_NOSPC;
// }
//
// return nsize;
//}
//
//static int lfs_dir_setattrs(lfs_t *lfs,
// lfs_dir_t *dir, lfs_entry_t *entry,
// const struct lfs_attr *attrs, int count) {
// // make sure attributes fit
// lfs_size_t oldlen = lfs_entry_alen(entry);
// lfs_ssize_t newlen = lfs_dir_checkattrs(lfs, dir, entry, attrs, count);
// if (newlen < 0) {
// return newlen;
// }
//
// // commit to entry, majority of work is in LFS_FROM_ATTRS
// entry->d.alen = (0xc0 & entry->d.alen) | newlen;
// return lfs_dir_set(lfs, dir, entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, 4, &entry->d, 4},
// {LFS_FROM_ATTRS, 4+lfs_entry_elen(entry), oldlen,
// &(struct lfs_region_attrs){attrs, count}, newlen}}, 2);
//}
//
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_deorphan(lfs);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t cwd;
int err = lfs_dir_find(lfs, &cwd, &path, &(int16_t){0});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err != LFS_ERR_NOENT || strchr(path, '/') != NULL) {
if (!err) {
return LFS_ERR_EXIST;
}
return err;
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// build up new directory
lfs_alloc_ack(lfs);
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t dir;
err = lfs_dir_alloc(lfs, &dir, false, cwd.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &dir, NULL);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// get next slot and commit
uint16_t id;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_append(lfs, &cwd, &id);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
cwd.tail[0] = dir.pair[0];
cwd.tail[1] = dir.pair[1];
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &cwd, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_NAME, id, nlen),
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
.u.buffer=(void*)path}, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_DIR | LFS_STRUCT_DIR, id, sizeof(dir.pair)),
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.u.buffer=dir.pair}, &(lfs_entrylist_t){
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x1ff, sizeof(cwd.tail)),
.u.buffer=cwd.tail}}}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// TODO need ack here?
lfs_alloc_ack(lfs);
return 0;
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_mkdir(lfs_t *lfs, const char *path) {
// // deorphan if we haven't yet, needed at most once after poweron
// if (!lfs->deorphaned) {
// int err = lfs_deorphan(lfs);
// if (err) {
// return err;
// }
// }
//
// // fetch parent directory
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry;
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err != LFS_ERR_NOENT || strchr(path, '/') != NULL) {
// return err ? err : LFS_ERR_EXIST;
// }
//
// // check that name fits
// lfs_size_t nlen = strlen(path);
// if (nlen > lfs->name_size) {
// return LFS_ERR_NAMETOOLONG;
// }
//
// // build up new directory
// lfs_alloc_ack(lfs);
//
// lfs_dir_t dir;
// err = lfs_dir_alloc(lfs, &dir);
// if (err) {
// return err;
// }
// dir.d.tail[0] = cwd.d.tail[0];
// dir.d.tail[1] = cwd.d.tail[1];
//
// err = lfs_dir_commit(lfs, &dir, NULL, 0);
// if (err) {
// return err;
// }
//
// entry.d.type = LFS_STRUCT_DIR | LFS_TYPE_DIR;
// entry.d.elen = sizeof(entry.d) - 4;
// entry.d.alen = 0;
// entry.d.nlen = nlen;
// entry.d.u.dir[0] = dir.pair[0];
// entry.d.u.dir[1] = dir.pair[1];
// entry.size = 0;
//
// cwd.d.tail[0] = dir.pair[0];
// cwd.d.tail[1] = dir.pair[1];
// lfs_entry_tole32(&entry.d);
// err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, 0, &entry.d, sizeof(entry.d)},
// {LFS_FROM_MEM, 0, 0, path, nlen}}, 2);
// if (err) {
// return err;
// }
//
// lfs_alloc_ack(lfs);
// return 0;
//}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
int16_t id;
int err = lfs_dir_find(lfs, dir, &path, &id);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_entry_t entry;
if (id < 0) {
// handle root dir separately
entry.u.pair[0] = lfs->root[0];
entry.u.pair[1] = lfs->root[1];
} else {
// get dir pair from parent
err = lfs_dir_getentry(lfs, dir, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, id, 0), &entry);
if (err) {
return err;
}
if (lfs_tag_subtype(entry.tag) != LFS_TYPE_DIR) {
return LFS_ERR_NOTDIR;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// fetch first pair
err = lfs_dir_fetch(lfs, dir, entry.u.pair);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// setup head dir
dir->head[0] = dir->pair[0];
dir->head[1] = dir->pair[1];
dir->pos = 0;
dir->id = 0;
// add to list of directories
dir->next = lfs->dirs;
lfs->dirs = dir;
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// remove from list of directories
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (lfs_dir_t **p = &lfs->dirs; *p; p = &(*p)->next) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (*p == dir) {
*p = dir->next;
break;
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
memset(info, 0, sizeof(*info));
// special offset for '.' and '..'
if (dir->pos == 0) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
dir->pos += 1;
return 1;
} else if (dir->pos == 1) {
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
dir->pos += 1;
return 1;
}
while (true) {
if (dir->id == dir->count) {
if (!dir->split) {
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
dir->id = 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_getinfo(lfs, dir, dir->id, info);
if (err && err != LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return err;
}
dir->id += 1;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (err != LFS_ERR_NOENT) {
break;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
dir->pos += 1;
return true;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// TODO does this work?
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// simply walk from head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_rewind(lfs, dir);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// first two for ./..
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->pos = lfs_min(2, off);
off -= dir->pos;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
while (off != 0) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
dir->id = lfs_min(dir->count, off);
dir->pos += dir->id;
off -= dir->id;
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (dir->id == dir->count) {
if (!dir->split) {
return LFS_ERR_INVAL;
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->tail);
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (err) {
return err;
}
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// reload the head dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->head);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
return err;
}
dir->pair[0] = dir->head[0];
dir->pair[1] = dir->head[1];
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
dir->pos = 0;
dir->id = 0;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
// dir->pair[0] = lfs->root[0];
// dir->pair[1] = lfs->root[1];
//
// int err = lfs_dir_fetch(lfs, dir, dir->pair);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry;
// err = lfs_dir_find(lfs, dir, &entry, &path);
// if (err) {
// return err;
// } else if (entry.d.type != (LFS_STRUCT_DIR | LFS_TYPE_DIR)) {
// return LFS_ERR_NOTDIR;
// }
//
// err = lfs_dir_fetch(lfs, dir, entry.d.u.dir);
// if (err) {
// return err;
// }
//
// // setup head dir
// // special offset for '.' and '..'
// dir->head[0] = dir->pair[0];
// dir->head[1] = dir->pair[1];
// dir->pos = sizeof(dir->d) - 2;
// dir->off = sizeof(dir->d);
//
// // add to list of directories
// dir->next = lfs->dirs;
// lfs->dirs = dir;
//
// return 0;
//}
//
//int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
// // remove from list of directories
// for (lfs_dir_t **p = &lfs->dirs; *p; p = &(*p)->next) {
// if (*p == dir) {
// *p = dir->next;
// break;
// }
// }
//
// return 0;
//}
//
//int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
// memset(info, 0, sizeof(*info));
//
// // special offset for '.' and '..'
// if (dir->pos == sizeof(dir->d) - 2) {
// info->type = LFS_TYPE_DIR;
// strcpy(info->name, ".");
// dir->pos += 1;
// return 1;
// } else if (dir->pos == sizeof(dir->d) - 1) {
// info->type = LFS_TYPE_DIR;
// strcpy(info->name, "..");
// dir->pos += 1;
// return 1;
// }
//
// lfs_entry_t entry;
// while (true) {
// int err = lfs_dir_next(lfs, dir, &entry);
// if (err) {
// return (err == LFS_ERR_NOENT) ? 0 : err;
// }
//
// if ((0xf & entry.d.type) != LFS_TYPE_REG &&
// (0xf & entry.d.type) != LFS_TYPE_DIR) {
// continue;
// }
//
// // check that entry has not been moved
// if (entry.d.type & LFS_STRUCT_MOVED) {
// int moved = lfs_moved(lfs, &entry.d.u);
// if (moved < 0) {
// return moved;
// }
//
// if (moved) {
// continue;
// }
//
// entry.d.type &= ~LFS_STRUCT_MOVED;
// }
//
// break;
// }
//
// int err = lfs_dir_getinfo(lfs, dir, &entry, info);
// if (err) {
// return err;
// }
//
// return 1;
//}
//
//int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
// // simply walk from head dir
// int err = lfs_dir_rewind(lfs, dir);
// if (err) {
// return err;
// }
// dir->pos = off;
//
// while (off > (0x7fffffff & dir->d.size)) {
// off -= 0x7fffffff & dir->d.size;
// if (!(0x80000000 & dir->d.size)) {
// return LFS_ERR_INVAL;
// }
//
// err = lfs_dir_fetch(lfs, dir, dir->d.tail);
// if (err) {
// return err;
// }
// }
//
// dir->off = off;
// return 0;
//}
//
//lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
// (void)lfs;
// return dir->pos;
//}
//
//int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
// // reload the head dir
// int err = lfs_dir_fetch(lfs, dir, dir->head);
// if (err) {
// return err;
// }
//
// dir->pair[0] = dir->head[0];
// dir->pair[1] = dir->head[1];
// dir->pos = sizeof(dir->d) - 2;
// dir->off = sizeof(dir->d);
// return 0;
//}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_find(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs_cache_read(lfs, rcache, pcache, head, 4*skip, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_extend(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
if (true) {
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
size -= 1;
lfs_off_t index = lfs_ctz_index(lfs, &size);
size += 1;
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_read(lfs, rcache, NULL,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = size;
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
for (lfs_off_t i = 0; i < skips; i++) {
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_tole32(head);
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
err = lfs_cache_read(lfs, rcache, NULL,
head, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
LFS_DEBUG("Bad block at %d", nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
pcache->block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_traverse(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int (*cb)(lfs_t*, void*, lfs_block_t), void *data) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (size == 0) {
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = cb(lfs, data, head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
err = lfs_cache_read(lfs, rcache, pcache, head, 0, &heads, count*4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = cb(lfs, data, heads[i]);
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
const char *path, int flags) {
// deorphan if we haven't yet, needed at most once after poweron
if ((flags & 3) != LFS_O_RDONLY && !lfs->deorphaned) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_deorphan(lfs);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
}
// allocate entry for file if it doesn't exist
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t cwd;
int16_t id;
int err = lfs_dir_find(lfs, &cwd, &path, &id);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err && (err != LFS_ERR_NOENT || strchr(path, '/') != NULL)) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_entry_t entry;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err == LFS_ERR_NOENT) {
if (!(flags & LFS_O_CREAT)) {
return LFS_ERR_NOENT;
}
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// get next slot and create entry to remember name
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_append(lfs, &cwd, &id);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// TODO do we need to make file registered to list to catch updates from this commit? ie if id/cwd change
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &cwd, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_NAME, id, nlen),
.u.buffer=(void*)path}, &(lfs_entrylist_t){
{lfs_mktag(LFS_TYPE_REG | LFS_STRUCT_INLINE, id, 0)}}});
if (err) {
return err;
}
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
// TODO eh
if (id >= cwd.count) {
// catch updates from a compact in the above commit
id -= cwd.count;
cwd.pair[0] = cwd.tail[0];
cwd.pair[1] = cwd.tail[1];
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
entry.tag = lfs_mktag(LFS_TYPE_REG | LFS_STRUCT_INLINE, id, 0);
} else {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
if (id == -1) {
return LFS_ERR_ISDIR;
} else if (flags & LFS_O_EXCL) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return LFS_ERR_EXIST;
}
entry.tag = lfs_mktag(LFS_TYPE_REG, id, 0);
err = lfs_dir_get(lfs, &cwd, 0x701ff000, &entry);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_subtype(entry.tag) != LFS_TYPE_REG) {
return LFS_ERR_ISDIR;
}
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
// setup file struct
file->pair[0] = cwd.pair[0];
file->pair[1] = cwd.pair[1];
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
file->id = id;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
file->flags = flags;
file->pos = 0;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->attrs = NULL;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
file->cache.block = 0xffffffff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (lfs->cfg->file_buffer) {
file->cache.buffer = lfs->cfg->file_buffer;
} else if ((file->flags & 3) == LFS_O_RDONLY) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->read_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_struct(entry.tag) == LFS_STRUCT_INLINE) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->head = 0xfffffffe;
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->size = lfs_tag_size(entry.tag);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
// don't always read (may be new file)
if (file->size > 0) {
err = lfs_bd_read(lfs, entry.u.d.block, entry.u.d.off,
file->cache.buffer, file->size);
if (err) {
lfs_free(file->cache.buffer);
return err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
} else {
// use ctz list from entry
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
err = lfs_bd_read(lfs, entry.u.d.block, entry.u.d.off,
&file->head, 2*sizeof(uint32_t));
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
}
// truncate if requested
if (flags & LFS_O_TRUNC) {
if (file->size != 0) {
file->flags |= LFS_F_DIRTY;
}
file->head = 0xfffffffe;
file->size = 0;
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// add to list of files
file->next = lfs->files;
lfs->files = file;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_sync(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// remove from list of files
for (lfs_file_t **p = &lfs->files; *p; p = &(*p)->next) {
if (*p == file) {
*p = file->next;
break;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
// clean up memory
if (!lfs->cfg->file_buffer) {
lfs_free(file->cache.buffer);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
return err;
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
relocate:;
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
Fixed missing erase during file relocation This was an easy fix, but highlighted the fact that the current testing framework doesn't detect when a block is written to without an associated erase. Added a quick solution that creates an empty file during an erase.
2017-06-25 21:56:12 +00:00
return err;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
err = lfs_bd_erase(lfs, nblock);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
err = lfs_cache_read(lfs, &lfs->rcache, &file->cache,
file->block, i, &data, 1);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
err = lfs_cache_prog(lfs, &lfs->pcache, &lfs->rcache,
nblock, i, &data, 1);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
// copy over new state of file
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->prog_size);
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
lfs->pcache.block = 0xffffffff;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
file->block = nblock;
return 0;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
if (file->flags & LFS_F_READING) {
file->flags &= ~LFS_F_READING;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Added tests for resizable entries and custom attributes Also found some bugs. Should now have a good amount of confidence in these features.
2018-04-11 00:55:17 +00:00
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
.head = file->head,
.size = file->size,
.flags = LFS_O_RDONLY,
.pos = file->pos,
.cache = lfs->rcache,
};
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
lfs->rcache.block = 0xffffffff;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
while (file->pos < file->size) {
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
if (lfs->rcache.block != 0xffffffff) {
orig.cache.block = 0xffffffff;
lfs->rcache.block = 0xffffffff;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
int err = lfs_cache_flush(lfs, &file->cache, &lfs->rcache);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
LFS_DEBUG("Bad block at %d", file->block);
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->size = lfs_max(file->pos, file->size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->head = file->block;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_flush(lfs, file);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
if ((file->flags & LFS_F_DIRTY) &&
!(file->flags & LFS_F_ERRED) &&
!lfs_pairisnull(file->pair)) {
// update dir entry
// TODO keep list of dirs including these guys for no
// need of another reload?
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t cwd;
err = lfs_dir_fetch(lfs, &cwd, file->pair);
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// either update the references or inline the whole file
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
int err = lfs_dir_commit_(lfs, &cwd, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_REG | LFS_STRUCT_CTZ, file->id,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
2*sizeof(uint32_t)), .u.buffer=&file->head},
file->attrs});
if (err) {
return err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
} else {
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
int err = lfs_dir_commit_(lfs, &cwd, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_REG | LFS_STRUCT_INLINE, file->id,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->size), .u.buffer=file->cache.buffer},
file->attrs});
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_DIRTY;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
lfs_size_t nsize = size;
if ((file->flags & 3) == LFS_O_WRONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (file->pos >= file->size) {
// eof if past end
return 0;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
size = lfs_min(size, file->size - file->pos);
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos, &file->block, &file->off);
if (err) {
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
int err = lfs_cache_read(lfs, &file->cache, NULL,
file->block, file->off, data, diff);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
if ((file->flags & LFS_O_APPEND) && file->pos < file->size) {
file->pos = file->size;
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->size) {
// fill with zeros
lfs_off_t pos = file->pos;
file->pos = file->size;
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
file->pos + nsize >= lfs->cfg->inline_size) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->block = 0xfffffffe;
file->off = file->pos;
lfs_alloc_ack(lfs);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
int err = lfs_file_relocate(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
}
// mark cache as dirty since we may have read data into it
file->cache.block = 0xffffffff;
}
// extend file with new blocks
lfs_alloc_ack(lfs);
int err = lfs_ctz_extend(lfs, &lfs->rcache, &file->cache,
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_prog(lfs, &file->cache, &lfs->rcache,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// update pos
if (whence == LFS_SEEK_SET) {
file->pos = off;
} else if (whence == LFS_SEEK_CUR) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->pos) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->pos + off;
} else if (whence == LFS_SEEK_END) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->size + off;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Fixed incorrect return value from lfs_file_seek lfs_file_seek returned the _previous_ file offset on success, where most standards return the _calculated_ offset on success. This just falls into me not noticing a mistake, and shows why it's always helpful to have a second set of eyes on code.
2017-09-27 00:50:39 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
// lookup new head in ctz skip list
err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
size, &file->head, &(lfs_off_t){0});
if (err) {
return err;
}
file->size = size;
file->flags |= LFS_F_DIRTY;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs_off_t pos = file->pos;
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
int err = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
Fixed error check when truncating files to larger size
2018-02-04 19:48:44 +00:00
if (err < 0) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
// restore pos
int err = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (err < 0) {
return err;
}
}
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
return res;
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
return lfs_max(file->pos, file->size);
} else {
return file->size;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
//int lfs_file_getattrs(lfs_t *lfs, lfs_file_t *file,
// const struct lfs_attr *attrs, int count) {
// // set to null in case we can't find the attrs (missing file?)
// for (int j = 0; j < count; j++) {
// memset(attrs[j].buffer, 0, attrs[j].size);
// }
//
// // load from disk if we haven't already been deleted
// if (!lfs_pairisnull(file->pair)) {
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, file->pair);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry = {.off = file->pairoff};
// err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// err = lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
// if (err) {
// return err;
// }
// }
//
// // override an attrs we have stored locally
// for (int i = 0; i < file->attrcount; i++) {
// for (int j = 0; j < count; j++) {
// if (attrs[j].type == file->attrs[i].type) {
// if (attrs[j].size < file->attrs[i].size) {
// return LFS_ERR_RANGE;
// }
//
// memset(attrs[j].buffer, 0, attrs[j].size);
// memcpy(attrs[j].buffer,
// file->attrs[i].buffer, file->attrs[i].size);
// }
// }
// }
//
// return 0;
//}
//int lfs_file_setattrs(lfs_t *lfs, lfs_file_t *file,
// const struct lfs_attr *attrs, int count) {
// if ((file->flags & 3) == LFS_O_RDONLY) {
// return LFS_ERR_BADF;
// }
//
// // at least make sure attributes fit
// if (!lfs_pairisnull(file->pair)) {
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, file->pair);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry = {.off = file->pairoff};
// err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// lfs_ssize_t res = lfs_dir_checkattrs(lfs, &cwd, &entry, attrs, count);
// if (res < 0) {
// return res;
// }
// }
//
// // just tack to the file, will be written at sync time
// file->attrs = attrs;
// file->attrcount = count;
// file->flags |= LFS_F_DIRTY;
//
// return 0;
//}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t cwd;
int16_t id;
int err = lfs_dir_find(lfs, &cwd, &path, &id);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
return lfs_dir_getinfo(lfs, &cwd, id, info);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
int lfs_remove(lfs_t *lfs, const char *path) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
int16_t id;
err = lfs_dir_find(lfs, &cwd, &path, &id);
if (err) {
return err;
}
// grab entry to see if we're dealing with a dir
lfs_entry_t entry;
err = lfs_dir_getentry(lfs, &cwd, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, id, 0), &entry);
if (err) {
return err;
}
if (lfs_tag_subtype(entry.tag) == LFS_TYPE_DIR) {
lfs_dir_t dir;
// must be empty before removal
err = lfs_dir_fetch(lfs, &dir, entry.u.pair);
if (err) {
return err;
}
if (dir.count > 0 || dir.split) {
return LFS_ERR_NOTEMPTY;
}
}
// delete the entry
err = lfs_dir_delete(lfs, &cwd, id);
if (err) {
return err;
}
// if we were a directory, find pred, replace tail
// TODO can this just deorphan?
if (lfs_tag_subtype(entry.tag) == LFS_TYPE_DIR) {
err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
// if (lfs_tag_subtype(entry.tag) == LFS_TYPE_DIR) {
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
// int res = lfs_pred(lfs, dir.pair, &cwd);
// if (res < 0) {
// return res;
// }
//
// LFS_ASSERT(res); // must have pred
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// cwd.tail[0] = dir.tail[0];
// cwd.tail[1] = dir.tail[1];
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//
// err = lfs_dir_commit(lfs, &cwd, NULL, 0);
// if (err) {
// return err;
// }
// }
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
// find old entry
lfs_dir_t oldcwd;
int16_t oldid;
int err = lfs_dir_find(lfs, &oldcwd, &oldpath, &oldid);
if (err) {
return err;
}
lfs_entry_t oldentry;
err = lfs_dir_getentry(lfs, &oldcwd, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, oldid, 0), &oldentry);
if (err) {
return err;
}
// find new entry
lfs_dir_t newcwd;
int16_t newid;
err = lfs_dir_find(lfs, &newcwd, &newpath, &newid);
if (err && err != LFS_ERR_NOENT) {
return err;
}
bool prevexists = (err != LFS_ERR_NOENT);
bool samepair = (lfs_paircmp(oldcwd.pair, newcwd.pair) == 0);
lfs_entry_t preventry;
if (prevexists) {
// get prev entry, check that we have same type
err = lfs_dir_getentry(lfs, &newcwd, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, newid, 0), &preventry);
if (err) {
return err;
}
if (lfs_tag_subtype(preventry.tag) != lfs_tag_subtype(oldentry.tag)) {
return LFS_ERR_ISDIR;
}
if (lfs_tag_subtype(preventry.tag) == LFS_TYPE_DIR) {
lfs_dir_t prevdir;
// must be empty before removal
err = lfs_dir_fetch(lfs, &prevdir, preventry.u.pair);
if (err) {
return err;
}
if (prevdir.count > 0 || prevdir.split) {
return LFS_ERR_NOTEMPTY;
}
}
} else {
// check that name fits
lfs_size_t nlen = strlen(newpath);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
// get next id
err = lfs_dir_append(lfs, &newcwd, &newid);
if (err) {
return err;
}
}
// mark as moving
//printf("RENAME MOVE %d %d %d\n", oldcwd.pair[0], oldcwd.pair[1], oldid);
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &oldcwd, &(lfs_entrylist_t){
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
{lfs_mktag(LFS_TYPE_MOVE, oldid, 0)}});
if (err) {
return err;
}
if (samepair) {
// update pair if newcwd == oldcwd
newcwd = oldcwd;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
// TODO check that all complaints are fixed
// // move to new location
// // TODO NAME?????
// // TODO HAH, move doesn't want to override things (due
// // to its use in compaction), but that's _exactly_ what we want here
// err = lfs_dir_commitwith(lfs, &newcwd, lfs_commit_move,
// &(struct lfs_commit_move){.dir=&oldcwd, .id={oldid, newid}});
// if (err) {
// return err;
// }
// // TODO NONONONONO
// // TODO also don't call strlen twice (see prev name check)
// err = lfs_dir_commit_(lfs, &newcwd, &(lfs_entrylist_t){
// {lfs_mktag(LFS_TYPE_NAME, newid, strlen(newpath)),
// .u.buffer=(void*)newpath}});
// if (err) {
// return err;
// }
err = lfs_dir_commit_(lfs, &newcwd, &(lfs_entrylist_t){
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
{lfs_mktag(LFS_TYPE_NAME, newid, strlen(newpath)),
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
.u.buffer=(void*)newpath}, &(lfs_entrylist_t){
{lfs_mktag(LFS_FROM_MOVE, newid, oldid),
.u.dir=&oldcwd}}});
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (err) {
return err;
}
if (samepair) {
// update pair if newcwd == oldcwd
oldcwd = newcwd;
}
// remove old entry
//printf("RENAME DELETE %d %d %d\n", oldcwd.pair[0], oldcwd.pair[1], oldid);
err = lfs_dir_delete(lfs, &oldcwd, oldid);
if (err) {
return err;
}
// if we were a directory, find pred, replace tail
// TODO can this just deorphan?
if (prevexists && lfs_tag_subtype(preventry.tag) == LFS_TYPE_DIR) {
err = lfs_deorphan(lfs);
if (err) {
return err;
}
}
return 0;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_getattrs(lfs_t *lfs, const char *path,
// const struct lfs_attr *attrs, int count) {
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry;
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
//}
//
//int lfs_setattrs(lfs_t *lfs, const char *path,
// const struct lfs_attr *attrs, int count) {
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry;
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_setattrs(lfs, &cwd, &entry, attrs, count);
//}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->rcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->pcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
// setup lookahead, round down to nearest 32-bits
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->lookahead % 32 == 0);
LFS_ASSERT(lfs->cfg->lookahead > 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead/8);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
// check that program and read sizes are multiples of the block size
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->prog_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->prog_size == 0);
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
// check that the block size is large enough to fit ctz pointers
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
<= lfs->cfg->block_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
LFS_ASSERT(lfs->cfg->inline_size <= LFS_INLINE_MAX);
LFS_ASSERT(lfs->cfg->inline_size <= lfs->cfg->read_size);
lfs->inline_size = lfs->cfg->inline_size;
if (!lfs->inline_size) {
lfs->inline_size = lfs_min(LFS_INLINE_MAX, lfs->cfg->read_size);
}
LFS_ASSERT(lfs->cfg->attrs_size <= LFS_ATTRS_MAX);
lfs->attrs_size = lfs->cfg->attrs_size;
if (!lfs->attrs_size) {
lfs->attrs_size = LFS_ATTRS_MAX;
}
LFS_ASSERT(lfs->cfg->name_size <= LFS_NAME_MAX);
lfs->name_size = lfs->cfg->name_size;
if (!lfs->name_size) {
lfs->name_size = LFS_NAME_MAX;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs->root[0] = 0xffffffff;
lfs->root[1] = 0xffffffff;
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
lfs->files = NULL;
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
lfs->dirs = NULL;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = false;
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create free lookahead
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->cfg->block_count);
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create superblock dir
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t dir;
err = lfs_dir_alloc(lfs, &dir, false,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
(const lfs_block_t[2]){0xffffffff, 0xffffffff});
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// write root directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t root;
err = lfs_dir_alloc(lfs, &root, false,
More progress integrating journaling - Integrated into lfs_file_t_, duplicating functions where necessary - Added lfs_dir_fetchwith_ as common parent to both lfs_dir_fetch_ and lfs_dir_find_ - Added similar parent with lfs_dir_commitwith_ - Made matching find/get operations with getbuffer/getentry and findbuffer/findentry - lfs_dir_alloc now populates tail, since almost all directory block allocations need to populate tail
2018-05-22 22:43:39 +00:00
(const lfs_block_t[2]){0xffffffff, 0xffffffff});
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &root, NULL);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
lfs->root[0] = root.pair[0];
lfs->root[1] = root.pair[1];
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
dir.tail[0] = lfs->root[0];
dir.tail[1] = lfs->root[1];
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// write one superblock
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock = {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
.root[0] = lfs->root[0],
.root[1] = lfs->root[1],
.magic = {"littlefs"},
.version = LFS_DISK_VERSION,
.block_size = lfs->cfg->block_size,
.block_count = lfs->cfg->block_count,
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
.inline_size = lfs->inline_size,
.attrs_size = lfs->attrs_size,
.name_size = lfs->name_size,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
};
dir.count += 1;
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &dir, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_SUPERBLOCK | LFS_STRUCT_DIR, 0,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
sizeof(superblock)), .u.buffer=&superblock}});
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// sanity check that fetch works
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
return lfs_deinit(lfs);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup free lookahead
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = 0;
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// load superblock
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t dir;
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_superblock_t superblock;
err = lfs_dir_getbuffer(lfs, &dir, 0x7ffff000, &(lfs_entry_t){
lfs_mktag(LFS_TYPE_SUPERBLOCK | LFS_STRUCT_DIR,
0, sizeof(superblock)),
.u.buffer=&superblock});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err && err != LFS_ERR_RANGE) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return err;
}
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (memcmp(superblock.magic, "littlefs", 8) != 0) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_CORRUPT;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
uint16_t major_version = (0xffff & (superblock.version >> 16));
uint16_t minor_version = (0xffff & (superblock.version >> 0));
Add version info for software library and on-disk structures An annoying part of filesystems is that the software library can change independently of the on-disk structures. For this reason versioning is very important, and must be handled separately for the software and on-disk parts. In this patch, littlefs provides two version numbers at compile time, with major and minor parts, in the form of 6 macros. LFS_VERSION // Library version, uint32_t encoded LFS_VERSION_MAJOR // Major - Backwards incompatible changes LFS_VERSION_MINOR // Minor - Feature additions LFS_DISK_VERSION // On-disk version, uint32_t encoded LFS_DISK_VERSION_MAJOR // Major - Backwards incompatible changes LFS_DISK_VERSION_MINOR // Minor - Feature additions Note that littlefs will error if it finds a major version number that is different, or a minor version number that has regressed.
2018-01-26 20:26:25 +00:00
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
LFS_ERROR("Invalid version %d.%d", major_version, minor_version);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_INVAL;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.inline_size) {
if (superblock.inline_size > lfs->inline_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported inline size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.inline_size, lfs->inline_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->inline_size = superblock.inline_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.attrs_size) {
if (superblock.attrs_size > lfs->attrs_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported attrs size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.attrs_size, lfs->attrs_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->attrs_size = superblock.attrs_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (superblock.name_size) {
if (superblock.name_size > lfs->name_size) {
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
LFS_ERROR("Unsupported name size (%d > %d)",
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
superblock.name_size, lfs->name_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
return LFS_ERR_INVAL;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->name_size = superblock.name_size;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->root[0] = superblock.root[0];
lfs->root[1] = superblock.root[1];
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return lfs_deinit(lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// Internal filesystem filesystem operations ///
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_fs_traverse(lfs_t *lfs,
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
int (*cb)(lfs_t *lfs, void *data, lfs_block_t block), void *data) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// iterate over metadata pairs
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
while (!lfs_pairisnull(dir.tail)) {
for (int i = 0; i < 2; i++) {
int err = cb(lfs, data, dir.tail[i]);
if (err) {
return err;
}
}
// iterate through ids in directory
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < dir.count; id++) {
lfs_entry_t entry;
int err = lfs_dir_getentry(lfs, &dir, 0x701ff000,
lfs_mktag(LFS_TYPE_REG, id, 0), &entry);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
if (err == LFS_ERR_NOENT) {
continue;
}
return err;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
if (lfs_tag_struct(entry.tag) == LFS_STRUCT_CTZ) {
err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
entry.u.ctz.head, entry.u.ctz.size, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
}
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->head, f->size, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
f->block, f->pos, cb, data);
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
// iterate over metadata pairs
lfs_block_t cwd[2] = {0, 1};
while (true) {
for (int i = 0; i < 2; i++) {
int err = cb(data, cwd[i]);
if (err) {
return err;
}
}
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
lfs_dir_t dir;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
int err = lfs_dir_fetch(lfs, &dir, cwd);
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
// iterate over contents
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
lfs_entry_t entry;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
while (dir.off + sizeof(entry.d) <= (0x7fffffff & dir.d.size)-4) {
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
err = lfs_dir_get(lfs, &dir,
dir.off, &entry.d, sizeof(entry.d));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_entry_fromle32(&entry.d);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
dir.off += lfs_entry_size(&entry);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0x70 & entry.d.type) == LFS_STRUCT_CTZ) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
entry.d.u.file.head, entry.d.u.file.size, cb, data);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (err) {
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
}
cwd[0] = dir.d.tail[0];
cwd[1] = dir.d.tail[1];
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (lfs_pairisnull(cwd)) {
break;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->head, f->size, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->block, f->pos, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
static int lfs_pred(lfs_t *lfs, const lfs_block_t pair[2], lfs_dir_t *pdir) {
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
pdir->tail[0] = 0;
pdir->tail[1] = 1;
while (!lfs_pairisnull(pdir->tail)) {
if (lfs_paircmp(pdir->tail, pair) == 0) {
Passing all of the basic functionality tests Integration with the new journaling metadata has now progressed to the point where all of the basic functionality tests are passing. This includes: - test_format - test_dirs - test_files - test_seek - test_truncate - test_interspersed - test_paths Some of the fixes: - Modified move to correctly change entry ids - Called lfs_commit_move directly from compact, avoiding commit parsing logic during a compact - Opened up commit filters to be passed down from compact for moves - Added correct drop logic to lfs_dir_delete - Updated lfs_dir_seek to use ids instead of offsets - Caught id updates manually where possible (this needs to be fixed)
2018-05-28 14:17:44 +00:00
return true; // TODO should we return true only if pred is part of dir?
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, pdir, pdir->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_dir_t *pdir) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate directories
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int err = lfs_dir_fetch(lfs, pdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (!lfs_pairisnull(pdir->d.tail)) {
if (lfs_paircmp(pdir->d.tail, dir) == 0) {
return true;
}
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, pdir, pdir->d.tail);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
static int lfs_parent(lfs_t *lfs, const lfs_block_t pair[2],
lfs_dir_t *parent, lfs_entry_t *entry) {
// iterate over all directory directory entries
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
parent->tail[0] = 0;
parent->tail[1] = 1;
while (!lfs_pairisnull(parent->tail)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, parent, parent->tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
// TODO make this O(n) by using fetchwith to match the pointers
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
for (uint16_t id = 0; id < parent->count; id++) {
int err = lfs_dir_getentry(lfs, parent, 0x43dff000,
lfs_mktag(LFS_STRUCT_DIR, id, 0), entry);
if (err) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err == LFS_ERR_NOENT) {
continue;
}
return err;
}
if (lfs_paircmp(entry->u.pair, pair) == 0) {
return true;
}
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_dir_t *parent, lfs_entry_t *entry) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
parent->d.tail[0] = 0;
parent->d.tail[1] = 1;
// iterate over all directory directory entries
while (!lfs_pairisnull(parent->d.tail)) {
int err = lfs_dir_fetch(lfs, parent, parent->d.tail);
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, parent, entry);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err && err != LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err == LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
break;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (((0x70 & entry->d.type) == LFS_STRUCT_DIR) &&
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_paircmp(entry->d.u.dir, dir) == 0) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return true;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
static int lfs_moved(lfs_t *lfs, lfs_dir_t *fromdir, uint16_t fromid) {
// grab entry pair we're looking for
fromdir->moveid = -1;
lfs_entry_t fromentry;
int err = lfs_dir_getentry(lfs, fromdir, 0x43dff000,
lfs_mktag(LFS_STRUCT_DIR, fromid, 0), &fromentry);
fromdir->moveid = fromid;
if (err) {
return err;
}
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// skip superblock
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs_dir_t todir;
err = lfs_dir_fetch(lfs, &todir, (const lfs_block_t[2]){0, 1});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// iterate over all directory directory entries
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
while (!lfs_pairisnull(todir.tail)) {
int err = lfs_dir_fetch(lfs, &todir, todir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
for (int toid = 0; toid < todir.count; toid++) {
if (lfs_paircmp(todir.pair, fromdir->pair) == 0 &&
toid == fromid) {
continue;
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
lfs_entry_t toentry;
int err = lfs_dir_getentry(lfs, &todir, 0x43dff000,
lfs_mktag(LFS_STRUCT_DIR, toid, 0), &toentry);
if (err) {
if (err == LFS_ERR_NOENT) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
continue;
}
return err;
}
More testing progress, combined dir/commit traversal Passing more tests now with the journalling change, but still have more work to do. The most humorous bug was a bug where during the three step move process, the entry move logic would dumbly copy over any tags associated with the moving entry, including the tag used to temporarily mark the entry as "moving". Also combined dir and commit traversal using a "stop_at_commit" flag in directory struct as a short-term hack to combine the code paths.
2018-05-27 15:15:28 +00:00
if (lfs_paircmp(toentry.u.pair, fromentry.u.pair) == 0) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
return true;
}
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
static int lfs_moved(lfs_t *lfs, const void *e) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// skip superblock
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
// iterate over all directory directory entries
lfs_entry_t entry;
while (!lfs_pairisnull(cwd.d.tail)) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (!(LFS_STRUCT_MOVED & entry.d.type) &&
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
memcmp(&entry.d.u, e, sizeof(entry.d.u)) == 0) {
return true;
}
}
}
return false;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]) {
// find parent
lfs_dir_t parent;
lfs_entry_t entry;
int res = lfs_parent(lfs, oldpair, &parent, &entry);
if (res < 0) {
return res;
}
if (res) {
// update disk, this creates a desync
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
entry.u.pair[0] = newpair[0];
entry.u.pair[1] = newpair[1];
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
int err = lfs_dir_commit_(lfs, &parent, &(lfs_entrylist_t){entry});
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update internal root
if (lfs_paircmp(oldpair, lfs->root) == 0) {
LFS_DEBUG("Relocating root %d %d", newpair[0], newpair[1]);
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
// clean up bad block, which should now be a desync
return lfs_deorphan(lfs);
}
// find pred
res = lfs_pred(lfs, oldpair, &parent);
if (res < 0) {
return res;
}
if (res) {
// just replace bad pair, no desync can occur
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
parent.tail[0] = newpair[0];
parent.tail[1] = newpair[1];
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
return lfs_dir_commit_(lfs, &parent, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL + parent.split*0x10, // TODO hm
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
0x1ff, sizeof(lfs_block_t[2])),
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
.u.pair[0]=newpair[0], .u.pair[1]=newpair[1]}});
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// couldn't find dir, must be new
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int lfs_deorphan(lfs_t *lfs) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
lfs->deorphaned = true;
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t pdir = {.split = true};
lfs_dir_t dir = {.tail = {0, 1}};
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(dir.tail)) {
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_dir_fetch(lfs, &dir, dir.tail);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
// check head blocks for orphans
if (!pdir.split) {
// check if we have a parent
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
lfs_dir_t parent;
lfs_entry_t entry;
int res = lfs_parent(lfs, pdir.tail, &parent, &entry);
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (res < 0) {
return res;
}
if (!res) {
// we are an orphan
LFS_DEBUG("Found orphan %d %d",
pdir.tail[0], pdir.tail[1]);
pdir.tail[0] = dir.tail[0];
pdir.tail[1] = dir.tail[1];
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &pdir, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x1ff,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
sizeof(pdir.tail)), .u.buffer=pdir.tail}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
break;
}
if (!lfs_pairsync(entry.u.pair, pdir.tail)) {
// we have desynced
LFS_DEBUG("Found desync %d %d",
entry.u.pair[0], entry.u.pair[1]);
pdir.tail[0] = entry.u.pair[0];
pdir.tail[1] = entry.u.pair[1];
Continued progress with reintroducing testing on the new metadata logging Now with some tweaks to commit/compact, and a committers for entrylists and moves specifically. No longer relying on a commitwith callback, the types of commits are now infered from their tags. This means we can now commit things atomically with special commits, such as moves. Now lfs_rename can move entries to new names correctly.
2018-05-28 07:08:16 +00:00
err = lfs_dir_commit_(lfs, &pdir, &(lfs_entrylist_t){
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
{lfs_mktag(LFS_TYPE_SOFTTAIL, 0x1ff,
Completed transition of files with journalling metadata This was the simpler part of transitioning since file operations only interact with metadata at sync time. Also switched from array to linked-list of entries.
2018-05-23 04:57:19 +00:00
sizeof(pdir.tail)), .u.buffer=pdir.tail}});
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
if (err) {
return err;
}
break;
}
}
// check entries for moves
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
if (dir.moveid >= 0) {
Progressed integration of journaling metadata pairs - Integrated journaling into lfs_dir_t_ struct and operations, duplicating functions where necessary - Added internal lfs_tag_t and lfs_stag_t - Consolidated lfs_region and lfs_entry structures
2018-05-21 05:56:20 +00:00
// TODO moves and stuff
// TODO need to load entry to find it
// // found moved entry
// int moved = lfs_moved(lfs, &entry.u);
// if (moved < 0) {
// return moved;
// }
//
// if (moved) {
// LFS_DEBUG("Found move %d %d",
// entry.d.u.dir[0], entry.d.u.dir[1]);
// err = lfs_dir_set(lfs, &dir, &entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
// if (err) {
// return err;
// }
// } else {
// LFS_DEBUG("Found partial move %d %d",
// entry.d.u.dir[0], entry.d.u.dir[1]);
// entry.d.type &= ~LFS_STRUCT_MOVED;
// err = lfs_dir_set(lfs, &dir, &entry, (struct lfs_region[]){
// {LFS_FROM_MEM, 0, 1, &entry.d, 1}}, 1);
// if (err) {
// return err;
// }
// }
}
memcpy(&pdir, &dir, sizeof(pdir));
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
/*
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
int lfs_deorphan(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = true;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
lfs_dir_t pdir = {.d.size = 0x80000000};
lfs_dir_t cwd = {.d.tail[0] = 0, .d.tail[1] = 1};
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(cwd.d.tail)) {
int err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check head blocks for orphans
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (!(0x80000000 & pdir.d.size)) {
// check if we have a parent
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_dir_t parent;
lfs_entry_t entry;
int res = lfs_parent(lfs, pdir.d.tail, &parent, &entry);
if (res < 0) {
return res;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (!res) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// we are an orphan
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found orphan %d %d",
pdir.d.tail[0], pdir.d.tail[1]);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
pdir.d.tail[0] = cwd.d.tail[0];
pdir.d.tail[1] = cwd.d.tail[1];
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
break;
}
if (!lfs_pairsync(entry.d.u.dir, pdir.d.tail)) {
// we have desynced
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found desync %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
pdir.d.tail[0] = entry.d.u.dir[0];
pdir.d.tail[1] = entry.d.u.dir[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
break;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check entries for moves
lfs_entry_t entry;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
// found moved entry
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (entry.d.type & LFS_STRUCT_MOVED) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int moved = lfs_moved(lfs, &entry.d.u);
if (moved < 0) {
return moved;
}
if (moved) {
LFS_DEBUG("Found move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
} else {
LFS_DEBUG("Found partial move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry.d.type &= ~LFS_STRUCT_MOVED;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
{LFS_FROM_MEM, 0, 1, &entry.d, 1}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
}
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
memcpy(&pdir, &cwd, sizeof(pdir));
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
}
return 0;
}
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
*/
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// External filesystem filesystem operations ///
Reorganized the internal operations to make more sense Also refactored lfs_dir_compact a bit, adding begin and end as arguments since they simplify a bit of the logic and can be found out much easier earlier in the commit logic. Also changed add -> append and drop -> delete and cleaned up some of the logic around there.
2018-05-26 00:04:01 +00:00
//int lfs_fs_getattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
// lfs_dir_t dir;
// int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
// if (err) {
// return err;
// }
//
// lfs_entry_t entry = {.off = sizeof(dir.d)};
// err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// if (err != LFS_ERR_NOENT) {
// if (!err) {
// break;
// }
// return err;
// }
//
// lfs_dir_t cwd;
// int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
// if (err) {
// return err;
// }
//
// lfs_entry_t entry;
// err = lfs_dir_find(lfs, &cwd, &entry, &path);
// if (err) {
// return err;
// }
//
// return lfs_dir_getinfo(lfs, &cwd, &entry, info);
// return lfs_dir_getattrs(lfs, &dir, &entry, attrs, count);
//}
//
//int lfs_fs_setattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
// lfs_dir_t dir;
// int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
// if (err) {
// return err;
// }
//
// lfs_entry_t entry = {.off = sizeof(dir.d)};
// err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
// if (err) {
// return err;
// }
// entry.size = lfs_entry_size(&entry);
//
// return lfs_dir_setattrs(lfs, &dir, &entry, attrs, count);
//}
Added lfs_fs_size for finding a count of used blocks This has existed for some time in the form of the lfs_traverse function, through which a user could provide a simple callback that would just count the number of blocks lfs_traverse finds. However, this approach is relatively unconventional and has proven to be confusing for most users.
2018-04-09 03:25:58 +00:00
static int lfs_fs_size_count(void *p, lfs_block_t block) {
lfs_size_t *size = p;
*size += 1;
return 0;
}
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
lfs_size_t size = 0;
Cleaned up enough things to pass basic file testing
2018-05-26 18:50:06 +00:00
int err = lfs_fs_traverse(lfs, lfs_fs_size_count, &size);
Added lfs_fs_size for finding a count of used blocks This has existed for some time in the form of the lfs_traverse function, through which a user could provide a simple callback that would just count the number of blocks lfs_traverse finds. However, this approach is relatively unconventional and has proven to be confusing for most users.
2018-04-09 03:25:58 +00:00
if (err) {
return err;
}
return size;
}
Reference in New Issue Copy Permalink