openharmony/third_party_littlefs
1
0
Fork 0
mirror of https://gitee.com/openharmony/third_party_littlefs synced 2024-11-23 14:59:50 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
ea4ded420c
third_party_littlefs/lfs.c

3203 lines
88 KiB
C
Raw Normal View History

Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
/*
* The little filesystem
*
Updated copyright Due to employee contract Per ARM license remains under Apache 2.0
2017-10-13 01:27:33 +00:00
* Copyright (c) 2017 ARM Limited
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
*/
#include "lfs.h"
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
#include "lfs_util.h"
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// Caching block device operations ///
static int lfs_cache_read(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (pcache && block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(data, &pcache->buffer[off-pcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
if (block == rcache->block && off >= rcache->off &&
off < rcache->off + lfs->cfg->read_size) {
// is already in rcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs->cfg->read_size - (off-rcache->off));
memcpy(data, &rcache->buffer[off-rcache->off], diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->read_size == 0 && size >= lfs->cfg->read_size) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
// bypass cache?
lfs_size_t diff = size - (size % lfs->cfg->read_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->read(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
data += diff;
off += diff;
size -= diff;
continue;
}
// load to cache, first condition can no longer fail
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
rcache->block = block;
rcache->off = off - (off % lfs->cfg->read_size);
int err = lfs->cfg->read(lfs->cfg, rcache->block,
rcache->off, rcache->buffer, lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_cmp(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
if (c != data[i]) {
return false;
}
}
return true;
}
static int lfs_cache_crc(lfs_t *lfs, lfs_cache_t *rcache,
const lfs_cache_t *pcache, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t c;
int err = lfs_cache_read(lfs, rcache, pcache,
block, off+i, &c, 1);
if (err) {
return err;
}
lfs_crc(crc, &c, 1);
}
return 0;
}
static int lfs_cache_flush(lfs_t *lfs,
lfs_cache_t *pcache, lfs_cache_t *rcache) {
if (pcache->block != 0xffffffff) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(pcache->block < lfs->cfg->block_count);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs->cfg->prog(lfs->cfg, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL, pcache->block,
pcache->off, pcache->buffer, lfs->cfg->prog_size);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
pcache->block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
return 0;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_cache_prog(lfs_t *lfs, lfs_cache_t *pcache,
lfs_cache_t *rcache, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block != 0xffffffff);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
LFS_ASSERT(off + size <= lfs->cfg->block_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
while (size > 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (block == pcache->block && off >= pcache->off &&
off < pcache->off + lfs->cfg->prog_size) {
// is already in pcache?
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = lfs_min(size,
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
lfs->cfg->prog_size - (off-pcache->off));
memcpy(&pcache->buffer[off-pcache->off], data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (off % lfs->cfg->prog_size == 0) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// eagerly flush out pcache if we fill up
int err = lfs_cache_flush(lfs, pcache, rcache);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (err) {
return err;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// pcache must have been flushed, either by programming and
// entire block or manually flushing the pcache
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(pcache->block == 0xffffffff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (off % lfs->cfg->prog_size == 0 &&
size >= lfs->cfg->prog_size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// bypass pcache?
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
lfs_size_t diff = size - (size % lfs->cfg->prog_size);
Adopted more conventional buffer parameter ordering Adopted buffer followed by size. The other order was original chosen due to some other functions with a more complicated parameter list. This convention is important, as the bd api is one of the main apis facing porting efforts.
2017-04-24 04:49:21 +00:00
int err = lfs->cfg->prog(lfs->cfg, block, off, data, diff);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (rcache) {
int res = lfs_cache_cmp(lfs, rcache, NULL,
block, off, data, diff);
if (res < 0) {
return res;
}
if (!res) {
return LFS_ERR_CORRUPT;
}
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
data += diff;
off += diff;
size -= diff;
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
// prepare pcache, first condition can no longer fail
pcache->block = block;
pcache->off = off - (off % lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// General lfs block device operations ///
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_read(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, void *buffer, lfs_size_t size) {
// if we ever do more than writes to alternating pairs,
// this may need to consider pcache
return lfs_cache_read(lfs, &lfs->rcache, NULL,
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_prog(lfs_t *lfs, lfs_block_t block,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_off_t off, const void *buffer, lfs_size_t size) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
return lfs_cache_prog(lfs, &lfs->pcache, NULL,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
block, off, buffer, size);
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_cmp(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, const void *buffer, lfs_size_t size) {
return lfs_cache_cmp(lfs, &lfs->rcache, NULL, block, off, buffer, size);
}
static int lfs_bd_crc(lfs_t *lfs, lfs_block_t block,
lfs_off_t off, lfs_size_t size, uint32_t *crc) {
return lfs_cache_crc(lfs, &lfs->rcache, NULL, block, off, size, crc);
}
static int lfs_bd_erase(lfs_t *lfs, lfs_block_t block) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT(block < lfs->cfg->block_count);
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->erase(lfs->cfg, block);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
static int lfs_bd_sync(lfs_t *lfs) {
Fixed dirty rcache during directory commit An interesting side-effect of adding internal checks to the littlefs for block errors, is that the littlefs starts to cover up its own flaws. Probably out of embarrassment. In this case, the relocation logic for directories left the littlefs rcache dirty with invalid data. The littlefs detected the error, treated it as a corrupted write, and just moved the "corrupted" block to a new block, which as a side-effect flushes the rcache. Since committing a dir will end up flushing the rcache to check for errors anyways, we can just drop the rcache in lfs_bd_sync.
2017-06-25 21:21:14 +00:00
lfs->rcache.block = 0xffffffff;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_flush(lfs, &lfs->pcache, NULL);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (err) {
return err;
}
Simplified config Before, the lfs had multiple paths to determine config options: - lfs_config struct passed during initialization - lfs_bd_info struct passed during block device initialization - compile time options This allowed different developers to provide their own needs to the filesystem, such as the block device capabilities and the higher level user's own tweaks. However, this comes with additional complexity and action required when the configurations are incompatible. For now, this has been reduced to all information (including block device function pointers) being passed through the lfs_config struct. We just defer more complicated handling of configuration options to the top level user. This simplifies configuration handling and gives the top level user the responsibility to handle configuration, which they probably would have wanted to do anyways.
2017-04-22 16:42:05 +00:00
return lfs->cfg->sync(lfs->cfg);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
/// Internal operations predeclared here ///
int lfs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_dir_t *pdir);
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_dir_t *parent, lfs_entry_t *entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
static int lfs_moved(lfs_t *lfs, const void *e);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]);
int lfs_deorphan(lfs_t *lfs);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
/// Block allocator ///
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static int lfs_alloc_lookahead(void *p, lfs_block_t block) {
lfs_t *lfs = p;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs_block_t off = ((block - lfs->free.off)
Fixed issue with negative modulo with unaligned lookaheads When the lookahead buffer wraps around in an unaligned filesystem, it's possible for blocks at the beginning of the disk to have a negative distance from the lookahead, but still reside in the lookahead buffer. Switching to signed modulo doesn't quite work due to how negative modulo is implemented in C, so the simple solution is to shift the region to be positive.
2017-09-17 19:52:25 +00:00
+ lfs->cfg->block_count) % lfs->cfg->block_count;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
if (off < lfs->free.size) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer[off / 32] |= 1U << (off % 32);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
return 0;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_alloc(lfs_t *lfs, lfs_block_t *block) {
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
while (true) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size) {
lfs_block_t off = lfs->free.i;
lfs->free.i += 1;
lfs->free.ack -= 1;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!(lfs->free.buffer[off / 32] & (1U << (off % 32)))) {
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// found a free block
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
*block = (lfs->free.off + off) % lfs->cfg->block_count;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
// eagerly find next off so an alloc ack can
// discredit old lookahead blocks
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
while (lfs->free.i != lfs->free.size &&
(lfs->free.buffer[lfs->free.i / 32]
& (1U << (lfs->free.i % 32)))) {
lfs->free.i += 1;
lfs->free.ack -= 1;
Fixed issue with lookahead trusting old lookahead blocks One of the big simplifications in littlefs's implementation is the complete lack of tracking free blocks, allowing operations to simply drop blocks that are no longer in use. However, this means the lookahead buffer can easily contain outdated blocks that were previously deleted. This is usually fine, as littlefs will rescan the storage if it can't find a free block in the lookahead buffer, but after changes that caused littlefs to more conservatively respect the alloc acks (e611cf5), any scanned blocks after an ack would be incorrectly trusted. The fix is to eagerly scan ahead in the lookahead when we allocate so that alloc acks are better able to discredit old lookahead blocks. Since usually alloc acks are tightly coupled to allocations of one or two blocks, this allows littlefs to properly rescan every set of allocations. This may still be a concern if there is a long series of worn out blocks, but in the worst case littlefs will conservatively avoid using blocks it's not sure about. Found by davidefer
2018-04-06 22:00:29 +00:00
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
// check if we have looked at all blocks since last ack
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
if (lfs->free.ack == 0) {
LFS_WARN("No more free space %d", lfs->free.i + lfs->free.off);
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
return LFS_ERR_NOSPC;
}
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.off = (lfs->free.off + lfs->free.size)
% lfs->cfg->block_count;
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->free.ack);
lfs->free.i = 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// find mask of free blocks from tree
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
int err = lfs_traverse(lfs, lfs_alloc_lookahead, lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static void lfs_alloc_ack(lfs_t *lfs) {
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.ack = lfs->cfg->block_count;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
/// Endian swapping functions ///
static void lfs_dir_fromle32(struct lfs_disk_dir *d) {
d->rev = lfs_fromle32(d->rev);
d->size = lfs_fromle32(d->size);
d->tail[0] = lfs_fromle32(d->tail[0]);
d->tail[1] = lfs_fromle32(d->tail[1]);
}
static void lfs_dir_tole32(struct lfs_disk_dir *d) {
d->rev = lfs_tole32(d->rev);
d->size = lfs_tole32(d->size);
d->tail[0] = lfs_tole32(d->tail[0]);
d->tail[1] = lfs_tole32(d->tail[1]);
}
static void lfs_entry_fromle32(struct lfs_disk_entry *d) {
d->u.dir[0] = lfs_fromle32(d->u.dir[0]);
d->u.dir[1] = lfs_fromle32(d->u.dir[1]);
}
Fixed big-endian support for entry structures
2018-03-16 02:26:03 +00:00
static void lfs_entry_tole32(struct lfs_disk_entry *d) {
d->u.dir[0] = lfs_tole32(d->u.dir[0]);
d->u.dir[1] = lfs_tole32(d->u.dir[1]);
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
static void lfs_superblock_fromle32(struct lfs_disk_superblock *d) {
d->root[0] = lfs_fromle32(d->root[0]);
d->root[1] = lfs_fromle32(d->root[1]);
d->block_size = lfs_fromle32(d->block_size);
d->block_count = lfs_fromle32(d->block_count);
d->version = lfs_fromle32(d->version);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
d->inline_size = lfs_fromle32(d->inline_size);
d->attrs_size = lfs_fromle32(d->attrs_size);
d->name_size = lfs_fromle32(d->name_size);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
}
static void lfs_superblock_tole32(struct lfs_disk_superblock *d) {
d->root[0] = lfs_tole32(d->root[0]);
d->root[1] = lfs_tole32(d->root[1]);
d->block_size = lfs_tole32(d->block_size);
d->block_count = lfs_tole32(d->block_count);
d->version = lfs_tole32(d->version);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
d->inline_size = lfs_tole32(d->inline_size);
d->attrs_size = lfs_tole32(d->attrs_size);
d->name_size = lfs_tole32(d->name_size);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
/// Other struct functions ///
static inline lfs_size_t lfs_entry_elen(const lfs_entry_t *entry) {
return (lfs_size_t)(entry->d.elen) |
((lfs_size_t)(entry->d.alen & 0xc0) << 2);
}
static inline lfs_size_t lfs_entry_alen(const lfs_entry_t *entry) {
return entry->d.alen & 0x3f;
}
static inline lfs_size_t lfs_entry_nlen(const lfs_entry_t *entry) {
return entry->d.nlen;
}
static inline lfs_size_t lfs_entry_size(const lfs_entry_t *entry) {
return 4 + lfs_entry_elen(entry) +
lfs_entry_alen(entry) +
lfs_entry_nlen(entry);
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Metadata pair and directory operations ///
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline void lfs_pairswap(lfs_block_t pair[2]) {
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_block_t t = pair[0];
pair[0] = pair[1];
pair[1] = t;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
static inline bool lfs_pairisnull(const lfs_block_t pair[2]) {
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
return pair[0] == 0xffffffff || pair[1] == 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
static inline int lfs_paircmp(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
return !(paira[0] == pairb[0] || paira[1] == pairb[1] ||
paira[0] == pairb[1] || paira[1] == pairb[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static inline bool lfs_pairsync(
const lfs_block_t paira[2],
const lfs_block_t pairb[2]) {
return (paira[0] == pairb[0] && paira[1] == pairb[1]) ||
(paira[0] == pairb[1] && paira[1] == pairb[0]);
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
static int lfs_dir_alloc(lfs_t *lfs, lfs_dir_t *dir) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// allocate pair of dir blocks
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
for (int i = 0; i < 2; i++) {
int err = lfs_alloc(lfs, &dir->pair[i]);
if (err) {
return err;
}
}
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// rather than clobbering one of the blocks we just pretend
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// the revision may be valid
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_bd_read(lfs, dir->pair[0], 0, &dir->d.rev, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
dir->d.rev = lfs_fromle32(dir->d.rev);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// set defaults
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
dir->d.rev += 1;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
dir->d.size = sizeof(dir->d)+4;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
dir->d.tail[0] = 0xffffffff;
dir->d.tail[1] = 0xffffffff;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
dir->off = sizeof(dir->d);
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// don't write out yet, let caller take care of that
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
}
static int lfs_dir_fetch(lfs_t *lfs,
lfs_dir_t *dir, const lfs_block_t pair[2]) {
// copy out pair, otherwise may be aliasing dir
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
const lfs_block_t tpair[2] = {pair[0], pair[1]};
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
bool valid = false;
// check both blocks for the most recent revision
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
for (int i = 0; i < 2; i++) {
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
struct lfs_disk_dir test;
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_bd_read(lfs, tpair[i], 0, &test, sizeof(test));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_dir_fromle32(&test);
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
if (err) {
return err;
}
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (valid && lfs_scmp(test.rev, dir->d.rev) < 0) {
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
continue;
}
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if ((0x7fffffff & test.size) < sizeof(test)+4 ||
(0x7fffffff & test.size) > lfs->cfg->block_size) {
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
continue;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
uint32_t crc = 0xffffffff;
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_dir_tole32(&test);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
lfs_crc(&crc, &test, sizeof(test));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_dir_fromle32(&test);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
err = lfs_bd_crc(lfs, tpair[i], sizeof(test),
(0x7fffffff & test.size) - sizeof(test), &crc);
if (err) {
return err;
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (crc != 0) {
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
continue;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
valid = true;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// setup dir in case it's valid
dir->pair[0] = tpair[(i+0) % 2];
dir->pair[1] = tpair[(i+1) % 2];
dir->off = sizeof(dir->d);
dir->d = test;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (!valid) {
LFS_ERROR("Corrupted dir pair at %d %d", tpair[0], tpair[1]);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_CORRUPT;
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
}
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
return 0;
}
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
struct lfs_region {
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
enum {
LFS_FROM_MEM,
LFS_FROM_REGION,
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
LFS_FROM_ATTRS,
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
} type;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_off_t oldoff;
lfs_size_t oldsize;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
const void *buffer;
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_size_t newsize;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
};
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
struct lfs_region_attrs {
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
const struct lfs_attr *attrs;
int count;
};
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
struct lfs_region_region {
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
lfs_block_t block;
lfs_off_t off;
struct lfs_region *regions;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int count;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
};
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
static int lfs_commit_region(lfs_t *lfs, uint32_t *crc,
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_block_t oldblock, lfs_off_t oldoff,
lfs_block_t newblock, lfs_off_t newoff,
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_off_t regionoff, lfs_size_t regionsize,
const struct lfs_region *regions, int count) {
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int i = 0;
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_size_t newend = newoff + regionsize;
while (newoff < newend) {
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// commit from different types of regions
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
if (i < count && regions[i].oldoff == oldoff - regionoff) {
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
switch (regions[i].type) {
case LFS_FROM_MEM: {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_crc(crc, regions[i].buffer, regions[i].newsize);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int err = lfs_bd_prog(lfs, newblock, newoff,
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
regions[i].buffer, regions[i].newsize);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
newoff += regions[i].newsize;
oldoff += regions[i].oldsize;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
break;
}
case LFS_FROM_REGION: {
const struct lfs_region_region *disk = regions[i].buffer;
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
int err = lfs_commit_region(lfs, crc,
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
disk->block, disk->off,
newblock, newoff,
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
disk->off, regions[i].newsize,
disk->regions, disk->count);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
newoff += regions[i].newsize;
oldoff -= regions[i].oldsize;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
break;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
case LFS_FROM_ATTRS: {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
const struct lfs_region_attrs *attrs = regions[i].buffer;
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
// order doesn't matter, so we write new attrs first. this
// is still O(n^2) but only O(n) disk access
for (int j = 0; j < attrs->count; j++) {
if (attrs->attrs[j].size == 0) {
continue;
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_entry_attr_t attr;
attr.d.type = attrs->attrs[j].type;
attr.d.len = attrs->attrs[j].size;
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
lfs_crc(crc, &attr.d, sizeof(attr.d));
int err = lfs_bd_prog(lfs, newblock, newoff,
&attr.d, sizeof(attr.d));
if (err) {
return err;
}
lfs_crc(crc,
attrs->attrs[j].buffer, attrs->attrs[j].size);
err = lfs_bd_prog(lfs, newblock, newoff+sizeof(attr.d),
attrs->attrs[j].buffer, attrs->attrs[j].size);
if (err) {
return err;
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
newoff += 2+attrs->attrs[j].size;
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
}
// copy over attributes without updates
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_off_t oldend = oldoff + regions[i].oldsize;
while (oldoff < oldend) {
lfs_entry_attr_t attr;
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
int err = lfs_bd_read(lfs, oldblock, oldoff,
&attr.d, sizeof(attr.d));
if (err) {
return err;
}
bool updating = false;
for (int j = 0; j < attrs->count; j++) {
if (attr.d.type == attrs->attrs[j].type) {
updating = true;
}
}
if (!updating) {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
err = lfs_commit_region(lfs, crc,
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
oldblock, oldoff,
newblock, newoff,
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
0, 2+attr.d.len,
NULL, 0);
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
if (err) {
return err;
}
newoff += 2+attr.d.len;
}
oldoff += 2+attr.d.len;
}
break;
}
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
i += 1;
} else {
// copy data from old block if not covered by region
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
uint8_t data;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int err = lfs_bd_read(lfs, oldblock, oldoff, &data, 1);
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_crc(crc, &data, 1);
err = lfs_bd_prog(lfs, newblock, newoff, &data, 1);
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
oldoff += 1;
newoff += 1;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
// sanity check our commit math
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
LFS_ASSERT(newoff == newend);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
return 0;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
static int lfs_dir_commit(lfs_t *lfs, lfs_dir_t *dir,
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
const struct lfs_region *regions, int count) {
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
// state for copying over
const lfs_block_t oldpair[2] = {dir->pair[1], dir->pair[0]};
bool relocated = false;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// increment revision count
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
dir->d.rev += 1;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// keep pairs in order such that pair[0] is most recent
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
lfs_pairswap(dir->pair);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
for (int i = 0; i < count; i++) {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
dir->d.size += regions[i].newsize;
dir->d.size -= regions[i].oldsize;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (true) {
int err = lfs_bd_erase(lfs, dir->pair[0]);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// commit header
uint32_t crc = 0xffffffff;
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_dir_tole32(&dir->d);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_crc(&crc, &dir->d, sizeof(dir->d));
err = lfs_bd_prog(lfs, dir->pair[0], 0, &dir->d, sizeof(dir->d));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_dir_fromle32(&dir->d);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// commit region
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
err = lfs_commit_region(lfs, &crc,
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
dir->pair[1], sizeof(dir->d),
dir->pair[0], sizeof(dir->d),
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
0, (0x7fffffff & dir->d.size)-sizeof(dir->d)-4,
regions, count);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
// commit crc
crc = lfs_tole32(crc);
err = lfs_bd_prog(lfs, dir->pair[0],
(0x7fffffff & dir->d.size)-4, &crc, 4);
crc = lfs_fromle32(crc);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
err = lfs_bd_sync(lfs);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
// successful commit, check checksum to make sure
Fixed issue with committing directories to bad-blocks that are stuck This is only an issue in the weird case that are worn down block is left in the odd state of not being able to change the data that resides on the block. That being said, this does pop up often when simulating wear on block devices. Currently, directory commits checked if the write succeeded by crcing the block to avoid the additional RAM cost for another buffer. However, before this commit, directory commits just checked if the block crc was valid, rather than comparing to the expected crc. This would usually work, unless the block was stuck in a state with valid crc. The fix is to simply compare with the expected crc to find errors.
2017-11-16 20:53:45 +00:00
uint32_t ncrc = 0xffffffff;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
err = lfs_bd_crc(lfs, dir->pair[0], 0,
Fixed issue with committing directories to bad-blocks that are stuck This is only an issue in the weird case that are worn down block is left in the odd state of not being able to change the data that resides on the block. That being said, this does pop up often when simulating wear on block devices. Currently, directory commits checked if the write succeeded by crcing the block to avoid the additional RAM cost for another buffer. However, before this commit, directory commits just checked if the block crc was valid, rather than comparing to the expected crc. This would usually work, unless the block was stuck in a state with valid crc. The fix is to simply compare with the expected crc to find errors.
2017-11-16 20:53:45 +00:00
(0x7fffffff & dir->d.size)-4, &ncrc);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (ncrc != crc) {
Fixed issue with committing directories to bad-blocks that are stuck This is only an issue in the weird case that are worn down block is left in the odd state of not being able to change the data that resides on the block. That being said, this does pop up often when simulating wear on block devices. Currently, directory commits checked if the write succeeded by crcing the block to avoid the additional RAM cost for another buffer. However, before this commit, directory commits just checked if the block crc was valid, rather than comparing to the expected crc. This would usually work, unless the block was stuck in a state with valid crc. The fix is to simply compare with the expected crc to find errors.
2017-11-16 20:53:45 +00:00
goto relocate;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Fixed issue with committing directories to bad-blocks that are stuck This is only an issue in the weird case that are worn down block is left in the odd state of not being able to change the data that resides on the block. That being said, this does pop up often when simulating wear on block devices. Currently, directory commits checked if the write succeeded by crcing the block to avoid the additional RAM cost for another buffer. However, before this commit, directory commits just checked if the block crc was valid, rather than comparing to the expected crc. This would usually work, unless the block was stuck in a state with valid crc. The fix is to simply compare with the expected crc to find errors.
2017-11-16 20:53:45 +00:00
break;
Traded enum-based DSL for full callback-based DSL Now, instead of passing an enum for mem/disk commits, we pass a function pointer that can specify any behaviour. This has the benefit of opening up the possibility to pass any sort of commit logic to the committers, and unused logic can be garbage-collected by the compiler if unused. The downside is that unfortunately compilers have a harder time optimizing around functions pointers than enums, and fitting the state into structs for the callbacks may be costly.
2018-03-11 16:28:13 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
//commit was corrupted
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
LFS_DEBUG("Bad block at %d", dir->pair[0]);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// drop caches and prepare to relocate block
relocated = true;
lfs->pcache.block = 0xffffffff;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// can't relocate superblock, filesystem is now frozen
if (lfs_paircmp(oldpair, (const lfs_block_t[2]){0, 1}) == 0) {
LFS_WARN("Superblock %d has become unwritable", oldpair[0]);
return LFS_ERR_CORRUPT;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// relocate half of pair
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
int err = lfs_alloc(lfs, &dir->pair[0]);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (relocated) {
// update references if we relocated
LFS_DEBUG("Relocating %d %d to %d %d",
oldpair[0], oldpair[1], dir->pair[0], dir->pair[1]);
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
int err = lfs_relocate(lfs, oldpair, dir->pair);
if (err) {
return err;
}
}
// shift over any directories that are affected
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
if (lfs_paircmp(d->pair, dir->pair) == 0) {
d->pair[0] = dir->pair[0];
d->pair[1] = dir->pair[1];
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
}
return 0;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
static int lfs_dir_get(lfs_t *lfs, const lfs_dir_t *dir,
lfs_off_t off, void *buffer, lfs_size_t size) {
return lfs_bd_read(lfs, dir->pair[0], off, buffer, size);
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
static int lfs_dir_set(lfs_t *lfs, lfs_dir_t *dir, lfs_entry_t *entry,
struct lfs_region *regions, int count) {
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_ssize_t diff = 0;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
for (int i = 0; i < count; i++) {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
diff += regions[i].newsize;
diff -= regions[i].oldsize;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
}
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_size_t oldsize = entry->size;
if (entry->off == 0) {
entry->off = (0x7fffffff & dir->d.size) - 4;
}
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if ((0x7fffffff & dir->d.size) + diff > lfs->cfg->block_size) {
lfs_dir_t olddir = *dir;
lfs_off_t oldoff = entry->off;
if (oldsize) {
// mark as moving
uint8_t type;
int err = lfs_dir_get(lfs, &olddir, oldoff, &type, 1);
if (err) {
return err;
}
type |= LFS_STRUCT_MOVED;
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &olddir, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, oldoff, 1, &type, 1}}, 1);
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
}
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_dir_t pdir = olddir;
// find available block or create a new one
while ((0x7fffffff & dir->d.size) + oldsize + diff
> lfs->cfg->block_size) {
// we need to allocate a new dir block
if (!(0x80000000 & dir->d.size)) {
pdir = *dir;
int err = lfs_dir_alloc(lfs, dir);
if (err) {
return err;
}
dir->d.tail[0] = pdir.d.tail[0];
dir->d.tail[1] = pdir.d.tail[1];
break;
Naive implementation of resizable entries Now, with the off, diff, and len parameters in each commit entry, we can build up directory commits that resize entries. This adds complexity but opens up the directory blocks to be much more flexible. The main concern is that resizing entries can push around neighboring entries in surprising ways, such as pushing them into new directory blocks when a directory splits. This can break littlefs's internal logic in how it tracks in-flight entries. The most problematic example being open files. Fortunately, this is helped by a global linked-list of all files and directories opened by the filesystem. As entries change size, the state of open files/dirs may be updated as needed. Note this already needed to exist for the ability to remove files/dirs, which has the same issue.
2018-03-11 01:27:57 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->d.tail);
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
}
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// writing out new entry
entry->off = dir->d.size - 4;
entry->size += diff;
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
int err = lfs_dir_commit(lfs, dir, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_REGION, entry->off, 0, &(struct lfs_region_region){
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
olddir.pair[0], oldoff,
regions, count}, entry->size}}, 1);
if (err) {
return err;
}
// update pred dir, unless pred == old we can coalesce
if (!oldsize || lfs_paircmp(pdir.pair, olddir.pair) != 0) {
pdir.d.size |= 0x80000000;
pdir.d.tail[0] = dir->pair[0];
pdir.d.tail[1] = dir->pair[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
}
} else if (oldsize) {
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
olddir.d.size |= 0x80000000;
olddir.d.tail[0] = dir->pair[0];
olddir.d.tail[1] = dir->pair[1];
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// remove old entry
if (oldsize) {
lfs_entry_t oldentry;
oldentry.off = oldoff;
err = lfs_dir_set(lfs, &olddir, &oldentry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, oldsize, NULL, 0}}, 1);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
}
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
goto shift;
Changed dir append to mirror commit DSL Expiremental implementation. This opens up the opportunity to use the same commit description for both commits and appends, which effectively do the same thing. This should lead to better code reuse.
2018-03-11 00:27:43 +00:00
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if ((0x7fffffff & dir->d.size) + diff == sizeof(dir->d)+4) {
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
lfs_dir_t pdir;
int res = lfs_pred(lfs, dir->pair, &pdir);
if (res < 0) {
return res;
}
if (pdir.d.size & 0x80000000) {
pdir.d.size &= dir->d.size | 0x7fffffff;
pdir.d.tail[0] = dir->d.tail[0];
pdir.d.tail[1] = dir->d.tail[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
int err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
}
goto shift;
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
}
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
for (int i = 0; i < count; i++) {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
regions[i].oldoff += entry->off;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
int err = lfs_dir_commit(lfs, dir, regions, count);
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
if (err) {
return err;
}
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
entry->size += diff;
shift:
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
// shift over any files/directories that are affected
for (lfs_file_t *f = lfs->files; f; f = f->next) {
if (lfs_paircmp(f->pair, dir->pair) == 0) {
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (f->pairoff == entry->off && entry->size == 0) {
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
f->pair[0] = 0xffffffff;
f->pair[1] = 0xffffffff;
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
} else if (f->pairoff > entry->off) {
f->pairoff += diff;
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
}
}
}
for (lfs_dir_t *d = lfs->dirs; d; d = d->next) {
if (lfs_paircmp(d->pair, dir->pair) == 0) {
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
if (d->off > entry->off) {
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
d->off += diff;
d->pos += diff;
Separated out version of dir remove/append for non-entries This allows updates to directories without needing to allocate an entry struct for every call.
2018-03-16 01:32:00 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
return 0;
Added better handling for metadata pairs The core algorithim that backs this filesystem's goal of fault tolerance is the alternating of "metadata pairs". Backed by a simple core function for reading and writing, makes heavy use of c99 designated initializers for passing info about multiple chunks in an erase block.
2017-03-12 20:11:52 +00:00
}
Added initial superblock definition Really started working out how the internal structure of the driver will be organized. There are a few hazy lines between the intended data structures with the goal of code reuse, so the function boundaries may end up a bit weird.
2017-03-05 20:11:52 +00:00
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
static int lfs_dir_next(lfs_t *lfs, lfs_dir_t *dir, lfs_entry_t *entry) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
while (dir->off >= (0x7fffffff & dir->d.size)-4) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (!(0x80000000 & dir->d.size)) {
entry->off = dir->off;
return LFS_ERR_NOENT;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->d.tail);
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
dir->off = sizeof(dir->d);
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
dir->pos += sizeof(dir->d) + 4;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
int err = lfs_dir_get(lfs, dir, dir->off, &entry->d, sizeof(entry->d));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_entry_fromle32(&entry->d);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
Extended entry tag to support attributes Zero attributes are actually supported at the moment, but this change will allow entry attribute to be added in a backwards compatible manner. Each dir entry is now prefixed with a 32 bit tag: 4b - entry type 4b - data structure 8b - entry len 8b - attribute len 8b - name len A full entry on disk looks a bit like this: [- 8 -|- 8 -|- 8 -|- 8 -|-- elen --|-- alen --|-- nlen --] [ type | elen | alen | nlen | entry | attrs | name ] The actually contents of the attributes section is a bit handwavey until the first attributes are implemented, but to put plans in place: Each attribute will be prefixed with only a byte that indicates the type of attribute. Attributes should be sorted based on portability, since unknown attributes will force attribute parsing to stop.
2017-07-18 07:09:35 +00:00
entry->off = dir->off;
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
entry->size = lfs_entry_size(entry);
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
dir->off += entry->size;
dir->pos += entry->size;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return 0;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
static int lfs_dir_find(lfs_t *lfs, lfs_dir_t *dir,
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
lfs_entry_t *entry, const char **path) {
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
const char *pathname = *path;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_size_t pathlen;
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
while (true) {
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
nextname:
// skip slashes
pathname += strspn(pathname, "/");
pathlen = strcspn(pathname, "/");
Fixed handling of root as target for create operations Before this patch, when calling lfs_mkdir or lfs_file_open with root as the target, littlefs wouldn't find the path properly and happily run into undefined behaviour. The fix is to populate a directory entry for root in the lfs_dir_find function. As an added plus, this allowed several special cases around root to be completely dropped.
2018-02-22 21:24:17 +00:00
// special case for root dir
if (pathname[0] == '\0') {
*entry = (lfs_entry_t){
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
.d.type = LFS_STRUCT_DIR | LFS_TYPE_DIR,
Fixed handling of root as target for create operations Before this patch, when calling lfs_mkdir or lfs_file_open with root as the target, littlefs wouldn't find the path properly and happily run into undefined behaviour. The fix is to populate a directory entry for root in the lfs_dir_find function. As an added plus, this allowed several special cases around root to be completely dropped.
2018-02-22 21:24:17 +00:00
.d.u.dir[0] = lfs->root[0],
.d.u.dir[1] = lfs->root[1],
};
return 0;
}
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
// skip '.' and root '..'
if ((pathlen == 1 && memcmp(pathname, ".", 1) == 0) ||
(pathlen == 2 && memcmp(pathname, "..", 2) == 0)) {
pathname += pathlen;
goto nextname;
}
// skip if matched by '..' in name
const char *suffix = pathname + pathlen;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_size_t sufflen;
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
int depth = 1;
while (true) {
suffix += strspn(suffix, "/");
sufflen = strcspn(suffix, "/");
if (sufflen == 0) {
break;
}
if (sufflen == 2 && memcmp(suffix, "..", 2) == 0) {
depth -= 1;
if (depth == 0) {
pathname = suffix + sufflen;
goto nextname;
}
} else {
depth += 1;
}
suffix += sufflen;
}
Fixed some corner cases with paths - Added handling for root to lfs_stat - Corrected lfs_dir_find to update path even on failures - Added more checks for missing directories in path
2017-09-17 21:46:09 +00:00
// update what we've found
*path = pathname;
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
// find path
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
while (true) {
int err = lfs_dir_next(lfs, dir, entry);
if (err) {
return err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (((0xf & entry->d.type) != LFS_TYPE_REG &&
(0xf & entry->d.type) != LFS_TYPE_DIR) ||
Extended entry tag to support attributes Zero attributes are actually supported at the moment, but this change will allow entry attribute to be added in a backwards compatible manner. Each dir entry is now prefixed with a 32 bit tag: 4b - entry type 4b - data structure 8b - entry len 8b - attribute len 8b - name len A full entry on disk looks a bit like this: [- 8 -|- 8 -|- 8 -|- 8 -|-- elen --|-- alen --|-- nlen --] [ type | elen | alen | nlen | entry | attrs | name ] The actually contents of the attributes section is a bit handwavey until the first attributes are implemented, but to put plans in place: Each attribute will be prefixed with only a byte that indicates the type of attribute. Attributes should be sorted based on portability, since unknown attributes will force attribute parsing to stop.
2017-07-18 07:09:35 +00:00
entry->d.nlen != pathlen) {
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
continue;
}
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int res = lfs_bd_cmp(lfs, dir->pair[0],
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
entry->off + entry->size - pathlen,
Extended entry tag to support attributes Zero attributes are actually supported at the moment, but this change will allow entry attribute to be added in a backwards compatible manner. Each dir entry is now prefixed with a 32 bit tag: 4b - entry type 4b - data structure 8b - entry len 8b - attribute len 8b - name len A full entry on disk looks a bit like this: [- 8 -|- 8 -|- 8 -|- 8 -|-- elen --|-- alen --|-- nlen --] [ type | elen | alen | nlen | entry | attrs | name ] The actually contents of the attributes section is a bit handwavey until the first attributes are implemented, but to put plans in place: Each attribute will be prefixed with only a byte that indicates the type of attribute. Attributes should be sorted based on portability, since unknown attributes will force attribute parsing to stop.
2017-07-18 07:09:35 +00:00
pathname, pathlen);
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (res < 0) {
return res;
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// found match
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
if (res) {
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
break;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
// check that entry has not been moved
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (entry->d.type & LFS_STRUCT_MOVED) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int moved = lfs_moved(lfs, &entry->d.u);
if (moved < 0 || moved) {
return (moved < 0) ? moved : LFS_ERR_NOENT;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry->d.type &= ~LFS_STRUCT_MOVED;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
}
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
pathname += pathlen;
pathname += strspn(pathname, "/");
if (pathname[0] == '\0') {
return 0;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added dir navigation without needing parent entries This should be the last step to removing the need for parent entries. Parent entries cause all sort of problems with atomic directory updates, especially related to moving/deleting directories. I couldn't figure out a parser for '..' entries without, O(n^2) runtime, a stack, or modifying the path itself. Since the goal is constant memory consumption, I went with the O(n^2) runtime solution, but this may need to be optimized later.
2017-04-14 23:27:06 +00:00
// continue on if we hit a directory
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0xf & entry->d.type) != LFS_TYPE_DIR) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOTDIR;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added path iteration and chained directories All path iteration all goes through the lfs_dir_find function, which manages the syntax of paths and updates the path pointer to just the name stored in the dir entry. Also added directory chaining, which allows more than one block per directory. This is a simple linked list.
2017-04-01 15:09:17 +00:00
int err = lfs_dir_fetch(lfs, dir, entry->d.u.dir);
if (err) {
return err;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
}
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
/// Internal attribute operations ///
static int lfs_dir_getinfo(lfs_t *lfs,
lfs_dir_t *dir, const lfs_entry_t *entry, struct lfs_info *info) {
memset(info, 0, sizeof(*info));
info->type = 0xf & entry->d.type;
if (entry->d.type == (LFS_STRUCT_CTZ | LFS_TYPE_REG)) {
info->size = entry->d.u.file.size;
} else if (entry->d.type == (LFS_STRUCT_INLINE | LFS_TYPE_REG)) {
info->size = lfs_entry_elen(entry);
}
if (lfs_paircmp(entry->d.u.dir, lfs->root) == 0) {
strcpy(info->name, "/");
} else {
int err = lfs_dir_get(lfs, dir,
entry->off + entry->size - entry->d.nlen,
info->name, entry->d.nlen);
if (err) {
return err;
}
}
return 0;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
static int lfs_dir_getattrs(lfs_t *lfs,
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_dir_t *dir, const lfs_entry_t *entry,
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
const struct lfs_attr *attrs, int count) {
// set to zero in case we can't find the attributes or size mismatch
for (int j = 0; j < count; j++) {
memset(attrs[j].buffer, 0, attrs[j].size);
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
// search for attribute in attribute region
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_off_t off = entry->off + 4+lfs_entry_elen(entry);
lfs_off_t end = off + lfs_entry_alen(entry);
while (off < end) {
lfs_entry_attr_t attr;
int err = lfs_dir_get(lfs, dir, off, &attr.d, sizeof(attr.d));
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
if (err) {
return err;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
for (int j = 0; j < count; j++) {
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (attrs[j].type == attr.d.type) {
if (attrs[j].size < attr.d.len) {
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return LFS_ERR_RANGE;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
err = lfs_dir_get(lfs, dir, off+sizeof(attr.d),
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
attrs[j].buffer, attr.d.len);
if (err) {
return err;
}
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
off += 2+attr.d.len;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return 0;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
static lfs_ssize_t lfs_dir_checkattrs(lfs_t *lfs,
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_dir_t *dir, lfs_entry_t *entry,
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
const struct lfs_attr *attrs, int count) {
// check that attributes fit
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
// two separate passes so disk access is O(n)
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
lfs_size_t nsize = 0;
for (int j = 0; j < count; j++) {
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
if (attrs[j].size > 0) {
nsize += 2+attrs[j].size;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_off_t off = entry->off + 4+lfs_entry_elen(entry);
lfs_off_t end = off + lfs_entry_alen(entry);
while (off < end) {
lfs_entry_attr_t attr;
int err = lfs_dir_get(lfs, dir, off, &attr.d, sizeof(attr.d));
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
if (err) {
return err;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
bool updated = false;
for (int j = 0; j < count; j++) {
if (attr.d.type == attrs[j].type) {
updated = true;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
if (!updated) {
nsize += 2+attr.d.len;
}
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
off += 2+attr.d.len;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
if (nsize > lfs->attrs_size || (
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_entry_size(entry) - lfs_entry_alen(entry) + nsize
> lfs->cfg->block_size)) {
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
return LFS_ERR_NOSPC;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return nsize;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
static int lfs_dir_setattrs(lfs_t *lfs,
lfs_dir_t *dir, lfs_entry_t *entry,
const struct lfs_attr *attrs, int count) {
// make sure attributes fit
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
lfs_size_t oldlen = lfs_entry_alen(entry);
lfs_ssize_t newlen = lfs_dir_checkattrs(lfs, dir, entry, attrs, count);
if (newlen < 0) {
return newlen;
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
// commit to entry, majority of work is in LFS_FROM_ATTRS
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
entry->d.alen = (0xc0 & entry->d.alen) | newlen;
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return lfs_dir_set(lfs, dir, entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, 4, &entry->d, 4},
{LFS_FROM_ATTRS, 4+lfs_entry_elen(entry), oldlen,
&(struct lfs_region_attrs){attrs, count}, newlen}}, 2);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level directory operations ///
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
int lfs_mkdir(lfs_t *lfs, const char *path) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// fetch parent directory
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
lfs_dir_t cwd;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
if (err) {
return err;
}
lfs_entry_t entry;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
err = lfs_dir_find(lfs, &cwd, &entry, &path);
Fixed some corner cases with paths - Added handling for root to lfs_stat - Corrected lfs_dir_find to update path even on failures - Added more checks for missing directories in path
2017-09-17 21:46:09 +00:00
if (err != LFS_ERR_NOENT || strchr(path, '/') != NULL) {
Fixed standard name mismatch LFS_ERR_EXISTS -> LFS_ERR_EXIST Matches the standard EEXIST name found on most systems. Other than this name, all other common constant names were consistent in this manner.
2017-11-16 23:50:14 +00:00
return err ? err : LFS_ERR_EXIST;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// build up new directory
lfs_alloc_ack(lfs);
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
lfs_dir_t dir;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
err = lfs_dir_alloc(lfs, &dir);
if (err) {
return err;
}
dir.d.tail[0] = cwd.d.tail[0];
dir.d.tail[1] = cwd.d.tail[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &dir, NULL, 0);
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
if (err) {
return err;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry.d.type = LFS_STRUCT_DIR | LFS_TYPE_DIR;
Extended entry tag to support attributes Zero attributes are actually supported at the moment, but this change will allow entry attribute to be added in a backwards compatible manner. Each dir entry is now prefixed with a 32 bit tag: 4b - entry type 4b - data structure 8b - entry len 8b - attribute len 8b - name len A full entry on disk looks a bit like this: [- 8 -|- 8 -|- 8 -|- 8 -|-- elen --|-- alen --|-- nlen --] [ type | elen | alen | nlen | entry | attrs | name ] The actually contents of the attributes section is a bit handwavey until the first attributes are implemented, but to put plans in place: Each attribute will be prefixed with only a byte that indicates the type of attribute. Attributes should be sorted based on portability, since unknown attributes will force attribute parsing to stop.
2017-07-18 07:09:35 +00:00
entry.d.elen = sizeof(entry.d) - 4;
entry.d.alen = 0;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
entry.d.nlen = nlen;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
entry.d.u.dir[0] = dir.pair[0];
entry.d.u.dir[1] = dir.pair[1];
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
entry.size = 0;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
cwd.d.tail[0] = dir.pair[0];
cwd.d.tail[1] = dir.pair[1];
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
lfs_entry_tole32(&entry.d);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, 0, &entry.d, sizeof(entry.d)},
{LFS_FROM_MEM, 0, 0, path, nlen}}, 2);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
lfs_alloc_ack(lfs);
return 0;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
int lfs_dir_open(lfs_t *lfs, lfs_dir_t *dir, const char *path) {
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
dir->pair[0] = lfs->root[0];
dir->pair[1] = lfs->root[1];
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
int err = lfs_dir_fetch(lfs, dir, dir->pair);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_entry_t entry;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
err = lfs_dir_find(lfs, dir, &entry, &path);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
} else if (entry.d.type != (LFS_STRUCT_DIR | LFS_TYPE_DIR)) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOTDIR;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
err = lfs_dir_fetch(lfs, dir, entry.d.u.dir);
if (err) {
return err;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// setup head dir
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
// special offset for '.' and '..'
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
dir->head[0] = dir->pair[0];
dir->head[1] = dir->pair[1];
dir->pos = sizeof(dir->d) - 2;
dir->off = sizeof(dir->d);
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
// add to list of directories
dir->next = lfs->dirs;
lfs->dirs = dir;
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
return 0;
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
}
int lfs_dir_close(lfs_t *lfs, lfs_dir_t *dir) {
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
// remove from list of directories
for (lfs_dir_t **p = &lfs->dirs; *p; p = &(*p)->next) {
if (*p == dir) {
*p = dir->next;
break;
}
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
return 0;
}
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
int lfs_dir_read(lfs_t *lfs, lfs_dir_t *dir, struct lfs_info *info) {
memset(info, 0, sizeof(*info));
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
// special offset for '.' and '..'
if (dir->pos == sizeof(dir->d) - 2) {
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
info->type = LFS_TYPE_DIR;
strcpy(info->name, ".");
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
dir->pos += 1;
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
return 1;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
} else if (dir->pos == sizeof(dir->d) - 1) {
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
info->type = LFS_TYPE_DIR;
strcpy(info->name, "..");
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
dir->pos += 1;
Removed .. and . entries No longer need to be stored on disk, can be simulated on the chip side. As mentioned in other commits, the parent entries had dozens of problems with atomic updates, as well as making everything just a bit more complex than is needed.
2017-04-15 16:26:37 +00:00
return 1;
}
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
lfs_entry_t entry;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
int err = lfs_dir_next(lfs, dir, &entry);
if (err) {
return (err == LFS_ERR_NOENT) ? 0 : err;
}
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if ((0xf & entry.d.type) != LFS_TYPE_REG &&
(0xf & entry.d.type) != LFS_TYPE_DIR) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
continue;
}
// check that entry has not been moved
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (entry.d.type & LFS_STRUCT_MOVED) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int moved = lfs_moved(lfs, &entry.d.u);
if (moved < 0) {
return moved;
}
if (moved) {
continue;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry.d.type &= ~LFS_STRUCT_MOVED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
break;
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
int err = lfs_dir_getinfo(lfs, dir, &entry, info);
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
if (err) {
return err;
}
return 1;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
int lfs_dir_seek(lfs_t *lfs, lfs_dir_t *dir, lfs_off_t off) {
// simply walk from head dir
int err = lfs_dir_rewind(lfs, dir);
if (err) {
return err;
}
dir->pos = off;
while (off > (0x7fffffff & dir->d.size)) {
off -= 0x7fffffff & dir->d.size;
if (!(0x80000000 & dir->d.size)) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_INVAL;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, dir, dir->d.tail);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
return err;
}
}
dir->off = off;
return 0;
}
lfs_soff_t lfs_dir_tell(lfs_t *lfs, lfs_dir_t *dir) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return dir->pos;
}
int lfs_dir_rewind(lfs_t *lfs, lfs_dir_t *dir) {
// reload the head dir
int err = lfs_dir_fetch(lfs, dir, dir->head);
if (err) {
return err;
}
dir->pair[0] = dir->head[0];
dir->pair[1] = dir->head[1];
dir->pos = sizeof(dir->d) - 2;
dir->off = sizeof(dir->d);
return 0;
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
/// File index list operations ///
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_index(lfs_t *lfs, lfs_off_t *off) {
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
lfs_off_t size = *off;
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
lfs_off_t b = lfs->cfg->block_size - 2*4;
lfs_off_t i = size / b;
Adopted lfs_ctz_index implementation using popcount This reduces the O(n^2logn) runtime to read a file to only O(nlog). The extra O(n) did not touch the disk, so it isn't a problem until the files become very large, but this solution comes with very little cost. Long story short, you can find the block index + offset pair for a CTZ linked-list with this series of formulas: n' = floor(N / (B - 2w/8)) N' = (B - 2w/8)n' + (w/8)popcount(n') off' = N - N' n, off = n'-1, off'+B if off' < 0 n', off'+(w/8)(ctz(n')+1) if off' >= 0 For the long story, you will need to see the updated DESIGN.md
2017-10-17 00:08:47 +00:00
if (i == 0) {
return 0;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Adopted alternative implementation for lfs_ctz_index Same runtime cost, however reduces the logic and avoids one of the two big branches. See the DESIGN.md for more info. Now uses these equations instead of the messy guess and correct method: n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8) off = N - (B-w2/8)n - w/8popcount(n)
2017-10-18 05:33:59 +00:00
i = (size - 4*(lfs_popc(i-1)+2)) / b;
*off = size - b*i - 4*lfs_popc(i);
return i;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_find(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
lfs_block_t head, lfs_size_t size,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t pos, lfs_block_t *block, lfs_off_t *off) {
if (size == 0) {
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
*off = 0;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t current = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
lfs_off_t target = lfs_ctz_index(lfs, &pos);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
while (current > target) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_size_t skip = lfs_min(
lfs_npw2(current-target+1) - 1,
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
lfs_ctz(current));
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
int err = lfs_cache_read(lfs, rcache, pcache, head, 4*skip, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
current -= 1 << skip;
}
*block = head;
*off = pos;
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_extend(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
lfs_block_t *block, lfs_off_t *off) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
// go ahead and grab a block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
if (err) {
return err;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(nblock >= 2 && nblock <= lfs->cfg->block_count);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
if (true) {
err = lfs_bd_erase(lfs, nblock);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (size == 0) {
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 0;
return 0;
}
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
size -= 1;
lfs_off_t index = lfs_ctz_index(lfs, &size);
size += 1;
// just copy out the last block if it is incomplete
if (size != lfs->cfg->block_size) {
for (lfs_off_t i = 0; i < size; i++) {
uint8_t data;
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_read(lfs, rcache, NULL,
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
head, i, &data, 1);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, i, &data, 1);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = size;
return 0;
}
// append block
index += 1;
lfs_size_t skips = lfs_ctz(index) + 1;
for (lfs_off_t i = 0; i < skips; i++) {
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_tole32(head);
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_cache_prog(lfs, pcache, rcache,
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
nblock, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (i != skips-1) {
err = lfs_cache_read(lfs, rcache, NULL,
head, 4*i, &head, 4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
head = lfs_fromle32(head);
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(head >= 2 && head <= lfs->cfg->block_count);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
*block = nblock;
Removed toolchain specific warnings - Comparisons with differently signed integer types - Incorrectly signed constant - Unreachable default returns - Leaked uninitialized variables in relocate goto statements
2017-10-17 00:31:56 +00:00
*off = 4*skips;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
relocate:
Modified lfs_ctz_extend to be a little bit safer Specifically around error handling. As is, incorrectly handled errors could cause higher code to get uninitialized blocks, potentially leading to writes to arbitray blocks on storage.
2017-11-16 21:10:17 +00:00
LFS_DEBUG("Bad block at %d", nblock);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// just clear cache and try a new block
pcache->block = 0xffffffff;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
static int lfs_ctz_traverse(lfs_t *lfs,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_cache_t *rcache, const lfs_cache_t *pcache,
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
lfs_block_t head, lfs_size_t size,
int (*cb)(void*, lfs_block_t), void *data) {
if (size == 0) {
return 0;
}
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
lfs_off_t index = lfs_ctz_index(lfs, &(lfs_off_t){size-1});
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
while (true) {
int err = cb(data, head);
if (err) {
return err;
}
if (index == 0) {
return 0;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
lfs_block_t heads[2];
int count = 2 - (index & 1);
err = lfs_cache_read(lfs, rcache, pcache, head, 0, &heads, count*4);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
heads[0] = lfs_fromle32(heads[0]);
heads[1] = lfs_fromle32(heads[1]);
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (err) {
return err;
}
Reduced ctz traverse runtime by 2x Unfortunately for us, the ctz skip-list does not offer very much benefit for full traversals. Since the information about which blocks are in use are spread throughout the file, we can't use the fast-lanes embedded in the skip-list without missing blocks. However, it turns out we can at least use the 2nd level of the skip-list without missing any blocks. From an asymptotic analysis, a constant speed up isn't interesting, but from a pragmatic perspective, a 2x speedup is not bad.
2017-12-27 18:30:01 +00:00
for (int i = 0; i < count-1; i++) {
err = cb(data, heads[i]);
if (err) {
return err;
}
}
head = heads[count-1];
index -= count;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
/// Top level file operations ///
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
int lfs_file_open(lfs_t *lfs, lfs_file_t *file,
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
const char *path, int flags) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// deorphan if we haven't yet, needed at most once after poweron
if ((flags & 3) != LFS_O_RDONLY && !lfs->deorphaned) {
int err = lfs_deorphan(lfs);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// allocate entry for file if it doesn't exist
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_dir_t cwd;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
lfs_entry_t entry;
err = lfs_dir_find(lfs, &cwd, &entry, &path);
Fixed some corner cases with paths - Added handling for root to lfs_stat - Corrected lfs_dir_find to update path even on failures - Added more checks for missing directories in path
2017-09-17 21:46:09 +00:00
if (err && (err != LFS_ERR_NOENT || strchr(path, '/') != NULL)) {
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return err;
}
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err == LFS_ERR_NOENT) {
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
if (!(flags & LFS_O_CREAT)) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOENT;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that name fits
lfs_size_t nlen = strlen(path);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// create entry to remember name
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
entry.d.type = LFS_STRUCT_INLINE | LFS_TYPE_REG;
entry.d.elen = 0;
Extended entry tag to support attributes Zero attributes are actually supported at the moment, but this change will allow entry attribute to be added in a backwards compatible manner. Each dir entry is now prefixed with a 32 bit tag: 4b - entry type 4b - data structure 8b - entry len 8b - attribute len 8b - name len A full entry on disk looks a bit like this: [- 8 -|- 8 -|- 8 -|- 8 -|-- elen --|-- alen --|-- nlen --] [ type | elen | alen | nlen | entry | attrs | name ] The actually contents of the attributes section is a bit handwavey until the first attributes are implemented, but to put plans in place: Each attribute will be prefixed with only a byte that indicates the type of attribute. Attributes should be sorted based on portability, since unknown attributes will force attribute parsing to stop.
2017-07-18 07:09:35 +00:00
entry.d.alen = 0;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
entry.d.nlen = nlen;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
entry.size = 0;
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, 0, &entry.d, 4},
{LFS_FROM_MEM, 0, 0, path, nlen}}, 2);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
if (err) {
return err;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
} else if ((0xf & entry.d.type) == LFS_TYPE_DIR) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_ISDIR;
Removed a layer of indirection for index-list lookup Files are now stored directly in the index-list, instead of being referenced by pointers that used to live there. This somewhat reduces the complexity around handling files, while still keeping the O(logn) lookup cost.
2017-04-23 00:48:31 +00:00
} else if (flags & LFS_O_EXCL) {
Fixed standard name mismatch LFS_ERR_EXISTS -> LFS_ERR_EXIST Matches the standard EEXIST name found on most systems. Other than this name, all other common constant names were consistent in this manner.
2017-11-16 23:50:14 +00:00
return LFS_ERR_EXIST;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
// allocate buffer if needed
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
file->cache.block = 0xffffffff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (lfs->cfg->file_buffer) {
file->cache.buffer = lfs->cfg->file_buffer;
} else if ((file->flags & 3) == LFS_O_RDONLY) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->read_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
file->cache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!file->cache.buffer) {
return LFS_ERR_NOMEM;
}
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
// setup file struct
file->pair[0] = cwd.pair[0];
file->pair[1] = cwd.pair[1];
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
file->pairoff = entry.off;
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
file->flags = flags;
file->pos = 0;
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// calculate max inline size based on the size of the entry
file->inline_size = lfs_min(lfs->inline_size,
lfs->cfg->block_size - (sizeof(cwd.d)+4) -
(lfs_entry_size(&entry) - lfs_entry_elen(&entry)));
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if ((0x70 & entry.d.type) == LFS_STRUCT_INLINE) {
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
// load inline files
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->head = 0xfffffffe;
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
file->size = lfs_entry_elen(&entry);
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
err = lfs_dir_get(lfs, &cwd,
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
entry.off + 4,
file->cache.buffer, file->size);
if (err) {
lfs_free(file->cache.buffer);
return err;
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
} else {
// use ctz list from entry
file->head = entry.d.u.file.head;
file->size = entry.d.u.file.size;
}
// truncate if requested
if (flags & LFS_O_TRUNC) {
if (file->size != 0) {
file->flags |= LFS_F_DIRTY;
}
file->head = 0xfffffffe;
file->size = 0;
file->flags |= LFS_F_INLINE;
file->cache.block = file->head;
file->cache.off = 0;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// add to list of files
file->next = lfs->files;
lfs->files = file;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
int lfs_file_close(lfs_t *lfs, lfs_file_t *file) {
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
int err = lfs_file_sync(lfs, file);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
// remove from list of files
for (lfs_file_t **p = &lfs->files; *p; p = &(*p)->next) {
if (*p == file) {
*p = file->next;
break;
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
// clean up memory
if (!lfs->cfg->file_buffer) {
lfs_free(file->cache.buffer);
Fixed missing erase during file relocation This was an easy fix, but highlighted the fact that the current testing framework doesn't detect when a block is written to without an associated erase. Added a quick solution that creates an empty file during an erase.
2017-06-25 21:56:12 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
Fixed missing erase during file relocation This was an easy fix, but highlighted the fact that the current testing framework doesn't detect when a block is written to without an associated erase. Added a quick solution that creates an empty file during an erase.
2017-06-25 21:56:12 +00:00
return err;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
static int lfs_file_relocate(lfs_t *lfs, lfs_file_t *file) {
relocate:;
// just relocate what exists into new block
lfs_block_t nblock;
int err = lfs_alloc(lfs, &nblock);
Adopted redundant cache read in lfs_file_relocate Previously had some custom logic that could be reduced
2017-06-25 22:23:40 +00:00
if (err) {
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
err = lfs_bd_erase(lfs, nblock);
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
// either read from dirty cache or disk
for (lfs_off_t i = 0; i < file->off; i++) {
uint8_t data;
err = lfs_cache_read(lfs, &lfs->rcache, &file->cache,
file->block, i, &data, 1);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
err = lfs_cache_prog(lfs, &lfs->pcache, &lfs->rcache,
nblock, i, &data, 1);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
}
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
// copy over new state of file
memcpy(file->cache.buffer, lfs->pcache.buffer, lfs->cfg->prog_size);
file->cache.block = lfs->pcache.block;
file->cache.off = lfs->pcache.off;
lfs->pcache.block = 0xffffffff;
file->block = nblock;
return 0;
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
static int lfs_file_flush(lfs_t *lfs, lfs_file_t *file) {
if (file->flags & LFS_F_READING) {
file->flags &= ~LFS_F_READING;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (file->flags & LFS_F_WRITING) {
lfs_off_t pos = file->pos;
if (!(file->flags & LFS_F_INLINE)) {
// copy over anything after current branch
lfs_file_t orig = {
.head = file->head,
.size = file->size,
.flags = LFS_O_RDONLY,
.pos = file->pos,
.cache = lfs->rcache,
};
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
lfs->rcache.block = 0xffffffff;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
while (file->pos < file->size) {
// copy over a byte at a time, leave it up to caching
// to make this efficient
uint8_t data;
lfs_ssize_t res = lfs_file_read(lfs, &orig, &data, 1);
if (res < 0) {
return res;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
res = lfs_file_write(lfs, file, &data, 1);
if (res < 0) {
return res;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// keep our reference to the rcache in sync
if (lfs->rcache.block != 0xffffffff) {
orig.cache.block = 0xffffffff;
lfs->rcache.block = 0xffffffff;
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
// write out what we have
while (true) {
int err = lfs_cache_flush(lfs, &file->cache, &lfs->rcache);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
return err;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
break;
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
relocate:
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
LFS_DEBUG("Bad block at %d", file->block);
err = lfs_file_relocate(lfs, file);
if (err) {
return err;
}
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->size = lfs_max(file->pos, file->size);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// actual file updates
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->head = file->block;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->size = file->pos;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_WRITING;
file->flags |= LFS_F_DIRTY;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = pos;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return 0;
}
int lfs_file_sync(lfs_t *lfs, lfs_file_t *file) {
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
if ((file->flags & LFS_F_DIRTY) &&
!(file->flags & LFS_F_ERRED) &&
!lfs_pairisnull(file->pair)) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// update dir entry
lfs_dir_t cwd;
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &cwd, file->pair);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (err) {
return err;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_entry_t entry = {.off = file->pairoff};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (err) {
return err;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
entry.size = lfs_entry_size(&entry);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
LFS_ASSERT((0xf & entry.d.type) == LFS_TYPE_REG);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_size_t oldelen = lfs_entry_elen(&entry);
lfs_size_t oldalen = lfs_entry_alen(&entry);
const void *buffer;
lfs_size_t size;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
// either update the references or inline the whole file
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
entry.d.type = LFS_STRUCT_CTZ | LFS_TYPE_REG;
entry.d.u.file.head = file->head;
entry.d.u.file.size = file->size;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
lfs_entry_tole32(&entry.d);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
buffer = (const uint8_t *)&entry.d + 4;
size = sizeof(entry.d) - 4;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
} else {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
entry.d.type = LFS_STRUCT_INLINE | LFS_TYPE_REG;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
buffer = file->cache.buffer;
size = file->size;
}
// get new alen from disk
lfs_ssize_t newalen = lfs_dir_checkattrs(lfs, &cwd, &entry,
file->attrs, file->attrcount);
if (newalen < 0) {
return newalen;
}
entry.d.elen = size & 0xff;
entry.d.alen = (newalen & 0x3f) | ((size >> 2) & 0xc0);
// write out update
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
{LFS_FROM_MEM, 0, 4, &entry.d, 4},
{LFS_FROM_MEM, 4, oldelen, buffer, size},
{LFS_FROM_ATTRS, 4+oldelen, oldalen,
&(struct lfs_region_attrs){file->attrs, file->attrcount},
newalen}}, 3);
if (err) {
return err;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags &= ~LFS_F_DIRTY;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return 0;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size) {
uint8_t *data = buffer;
lfs_size_t nsize = size;
if ((file->flags & 3) == LFS_O_WRONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_WRITING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// flush out any writes
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (file->pos >= file->size) {
// eof if past end
return 0;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
size = lfs_min(size, file->size - file->pos);
nsize = size;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
while (nsize > 0) {
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_READING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos, &file->block, &file->off);
if (err) {
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->flags |= LFS_F_READING;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
// read as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
int err = lfs_cache_read(lfs, &file->cache, NULL,
file->block, file->off, data, diff);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
data += diff;
nsize -= diff;
}
return size;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size) {
const uint8_t *data = buffer;
lfs_size_t nsize = size;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (file->flags & LFS_F_READING) {
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// drop any reads
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
}
if ((file->flags & LFS_O_APPEND) && file->pos < file->size) {
file->pos = file->size;
}
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > file->size) {
// fill with zeros
lfs_off_t pos = file->pos;
file->pos = file->size;
while (file->pos < pos) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if ((file->flags & LFS_F_INLINE) &&
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
file->pos + nsize >= file->inline_size) {
Bumped versions, cleaned up some TODOs and missing comments
2018-04-03 13:29:28 +00:00
// inline file doesn't fit anymore
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
file->block = 0xfffffffe;
file->off = file->pos;
lfs_alloc_ack(lfs);
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
int err = lfs_file_relocate(lfs, file);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
}
Better implementation of inline files, now with overflowing Now when a file overflows the max inline file size, it will be correctly written out to a proper block. Additionally, tweaked corner cases around inline file, however this still needs significant testing. A real neat part that surprised me is that littlefs _already_ contains the logic for writing out inline files: in lfs_file_relocate! With a bit of tweaking, littlefs can pull off both the overflow from inline to normal files _and_ the relocating of bad blocks in files with the same piece of logic.
2018-03-18 01:32:16 +00:00
file->flags &= ~LFS_F_INLINE;
file->flags |= LFS_F_WRITING;
}
while (nsize > 0) {
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// check if we need a new block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
if (!(file->flags & LFS_F_WRITING) ||
file->off == lfs->cfg->block_size) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if (!(file->flags & LFS_F_INLINE)) {
Shoehorned in hacky implementation of inline files Proof-of-concept implementation of inline files that stores the file's content directly in its parent's directory pair. Inline files are indicated by a different type stored in an entry's struct field, and take advantage of resizable entries. Where a normal file's entry would normally hold the reference to the CTZ skip-list, an inline file's entry contains the contents of the actual file. Unfortunately, storing the inline file on disk is the easy part. We also need to manage inline files in the internals of littlefs and provide the same operations that we do on normal files, all while reusing as much code as possible to avoid a significant increase in code cost. There is a relatively simple, though maybe a bit hacky, solution here. If a file fits entirely in a cache line, the file logic never actually has to go to disk. This means we can just give the file a "pretend" block (hopefully one that would assert if ever written to), and carry out file operations as normal, as long as we catch the file before it exceeds the cache line and write out the file to an actual disk.
2018-03-17 15:28:14 +00:00
if (!(file->flags & LFS_F_WRITING) && file->pos > 0) {
// find out which block we're extending from
int err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
file->pos-1, &file->block, &file->off);
if (err) {
file->flags |= LFS_F_ERRED;
return err;
}
// mark cache as dirty since we may have read data into it
file->cache.block = 0xffffffff;
}
// extend file with new blocks
lfs_alloc_ack(lfs);
int err = lfs_ctz_extend(lfs, &lfs->rcache, &file->cache,
file->block, file->pos,
&file->block, &file->off);
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
return err;
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
} else {
file->block = 0xfffffffe;
file->off = file->pos;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed issue with cold-write after seek to block boundary This off-by-one error was caused by a slight difference between the pos argument to lfs_index_find and lfs_index_extend. When pos is on a block boundary, lfs_index_extend expects block to point before pos, as it would when writing a file linearly. But when seeking to that pos, the lfs_index_find to warm things up just supplies the block it expects pos to be in. Fixed the off-by-one error and added a test case for several of these cold seek+writes.
2017-09-17 17:53:18 +00:00
file->flags |= LFS_F_WRITING;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
// program as much as we can in current block
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
lfs_size_t diff = lfs_min(nsize, lfs->cfg->block_size - file->off);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (true) {
Added internal check of data written to disk Before, the littlefs relied on the underlying block device to report corruption that occurs when writing data to disk. This requirement is easy to miss or implement incorrectly, since the error detection is only required when a block becomes corrupted, which is very unlikely to happen until late in the block device's lifetime. The littlefs can detect corruption itself by reading back written data. This requires a bit of care to reuse the available buffers, and may rely on checksums to avoid additional RAM requirements. This does have a runtime penalty with the extra read operations, but should make the littlefs much more robust to different implementations.
2017-06-24 05:43:05 +00:00
int err = lfs_cache_prog(lfs, &file->cache, &lfs->rcache,
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
file->block, file->off, data, diff);
if (err) {
if (err == LFS_ERR_CORRUPT) {
goto relocate;
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
break;
relocate:
Fixed relocation bug when a file is closed with lingering caches This bug required larger cache sizes to notice, since block errors usually get detected in the early stages of writing to files. Since the fix here requires both lfs_file_write and lfs_file_flush relocate file blocks, the relocation logic was moved out into a seperate function.
2017-06-25 19:01:33 +00:00
err = lfs_file_relocate(lfs, file);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags |= LFS_F_ERRED;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return err;
}
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos += diff;
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
file->off += diff;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
data += diff;
nsize -= diff;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_alloc_ack(lfs);
Added correct handling of file syncing around overwrites Now all of the open flags are correctly handled Even annoying cases where we can't trust the blocks that are already on file, such as appending existing files and writing to the middle of files.
2017-04-23 02:42:22 +00:00
}
Added sticky-bit for preventing file syncs after write errors Short story, files are no longer committed to directories during file sync/close if the last write did not complete successfully. This avoids a set of interesting user-experience issues related to the end-of-life behaviour of the filesystem. As a filesystem approaches end-of-life, the chances of running into LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after at the end of a devices life, it's likely that operating in these conditions hasn't been tested thoroughly. In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after parts of the file have been written out. If the program simply continues and closes the file, the file is written out half completed. Since littlefs has a strong garuntee the prevents half-writes, it's unlikely this state of the file would be expected. To make things worse, since close is also responsible for memory cleanup, it's actually _impossible_ to continue working as it was without leaking memory. By prevent the file commits, end-of-life behaviour should at least retain a previous copy of the filesystem without any surprises.
2017-11-16 23:25:41 +00:00
file->flags &= ~LFS_F_ERRED;
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
return size;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence) {
// write out everything beforehand, may be noop if rdonly
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
// update pos
if (whence == LFS_SEEK_SET) {
file->pos = off;
} else if (whence == LFS_SEEK_CUR) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->pos) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->pos + off;
} else if (whence == LFS_SEEK_END) {
Fixed positive seek bounds checking This bug was a result of an annoying corner case around intermingling signed and unsigned offsets. The boundary check that prevents seeking a file to a position before the file was preventing valid seeks with positive offsets. This corner case is a bit more complicated than it looks because the offset is signed, while the size of the file is unsigned. Simply casting both to signed or unsigned offsets won't handle large files.
2018-01-03 21:00:04 +00:00
if (off < 0 && (lfs_off_t)-off > file->size) {
Added checks for out-of-bound seeks - out-of-bound read results in eof - out-of-bound write will fill missing area with zeros The write behaviour matches expected posix behaviour, but was under consideration for not being dropped, since littlefs does not support holes, and support of out-of-band seeks adds complexity. However, it turned out filling with zeros was trivial, and only cost an extra 74 bytes of flash (0.48%).
2017-09-17 22:57:12 +00:00
return LFS_ERR_INVAL;
}
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
file->pos = file->size + off;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Fixed incorrect return value from lfs_file_seek lfs_file_seek returned the _previous_ file offset on success, where most standards return the _calculated_ offset on success. This just falls into me not noticing a mistake, and shows why it's always helpful to have a second set of eyes on code.
2017-09-27 00:50:39 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size) {
if ((file->flags & 3) == LFS_O_RDONLY) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_BADF;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
}
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
lfs_off_t oldsize = lfs_file_size(lfs, file);
if (size < oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
// need to flush since directly changing metadata
int err = lfs_file_flush(lfs, file);
if (err) {
return err;
}
// lookup new head in ctz skip list
err = lfs_ctz_find(lfs, &file->cache, NULL,
file->head, file->size,
size, &file->head, &(lfs_off_t){0});
if (err) {
return err;
}
file->size = size;
file->flags |= LFS_F_DIRTY;
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
} else if (size > oldsize) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
lfs_off_t pos = file->pos;
// flush+seek if not already at end
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
if (file->pos != oldsize) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
int err = lfs_file_seek(lfs, file, 0, LFS_SEEK_END);
Fixed error check when truncating files to larger size
2018-02-04 19:48:44 +00:00
if (err < 0) {
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
return err;
}
}
// fill with zeros
while (file->pos < size) {
lfs_ssize_t res = lfs_file_write(lfs, file, &(uint8_t){0}, 1);
if (res < 0) {
return res;
}
}
// restore pos
int err = lfs_file_seek(lfs, file, pos, LFS_SEEK_SET);
if (err < 0) {
return err;
}
}
return 0;
}
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
lfs_soff_t lfs_file_tell(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
return file->pos;
Added support for full seek operations A rather involved upgrade for both files and directories, seek and related functions are now completely supported: - lfs_file_seek - lfs_file_tell - lfs_file_rewind - lfs_file_size - lfs_dir_seek - lfs_dir_tell - lfs_dir_rewind This change also highlighted the concern that lfs_off_t is unsigned, whereas off_t is traditionally signed. Unfortunately, lfs_off_t is already used intensively through the codebase, so in focusing on moving forward and avoiding getting bogged down by details, I'm going to keep it as is and use the signed type lfs_soff_t where necessary.
2017-04-23 04:11:13 +00:00
}
int lfs_file_rewind(lfs_t *lfs, lfs_file_t *file) {
lfs_soff_t res = lfs_file_seek(lfs, file, 0, LFS_SEEK_SET);
if (res < 0) {
return res;
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {
Silenced more of aldot's warnings Flags used: -Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes -Wunused -Wunused-parameter -Wunused-function -Wunused-value -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
2018-02-04 19:10:07 +00:00
(void)lfs;
Added lfs_file_truncate As a copy-on-write filesystem, the truncate function is a very nice function to have, as it can take advantage of reusing the data already written out to disk.
2018-01-20 23:30:40 +00:00
if (file->flags & LFS_F_WRITING) {
return lfs_max(file->pos, file->size);
} else {
return file->size;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
int lfs_file_getattrs(lfs_t *lfs, lfs_file_t *file,
const struct lfs_attr *attrs, int count) {
// set to null in case we can't find the attrs (missing file?)
for (int j = 0; j < count; j++) {
memset(attrs[j].buffer, 0, attrs[j].size);
}
// load from disk if we haven't already been deleted
if (!lfs_pairisnull(file->pair)) {
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, file->pair);
if (err) {
return err;
}
lfs_entry_t entry = {.off = file->pairoff};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (err) {
return err;
}
entry.size = lfs_entry_size(&entry);
err = lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
if (err) {
return err;
}
}
// override an attrs we have stored locally
for (int i = 0; i < file->attrcount; i++) {
for (int j = 0; j < count; j++) {
if (attrs[j].type == file->attrs[i].type) {
if (attrs[j].size < file->attrs[i].size) {
return LFS_ERR_RANGE;
}
memcpy(attrs[j].buffer,
file->attrs[i].buffer, file->attrs[i].size);
}
}
}
return 0;
}
int lfs_file_setattrs(lfs_t *lfs, lfs_file_t *file,
const struct lfs_attr *attrs, int count) {
// just tack to the file, will be written at sync time
file->attrs = attrs;
file->attrcount = count;
// at least make sure attributes fit
if (!lfs_pairisnull(file->pair)) {
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, file->pair);
if (err) {
return err;
}
lfs_entry_t entry = {.off = file->pairoff};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &cwd, entry.off, &entry.d, 4);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (err) {
return err;
}
entry.size = lfs_entry_size(&entry);
lfs_ssize_t res = lfs_dir_checkattrs(lfs, &cwd, &entry, attrs, count);
if (res < 0) {
return res;
}
}
return 0;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Commentary typo fix
2018-01-30 19:07:37 +00:00
/// General fs operations ///
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info) {
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t entry;
err = lfs_dir_find(lfs, &cwd, &entry, &path);
if (err) {
return err;
}
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
return lfs_dir_getinfo(lfs, &cwd, &entry, info);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
int lfs_remove(lfs_t *lfs, const char *path) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t entry;
err = lfs_dir_find(lfs, &cwd, &entry, &path);
if (err) {
return err;
}
lfs_dir_t dir;
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0xf & entry.d.type) == LFS_TYPE_DIR) {
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
// must be empty before removal, checking size
// without masking top bit checks for any case where
// dir is not empty
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &dir, entry.d.u.dir);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
} else if (dir.d.size != sizeof(dir.d)+4) {
Added error code LFS_ERR_NOTEMPTY As noted by itayzafrir, removing a non-empty directory should error with ENOTEMPTY, not EINVAL
2017-12-27 17:47:48 +00:00
return LFS_ERR_NOTEMPTY;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
}
// remove the entry
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// if we were a directory, find pred, replace tail
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0xf & entry.d.type) == LFS_TYPE_DIR) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
int res = lfs_pred(lfs, dir.pair, &cwd);
if (res < 0) {
return res;
Added proper handling for removing open files Conveniently we previously added a linked-list of files for things like this. This should handle most of the corner cases where files are open during strange operations. This also brings up the point that we aren't doing anything similar for directories and don't even have a dir linked-list. After thinking about it for a while, I've decided to leave out this handling for dirs. It will likely be very complicated, with little gains as directories are less used in embedded systems. Additionally, dirs are only open for reading, and corruption will probably just cause the dir iteration to terminate. If needed, correct handling of open directories can be added later.
2017-04-30 17:35:50 +00:00
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(res); // must have pred
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
cwd.d.tail[0] = dir.d.tail[0];
cwd.d.tail[1] = dir.d.tail[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &cwd, NULL, 0);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
}
}
return 0;
}
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// deorphan if we haven't yet, needed at most once after poweron
if (!lfs->deorphaned) {
int err = lfs_deorphan(lfs);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
// find old entry
lfs_dir_t oldcwd;
int err = lfs_dir_fetch(lfs, &oldcwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t oldentry;
err = lfs_dir_find(lfs, &oldcwd, &oldentry, &oldpath);
if (err) {
return err;
}
// allocate new entry
lfs_dir_t newcwd;
err = lfs_dir_fetch(lfs, &newcwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t preventry;
err = lfs_dir_find(lfs, &newcwd, &preventry, &newpath);
Fixed some corner cases with paths - Added handling for root to lfs_stat - Corrected lfs_dir_find to update path even on failures - Added more checks for missing directories in path
2017-09-17 21:46:09 +00:00
if (err && (err != LFS_ERR_NOENT || strchr(newpath, '/') != NULL)) {
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
bool prevexists = (err != LFS_ERR_NOENT);
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
bool samepair = (lfs_paircmp(oldcwd.pair, newcwd.pair) == 0);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that name fits
lfs_size_t nlen = strlen(newpath);
if (nlen > lfs->name_size) {
return LFS_ERR_NAMETOOLONG;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
// must have same type
if (prevexists && preventry.d.type != oldentry.d.type) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_ISDIR;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
lfs_dir_t dir;
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (prevexists && (0xf & preventry.d.type) == LFS_TYPE_DIR) {
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
// must be empty before removal, checking size
// without masking top bit checks for any case where
// dir is not empty
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &dir, preventry.d.u.dir);
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
if (err) {
return err;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
} else if (dir.d.size != sizeof(dir.d)+4) {
Fixed some minor error code differences - Write on read-only file to return LFS_ERR_BADF - Renaming directory onto file to return LFS_ERR_NOTEMPTY - Changed LFS_ERR_INVAL in lfs_file_seek to assert
2018-02-04 20:36:36 +00:00
return LFS_ERR_NOTEMPTY;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// mark as moving
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
oldentry.d.type |= LFS_STRUCT_MOVED;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &oldcwd, &oldentry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, 1, &oldentry.d.type, 1}}, 1);
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
oldentry.d.type &= ~LFS_STRUCT_MOVED;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
if (err) {
return err;
}
// update pair if newcwd == oldcwd
if (samepair) {
newcwd = oldcwd;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
// move to new location
lfs_entry_t newentry = preventry;
newentry.d = oldentry.d;
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
newentry.d.type &= ~LFS_STRUCT_MOVED;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
newentry.d.nlen = nlen;
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
newentry.size = prevexists ? preventry.size : 0;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
lfs_size_t newsize = oldentry.size - oldentry.d.nlen + newentry.d.nlen;
err = lfs_dir_set(lfs, &newcwd, &newentry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_REGION, 0, prevexists ? preventry.size : 0,
&(struct lfs_region_region){
oldcwd.pair[0], oldentry.off, (struct lfs_region[]){
{LFS_FROM_MEM, 0, 4, &newentry.d, 4},
{LFS_FROM_MEM, newsize-nlen, 0, newpath, nlen}}, 2},
newsize}}, 1);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
if (err) {
return err;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// update pair if newcwd == oldcwd
if (samepair) {
oldcwd = newcwd;
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// remove old entry
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &oldcwd, &oldentry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, oldentry.size, NULL, 0}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// if we were a directory, find pred, replace tail
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (prevexists && (0xf & preventry.d.type) == LFS_TYPE_DIR) {
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
int res = lfs_pred(lfs, dir.pair, &newcwd);
if (res < 0) {
return res;
}
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(res); // must have pred
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
newcwd.d.tail[0] = dir.d.tail[0];
newcwd.d.tail[1] = dir.d.tail[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &newcwd, NULL, 0);
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
if (err) {
return err;
}
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
return 0;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
int lfs_getattrs(lfs_t *lfs, const char *path,
const struct lfs_attr *attrs, int count) {
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t entry;
err = lfs_dir_find(lfs, &cwd, &entry, &path);
if (err) {
return err;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return lfs_dir_getattrs(lfs, &cwd, &entry, attrs, count);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
int lfs_setattrs(lfs_t *lfs, const char *path,
const struct lfs_attr *attrs, int count) {
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, lfs->root);
if (err) {
return err;
}
lfs_entry_t entry;
err = lfs_dir_find(lfs, &cwd, &entry, &path);
if (err) {
return err;
}
Added support for atomically committing custom attributes Although it's simple and probably what most users expect, the previous custom attributes API suffered from one problem: the inability to update attributes atomically. If we consider our timestamp use case, updating a file would require: 1. Update the file 2. Update the timestamp If a power loss occurs during this sequence of updates, we could end up with a file with an incorrect timestamp. Is this a big deal? Probably not, but it could be a surprise only found after a power-loss. And littlefs was developed with the _specifically_ to avoid suprises during power-loss. The littlefs is perfectly capable of bundling multiple attribute updates in a single directory commit. That's kind of what it was designed to do. So all we need is a new committer opcode for list of attributes, and then poking that list of attributes through the API. We could provide the single-attribute functions, but don't, because the fewer functions makes for a smaller codebase, and these are already the more advanced functions so we can expect more from users. This also changes semantics about what happens when we don't find an attribute, since erroring would throw away all of the other attributes we're processing. To atomically commit both custom attributes and file updates, we need a new API, lfs_file_setattr. Unfortunately the semantics are a bit more confusing than lfs_setattr, since the attributes aren't written out immediately.
2018-04-06 04:23:14 +00:00
return lfs_dir_setattrs(lfs, &cwd, &entry, attrs, count);
Added simple custom attributes A much requested feature (mostly because of littlefs's notable lack of timestamps), this commits adds support for user-specified custom attributes. Planned (though underestimated) since v1, custom attributes provide a route for OSs and applications to provide their own metadata in littlefs, without limiting portability. However, unlike custom attributes that can be found on much more powerful PC filesystems, these custom attributes are very limited, intended for only a handful of bytes for very important metadata. Each attribute has only a single byte to identify the attribute, and the size of all attributes attached to a file is limited to 64 bytes. Custom attributes can be accessed through the lfs_getattr, lfs_setattr, and lfs_removeattr functions.
2018-04-06 00:03:58 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
/// Filesystem operations ///
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
static int lfs_init(lfs_t *lfs, const struct lfs_config *cfg) {
lfs->cfg = cfg;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup read cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->rcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->read_buffer) {
lfs->rcache.buffer = lfs->cfg->read_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->rcache.buffer = lfs_malloc(lfs->cfg->read_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->rcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup program cache
Adopted 0xffffffff as null pointer - Default value of most flash-based storage - Avoids 0 == superblock/dir issue - Usually causes assertions in bd driver layer - Easier to notice in hex dumps
2017-04-30 16:54:27 +00:00
lfs->pcache.block = 0xffffffff;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (lfs->cfg->prog_buffer) {
lfs->pcache.buffer = lfs->cfg->prog_buffer;
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->pcache.buffer = lfs_malloc(lfs->cfg->prog_size);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->pcache.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
}
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
// setup lookahead, round down to nearest 32-bits
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->lookahead % 32 == 0);
LFS_ASSERT(lfs->cfg->lookahead > 0);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
if (lfs->cfg->lookahead_buffer) {
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
lfs->free.buffer = lfs->cfg->lookahead_buffer;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
} else {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs->free.buffer = lfs_malloc(lfs->cfg->lookahead/8);
Fixed problem with lookaheads larger than block device Simply limiting the lookahead region to the size of the block device fixes the problem. Also added logic to limit the allocated region and floor to nearest word, since the additional memory couldn't really be used effectively.
2017-09-19 02:20:33 +00:00
if (!lfs->free.buffer) {
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_NOMEM;
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
}
}
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
// check that program and read sizes are multiples of the block size
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(lfs->cfg->prog_size % lfs->cfg->read_size == 0);
LFS_ASSERT(lfs->cfg->block_size % lfs->cfg->prog_size == 0);
Added asserts on geometry and updated config documentation littlefs had an unwritten assumption that the block device's program size would be a multiple of the read size, and the block size would be a multiple of the program size. This has already caused confusion for users. Added a note and assert to catch unexpected geometries early. Also found that the prog/erase functions indicated they must return LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors are found by CRC.
2018-01-11 17:56:09 +00:00
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
// check that the block size is large enough to fit ctz pointers
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
LFS_ASSERT(4*lfs_npw2(0xffffffff / (lfs->cfg->block_size-2*4))
Removed clamping to block size in ctz linked-list Initially, I was concerned that the number of pointers in the ctz linked-list could exceed the storage in a block. Long story short this isn't really possible outside of extremely small block sizes. Since clamping impacts the layout of files on disk, removing the block size removed quite a bit of logic and corner cases. Replaced with an assert on block size during initialization. --- Long story long, the minimum block size needed to store all ctz pointers in a filesystem can be found with this formula: B = (w/8)*log2(2^w / (B - 2*(w/8))) where: B = block size in bytes w = pointer width in bits It's not a very pretty formula, but does give us some useful info if we apply some math: min block size: 32 bit ctz linked-list = 104 bytes 64 bit ctz linked-list = 448 bytes For littlefs, 128 bytes is a perfectly reasonable minimum block size.
2017-10-10 23:48:24 +00:00
<= lfs->cfg->block_size);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// check that the size limits are sane
LFS_ASSERT(lfs->cfg->inline_size <= LFS_INLINE_MAX);
LFS_ASSERT(lfs->cfg->inline_size <= lfs->cfg->read_size);
lfs->inline_size = lfs->cfg->inline_size;
if (!lfs->inline_size) {
lfs->inline_size = lfs_min(LFS_INLINE_MAX, lfs->cfg->read_size);
}
LFS_ASSERT(lfs->cfg->attrs_size <= LFS_ATTRS_MAX);
lfs->attrs_size = lfs->cfg->attrs_size;
if (!lfs->attrs_size) {
lfs->attrs_size = LFS_ATTRS_MAX;
}
LFS_ASSERT(lfs->cfg->name_size <= LFS_NAME_MAX);
lfs->name_size = lfs->cfg->name_size;
if (!lfs->name_size) {
lfs->name_size = LFS_NAME_MAX;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// setup default state
lfs->root[0] = 0xffffffff;
lfs->root[1] = 0xffffffff;
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
lfs->files = NULL;
Added directory list for synchronizing in flight directories As it was, if a user operated on a directory while at the same time iterating over the directory, the directory objects could fall out of sync. In the best case, files may be skipped while removing everything in a file, in the worst case, a very poorly timed directory relocate could be missed. Simple fix is to add the same directory tracking that is currently in use for files, at a small code+complexity cost.
2017-11-22 02:53:15 +00:00
lfs->dirs = NULL;
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = false;
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
static int lfs_deinit(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// free allocated memory
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
if (!lfs->cfg->read_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->rcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
if (!lfs->cfg->prog_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->pcache.buffer);
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
}
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
if (!lfs->cfg->lookahead_buffer) {
Added more configurable utils Note: It's still expected to modify lfs_utils.h when porting littlefs to a new target/system. There's just too much room for system-specific improvements, such as taking advantage of CRC hardware. Rather, encouraging modification of lfs_util.h and making it easy to modify and debug should result in better integration with the consuming systems. This just adds a bunch of quality-of-life improvements that should help development and integration in littlefs. - Macros that require no side-effects are all-caps - System includes are only brought in when needed - Malloc/free wrappers - LFS_NO_* checks for quickly disabling things at the command line - At least a little-bit more docs
2018-01-29 21:20:12 +00:00
lfs_free(lfs->free.buffer);
Fixed memory leak for lookahead buffer
2017-04-29 17:50:23 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return 0;
}
int lfs_format(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create free lookahead
Fixed issue with aggressively rounding down lookahead configuration The littlefs allows buffers to be passed statically in the case that a system does not have a heap. Unfortunately, this means we can't round up in the case of an unaligned lookahead buffer. Double unfortunately, rounding down after clamping to the block device size could result in a lookahead of zero for block devices < 32 blocks large. The assert in littlefs does catch this case, but rounding down prevents support for < 32 block devices. The solution is to simply require a 32-bit aligned buffer with an assert. This avoids runtime problems while allowing a user to pass in the correct buffer for < 32 block devices. Rounding up can be handled at higher API levels.
2017-11-10 01:10:08 +00:00
memset(lfs->free.buffer, 0, lfs->cfg->lookahead/8);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = lfs_min(lfs->cfg->lookahead, lfs->cfg->block_count);
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Added support for the basic file operation Missing seek, but these are the core filesystem operations provided by this filesystem: - Read a file - Append to a file Additional work is needed around freeing the previous file, so right now it's limited to appending to existing files, a real append only filesystem. Unfortunately the overhead of the free list with multiple open files is becoming tricky.
2017-03-20 03:00:56 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// create superblock dir
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
lfs_dir_t superdir;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
err = lfs_dir_alloc(lfs, &superdir);
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
if (err) {
return err;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// write root directory
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
lfs_dir_t root;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
err = lfs_dir_alloc(lfs, &root);
if (err) {
return err;
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &root, NULL, 0);
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
Added dir tests, test fixes, config
2017-03-25 23:11:45 +00:00
lfs->root[0] = root.pair[0];
lfs->root[1] = root.pair[1];
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
superdir.d.tail[0] = lfs->root[0];
superdir.d.tail[1] = lfs->root[1];
Restructured the major interfaces of the filesystem
2017-03-25 21:20:31 +00:00
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
// write one superblock
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
lfs_superblock_t superblock;
superblock.d.version = LFS_DISK_VERSION,
superblock.d.root[0] = lfs->root[0];
superblock.d.root[1] = lfs->root[1];
superblock.d.block_size = lfs->cfg->block_size;
superblock.d.block_count = lfs->cfg->block_count;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
superblock.d.inline_size = lfs->inline_size;
superblock.d.attrs_size = lfs->attrs_size;
superblock.d.name_size = lfs->name_size;
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
lfs_entry_t superentry;
superentry.d.type = LFS_STRUCT_DIR | LFS_TYPE_SUPERBLOCK;
superentry.d.elen = sizeof(superblock.d);
superentry.d.alen = 0;
superentry.d.nlen = strlen("littlefs");
superentry.off = sizeof(superdir.d);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
superentry.size = 0;
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
lfs_entry_tole32(&superentry.d);
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_superblock_tole32(&superblock.d);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &superdir, &superentry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, 0, &superentry.d, 4},
{LFS_FROM_MEM, 0, 0, &superblock.d, sizeof(superblock.d)},
{LFS_FROM_MEM, 0, 0, "littlefs", superentry.d.nlen}}, 3);
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
if (err) {
return err;
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
// sanity check that fetch works
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
err = lfs_dir_fetch(lfs, &superdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
return lfs_deinit(lfs);
Initial commit of progress, minimal formatting niave free list
2017-02-27 00:05:27 +00:00
}
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
int lfs_mount(lfs_t *lfs, const struct lfs_config *cfg) {
int err = lfs_init(lfs, cfg);
if (err) {
return err;
}
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// setup free lookahead
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs->free.off = 0;
Fixed lookahead overflow and removed unbounded lookahead pointers As pointed out by davidefer, the lookahead pointer modular arithmetic does not work around integer overflow when the pointer size is not a multiple of the block count. To avoid overflow problems, the easy solution is to stop trying to work around integer overflows and keep the lookahead offset inside the block device. To make this work, the ack was modified into a resetable counter that is decremented every block allocation. As a plus, quite a bit of the allocation logic ended up simplified.
2018-04-10 20:14:27 +00:00
lfs->free.size = 0;
lfs->free.i = 0;
Fix incorrect lookahead population before ack Rather than tracking all in-flight blocks blocks during a lookahead, littlefs uses an ack scheme to mark the first allocated block that hasn't reached the disk yet. littlefs assumes all blocks since the last ack are bad or in-flight, and uses this to know when it's out of storage. However, these unacked allocations were still being populated in the lookahead buffer. If the whole block device fits in the lookahead buffer, _and_ littlefs managed to scan around the whole storage while an unacked block was still in-flight, it would assume the block was free and misallocate it. The fix is to only fill the lookahead buffer up to the last ack. The internal free structure was restructured to simplify the runtime calculation of lookahead size.
2018-02-08 07:30:21 +00:00
lfs_alloc_ack(lfs);
Cleaned up block allocator Removed scanning for stride - Adds complexity with questionable benefit - Can be added as an optimization later Fixed handling around device boundaries and where lookahead may not be a factor of the device size (consider small devices with only a few blocks) Added support for configuration with optional dynamic memory as found in the caching configuration
2017-04-22 19:56:12 +00:00
// load superblock
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
lfs_dir_t dir;
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (err) {
if (err == LFS_ERR_CORRUPT) {
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return err;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_entry_t entry = {.off = sizeof(dir.d)};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (err) {
return err;
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
lfs_superblock_t superblock;
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
memset(&superblock.d, 0, sizeof(superblock.d));
err = lfs_dir_get(lfs, &dir,
sizeof(dir.d)+4, &superblock.d,
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
lfs_min(sizeof(superblock.d), lfs_entry_elen(&entry)));
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
lfs_superblock_fromle32(&superblock.d);
if (err) {
return err;
}
Changed how we write out superblock to use append Making the superblock look like "just another entry" allows us to treat the superblock like "just another entry" and reuse a decent amount of logic that would otherwise only be used a format and mount time. In this case we can use append to write out the superblock like it was creating a new entry on the filesystem.
2018-03-23 21:11:36 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
char magic[8];
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
err = lfs_dir_get(lfs, &dir,
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
sizeof(dir.d)+lfs_entry_size(&entry)-entry.d.nlen, magic,
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
lfs_min(sizeof(magic), entry.d.nlen));
if (err) {
return err;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (memcmp(magic, "littlefs", 8) != 0) {
LFS_ERROR("Invalid superblock at %d %d", 0, 1);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_CORRUPT;
Added limited support for directories This comes with a lot of scafolding put into place around the core of the filesystem. Added operations: - append an entry to a directory - find an entry in a directory - iterate over entries in a directory Some to do: - Chaining multiple directory blocks - Recursion on directory operations
2017-03-13 00:41:08 +00:00
}
Add version info for software library and on-disk structures An annoying part of filesystems is that the software library can change independently of the on-disk structures. For this reason versioning is very important, and must be handled separately for the software and on-disk parts. In this patch, littlefs provides two version numbers at compile time, with major and minor parts, in the form of 6 macros. LFS_VERSION // Library version, uint32_t encoded LFS_VERSION_MAJOR // Major - Backwards incompatible changes LFS_VERSION_MINOR // Minor - Feature additions LFS_DISK_VERSION // On-disk version, uint32_t encoded LFS_DISK_VERSION_MAJOR // Major - Backwards incompatible changes LFS_DISK_VERSION_MINOR // Minor - Feature additions Note that littlefs will error if it finds a major version number that is different, or a minor version number that has regressed.
2018-01-26 20:26:25 +00:00
uint16_t major_version = (0xffff & (superblock.d.version >> 16));
uint16_t minor_version = (0xffff & (superblock.d.version >> 0));
if ((major_version != LFS_DISK_VERSION_MAJOR ||
minor_version > LFS_DISK_VERSION_MINOR)) {
LFS_ERROR("Invalid version %d.%d", major_version, minor_version);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
return LFS_ERR_INVAL;
Restructured directory code After quite a bit of prototyping, settled on the following functions: - lfs_dir_alloc - create a new dir - lfs_dir_fetch - load and check a dir pair from disk - lfs_dir_commit - save a dir pair to disk - lfs_dir_shift - shrink a dir pair to disk - lfs_dir_append - add a dir entry, creating dirs if needed - lfs_dir_remove - remove a dir entry, dropping dirs if needed Additionally, followed through with a few other tweaks
2017-04-18 03:27:06 +00:00
}
Added disk-backed limits on the name/attrs/inline sizes Being a portable, microcontroller-scale embedded filesystem, littlefs is presented with a relatively unique challenge. The amount of RAM available is on completely different scales from machine to machine, and what is normally a reasonable RAM assumption may break completely on an embedded system. A great example of this is file names. On almost every PC these days, the limit for a file name is 255 bytes. It's a very convenient limit for a number of reasons. However, on microcontrollers, allocating 255 bytes of RAM to do a file search can be unreasonable. The simplest solution (and one that has existing in littlefs for a while), is to let this limit be redefined to a smaller value on devices that need to save RAM. However, this presents an interesting portability issue. If these devices are plugged into a PC with relatively infinite RAM, nothing stops the PC from writing files with full 255-byte file names, which can't be read on the small device. One solution here is to store this limit on the superblock during format time. When mounting a disk, the filesystem implementation is responsible for checking this limit in the superblock. If it's larger than what can be read, raise an error. If it's smaller, respect the limit on the superblock and raise an error if the user attempts to exceed it. In this commit, this strategy is adopted for file names, inline files, and the size of all attributes, since these could impact the memory consumption of the filesystem. (Recording the attribute's limit is iffy, but is the only other arbitrary limit and could be used for disabling support of custom attributes). Note! This changes makes it very important to configure littlefs correctly at format time. If littlefs is formatted on a PC without changing the limits appropriately, it will be rejected by a smaller device.
2018-04-01 20:36:29 +00:00
if (superblock.d.inline_size) {
if (superblock.d.inline_size > lfs->inline_size) {
LFS_ERROR("Unsupported inline size (%d > %d)",
superblock.d.inline_size, lfs->inline_size);
return LFS_ERR_INVAL;
}
lfs->inline_size = superblock.d.inline_size;
}
if (superblock.d.attrs_size) {
if (superblock.d.attrs_size > lfs->attrs_size) {
LFS_ERROR("Unsupported attrs size (%d > %d)",
superblock.d.attrs_size, lfs->attrs_size);
return LFS_ERR_INVAL;
}
lfs->attrs_size = superblock.d.attrs_size;
}
if (superblock.d.name_size) {
if (superblock.d.name_size > lfs->name_size) {
LFS_ERROR("Unsupported name size (%d > %d)",
superblock.d.name_size, lfs->name_size);
return LFS_ERR_INVAL;
}
lfs->name_size = superblock.d.name_size;
}
lfs->root[0] = superblock.d.root[0];
lfs->root[1] = superblock.d.root[1];
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
int lfs_unmount(lfs_t *lfs) {
Added optional block-level caching This adds caching of the most recent read/program blocks, allowing support of devices that don't have byte-level read+writes, along with reduced device access on devices that do support byte-level read+writes. Note: The current implementation is a bit eager to drop caches where it simplifies the cache layer. This layer is already complex enough. Note: It may be worthwhile to add a compile switch for caching to reduce code size, note sure. Note: This does add a dependency on malloc, which could have a porting layer, but I'm just using the functions from stdlib for now. These can be overwritten with noops if the user controls the system, and keeps things simple for now.
2017-04-22 18:30:40 +00:00
return lfs_deinit(lfs);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Structured some of the bulk of the codebase - Removed lfs_config.h, distributed between lfs.h and lfs_util.h - Moved some functions that felt out of place
2017-04-24 02:40:03 +00:00
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// Internal filesystem filesystem operations ///
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
int lfs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
// iterate over metadata pairs
lfs_block_t cwd[2] = {0, 1};
while (true) {
for (int i = 0; i < 2; i++) {
int err = cb(data, cwd[i]);
if (err) {
return err;
}
}
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
lfs_dir_t dir;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
int err = lfs_dir_fetch(lfs, &dir, cwd);
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
// iterate over contents
Added size field to entry structure The size field is redundant, since an entry's size can be determined from the nlen+elen+alen+4. However, as you may have guessed from that expression, calculating the size this way is a bit roundabout and inefficient. Despite its redundancy, it's cheaper to store the size in the entry, though with a minor RAM cost. Note, extra care must now be taken to make sure these size and len fields don't fall out of sync.
2018-03-16 01:58:29 +00:00
lfs_entry_t entry;
Shrinked on-disk directory program size Directories still consume two full erase blocks, but now only program the exact on-disk region to store the directory contents. This results in a decent improvement in the amount of data written and read to the device when doing directory operations. Calculating the checksum of dynamically sized data is surprisingly tricky, since the size of the data could also contain errors. For the littlefs, we can assume the data size must fit in an erase block. If the data size is invalid, we can just treat the block as corrupted.
2017-06-24 01:03:44 +00:00
while (dir.off + sizeof(entry.d) <= (0x7fffffff & dir.d.size)-4) {
Added internal lfs_dir_get to consolidate logic for reading dir entries It's a relatively simple function but offers some code reuse as well as making the dir entry operations a bit more readable.
2018-03-23 23:35:55 +00:00
err = lfs_dir_get(lfs, &dir,
dir.off, &entry.d, sizeof(entry.d));
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
lfs_entry_fromle32(&entry.d);
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
if (err) {
return err;
}
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
dir.off += lfs_entry_size(&entry);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if ((0x70 & entry.d.type) == LFS_STRUCT_CTZ) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_ctz_traverse(lfs, &lfs->rcache, NULL,
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
entry.d.u.file.head, entry.d.u.file.size, cb, data);
Fixed non-standard behaviour of rdwr streams Originally had two seperate positions for reading/writing, but this is inconsistent with the the posix standard, which has a single position for reading and writing. Also added proper handling of when the file is dirty, just added an internal flag for this state. Also moved the entry out of the file struct, and rearranged some members to clean things up.
2017-04-24 04:39:50 +00:00
if (err) {
return err;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
}
cwd[0] = dir.d.tail[0];
cwd[1] = dir.d.tail[1];
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (lfs_pairisnull(cwd)) {
break;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
}
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
// iterate over any open files
for (lfs_file_t *f = lfs->files; f; f = f->next) {
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_DIRTY) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->head, f->size, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
Fixed a handful of bugs as result of testing
2018-03-19 01:36:48 +00:00
if ((f->flags & LFS_F_WRITING) && !(f->flags & LFS_F_INLINE)) {
Slight name change with ctz skip-list functions changed: lfs_index -> lfs_ctz_index lfs_index_find -> lfs_ctz_find lfs_index_append -> lfs_ctz_append lfs_index_traverse -> lfs_ctz_traverse
2017-10-18 05:41:43 +00:00
int err = lfs_ctz_traverse(lfs, &lfs->rcache, &f->cache,
Added caching with managed caches at the file level This adds a fully independent layer between the rest of the filesystem and the block device. This requires some additionally logic around cache invalidation and flushing, but removes the need for any higher layer to consider read/write sizes less than what is supported by the hardware. Additionally, these caches can be used for possible speed improvements. This is left up to the user to optimize for their use cases. For very limited embedded systems with byte-level read/writes, the caches could be omitted completely, or they could even be the size of a full block for minimizing storage access. (A full block may not be the best for speed, consider if only a small portion of the read block is used, but I'll leave that evaluation as an exercise for any consumers of this library)
2017-04-30 16:19:37 +00:00
f->block, f->pos, cb, data);
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
if (err) {
return err;
}
}
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added file list for tracking in flight allocations Needed primarily for tracking block allocations, unfortunately this prevents the freedom for the user to bitwise copy files.
2017-04-29 15:22:01 +00:00
return 0;
Added full dir list and rudimentary block allocator In writing the initial allocator, I ran into the rather difficult problem of trying to iterate through the entire filesystem cheaply and with only constant memory consumption (which prohibits recursive functions). The solution was to simply thread all directory blocks onto a massive linked-list that spans the entire filesystem. With the linked-list it was easy to create a traverse function for all blocks in use on the filesystem (which has potential for other utility), and add the rudimentary block allocator using a bit-vector. While the linked-list may add complexity (especially where needing to maintain atomic operations), the linked-list helps simplify what is currently the most expensive operation in the filesystem, with no cost to space (the linked-list can reuse the pointers used for chained directory blocks).
2017-04-01 15:44:17 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_pred(lfs_t *lfs, const lfs_block_t dir[2], lfs_dir_t *pdir) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
// iterate over all directory directory entries
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
int err = lfs_dir_fetch(lfs, pdir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
while (!lfs_pairisnull(pdir->d.tail)) {
if (lfs_paircmp(pdir->d.tail, dir) == 0) {
return true;
}
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, pdir, pdir->d.tail);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
}
return false;
}
static int lfs_parent(lfs_t *lfs, const lfs_block_t dir[2],
lfs_dir_t *parent, lfs_entry_t *entry) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Added conversion to/from little-endian on disk Required to support big-endian processors, with the most notable being the PowerPC architecture. On little-endian architectures, these conversions can be optimized out and have no code impact. Initial patch provided by gmouchard
2018-02-02 11:58:43 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
parent->d.tail[0] = 0;
parent->d.tail[1] = 1;
// iterate over all directory directory entries
while (!lfs_pairisnull(parent->d.tail)) {
int err = lfs_dir_fetch(lfs, parent, parent->d.tail);
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, parent, entry);
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err && err != LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return err;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Standardized error values Now matches the commonly used errno codes in name with the value encoded as the negative errno code
2017-04-24 03:10:16 +00:00
if (err == LFS_ERR_NOENT) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
break;
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (((0x70 & entry->d.type) == LFS_STRUCT_DIR) &&
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_paircmp(entry->d.u.dir, dir) == 0) {
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
return true;
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
}
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
return false;
}
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
static int lfs_moved(lfs_t *lfs, const void *e) {
if (lfs_pairisnull(lfs->root)) {
return 0;
}
// skip superblock
lfs_dir_t cwd;
int err = lfs_dir_fetch(lfs, &cwd, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
// iterate over all directory directory entries
lfs_entry_t entry;
while (!lfs_pairisnull(cwd.d.tail)) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (!(LFS_STRUCT_MOVED & entry.d.type) &&
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
memcmp(&entry.d.u, e, sizeof(entry.d.u)) == 0) {
return true;
}
}
}
return false;
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
static int lfs_relocate(lfs_t *lfs,
const lfs_block_t oldpair[2], const lfs_block_t newpair[2]) {
// find parent
lfs_dir_t parent;
lfs_entry_t entry;
int res = lfs_parent(lfs, oldpair, &parent, &entry);
if (res < 0) {
return res;
}
if (res) {
// update disk, this creates a desync
entry.d.u.dir[0] = newpair[0];
entry.d.u.dir[1] = newpair[1];
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
lfs_entry_tole32(&entry.d);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
int err = lfs_dir_set(lfs, &parent, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, sizeof(entry.d),
&entry.d, sizeof(entry.d)}}, 1);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// update internal root
if (lfs_paircmp(oldpair, lfs->root) == 0) {
LFS_DEBUG("Relocating root %d %d", newpair[0], newpair[1]);
lfs->root[0] = newpair[0];
lfs->root[1] = newpair[1];
}
// clean up bad block, which should now be a desync
return lfs_deorphan(lfs);
}
// find pred
res = lfs_pred(lfs, oldpair, &parent);
if (res < 0) {
return res;
}
if (res) {
// just replace bad pair, no desync can occur
parent.d.tail[0] = newpair[0];
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
parent.d.tail[1] = newpair[1];
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
return lfs_dir_commit(lfs, &parent, NULL, 0);
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
// couldn't find dir, must be new
return 0;
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Added proper handling of orphans Unfortunately, threading all dir blocks in a linked-list did not come without problems. While it's possible to atomically add a dir to the linked list (by adding the new dir into the linked-list position immediately after it's parent, requiring only one atomic update to the parent block), it is not easy to make sure the linked-list is in a state that always allows atomic removal of dirs. The simple solution is to allow this non-atomic removal, with an additional step to remove any orphans that could have been created by a power-loss. This deorphan step is only run if the normal allocator has failed.
2017-04-01 17:23:15 +00:00
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
int lfs_deorphan(lfs_t *lfs) {
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs->deorphaned = true;
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (lfs_pairisnull(lfs->root)) {
return 0;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
lfs_dir_t pdir = {.d.size = 0x80000000};
lfs_dir_t cwd = {.d.tail[0] = 0, .d.tail[1] = 1};
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// iterate over all directory directory entries
while (!lfs_pairisnull(cwd.d.tail)) {
int err = lfs_dir_fetch(lfs, &cwd, cwd.d.tail);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
if (err) {
return err;
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check head blocks for orphans
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (!(0x80000000 & pdir.d.size)) {
// check if we have a parent
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
lfs_dir_t parent;
lfs_entry_t entry;
int res = lfs_parent(lfs, pdir.d.tail, &parent, &entry);
if (res < 0) {
return res;
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (!res) {
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
// we are an orphan
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found orphan %d %d",
pdir.d.tail[0], pdir.d.tail[1]);
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
pdir.d.tail[0] = cwd.d.tail[0];
pdir.d.tail[1] = cwd.d.tail[1];
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
if (err) {
return err;
}
break;
}
if (!lfs_pairsync(entry.d.u.dir, pdir.d.tail)) {
// we have desynced
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
LFS_DEBUG("Found desync %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added support for handling corrupted blocks This provides a limited form of wear leveling. While wear is not actually balanced across blocks, the filesystem can recover from corrupted blocks and extend the lifetime of a device nearly as much as dynamic wear leveling. For use-cases where wear is important, it would be better to use a full form of dynamic wear-leveling at the block level. (or consider a logging filesystem). Corrupted block handling was simply added on top of the existing logic in place for the filesystem, so it's a bit more noodly than it may have to be, but it gets the work done.
2017-05-14 17:01:45 +00:00
pdir.d.tail[0] = entry.d.u.dir[0];
pdir.d.tail[1] = entry.d.u.dir[1];
Expanded inline files up to a limit of 1023 bytes One of the big benefits of inline files is that small files no longer need to take up a full block. This opens up an opportunity to provide much better support for storage devices with only a handful of very large blocks. Such as the internal flash found on most microcontrollers. After investigating some use cases for a filesystem on internal flash, it has become apparent that the 255-byte limit is going to be too restrictive to be useful in many cases. Most uses I found needed files ~4-64 bytes in size, but it wasn't uncommon to find files ~512 bytes in length. To try to remedy this, I've pushed the 255 byte limit up to 1023 bytes, by stealing some bits from the previously-unused attributes's size. Unfortunately this limits attributes to 63 bytes in total and has a minor code cost, but I'm not sure even 1023 bytes will be sufficient for a lot of cases. The littlefs will probably never be as efficient with internal flash as other filesystems such as SPIFFS, it just wasn't designed for this sort of limited geometry. However, this feature has been heavily requested, even with limitations, because of the opportunity for code reuse on microcontrollers with both internal and external flash.
2018-04-03 13:28:09 +00:00
err = lfs_dir_commit(lfs, &pdir, NULL, 0);
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
if (err) {
return err;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
Fixed allocation bugs near the end of storage Also added better testing specifically for these corner cases around the end of storage
2017-04-29 17:41:53 +00:00
break;
}
Moved to brute-force deorphan without parent pointers Removing the dependency to the parent pointer solves many issues with non-atomic updates of children's parent pointers with respect to any move operations. However, this comes with an embarrassingly terrible runtime as the only other option is to exhaustively check every dir entry to find a child's parent. Fortunately, deorphaning should be a relatively rare operation.
2017-04-14 22:33:36 +00:00
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
// check entries for moves
lfs_entry_t entry;
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
while (true) {
Silence shadow warnings
2018-01-29 19:53:28 +00:00
err = lfs_dir_next(lfs, &cwd, &entry);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err && err != LFS_ERR_NOENT) {
return err;
}
if (err == LFS_ERR_NOENT) {
break;
}
// found moved entry
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
if (entry.d.type & LFS_STRUCT_MOVED) {
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
int moved = lfs_moved(lfs, &entry.d.u);
if (moved < 0) {
return moved;
}
if (moved) {
LFS_DEBUG("Found move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Modified commit regions to work better with custom attributes Mostly just removed LFS_FROM_DROP and changed the DSL grammar a bit to allow drops to occur naturally through oldsize -> newsize diff expressed in the region struct. This prevents us from having to add a drop every time we want to update an entry in-place.
2018-04-08 09:23:23 +00:00
{LFS_FROM_MEM, 0, entry.size, NULL, 0}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
} else {
LFS_DEBUG("Found partial move %d %d",
entry.d.u.dir[0], entry.d.u.dir[1]);
Separated type/struct fields in dir entries The separation of data-structure vs entry type has been implicit for a while now, and even taken advantage of to simplify the traverse logic. Explicitely separating the data-struct and entry types allows us to introduce new data structures (inlined files).
2018-03-03 16:26:06 +00:00
entry.d.type &= ~LFS_STRUCT_MOVED;
Added internal lfs_dir_set, an umbrella to dir append/update/remove operations This move was surprisingly complex, but offers the ultimate opportunity for code reuse in terms of resizable entries. Instead of needing to provide separate functions for adding and removing entries, adding and removing entries can just be viewed as changing an entry's size to-and-from zero. Unfortunately, it's not _quite_ that simple, since append and remove hide some relatively complex operations for when directory blocks overflow or need to be cleaned up. However, with enough shoehorning, and a new committer type that allows specifying recursive commit lists (is this now a push-down automata?), it does seem to be possible to shove all of the entry update logic into a single function. Sidenote, I switched back to an enum-based DSL, since the addition of a recursive region opcode breaks the consistency of what needs to be passed to the DSL callback functions. It's much simpler to handle each opcode explicitly inside a recursive lfs_commit_region function.
2018-03-27 22:57:07 +00:00
err = lfs_dir_set(lfs, &cwd, &entry, (struct lfs_region[]){
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
{LFS_FROM_MEM, 0, 1, &entry.d, 1}}, 1);
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
if (err) {
return err;
}
}
}
}
Refactored deduplicate/deorphan step to single deorphan step Deduplication and deorphan steps aren't required under indentical conditions, but they can be processed in the same iteration of the filesystem. Since lfs_alloc (requires deorphan) occurs on most write calls to the filesystem (requires deduplication), it was simpler to just compine the steps into a single lfs_deorphan step. Also traded out the places where lfs_rename/lfs_remove just defer operations to the deorphan step. This adds a bit of code, but also significantly speeds up directory operations.
2017-10-07 21:56:00 +00:00
memcpy(&pdir, &cwd, sizeof(pdir));
Added atomic move using dirty tag in entry type The "move problem" has been present in littlefs for a while, but I haven't come across a solution worth implementing for various reasons. The problem is simple: how do we move directory entries across directories atomically? Since multiple directory entries are involved, we can't rely entirely on the atomic block updates. It ends up being a bit of a puzzle. To make the problem more complicated, any directory block update can fail due to wear, and cause the directory block to need to be relocated. This happens rarely, but brings a large number of corner cases. --- The solution in this patch is simple: 1. Mark source as "moving" 2. Copy source to destination 3. Remove source If littlefs ever runs into a "moving" entry, that means a power loss occured during a move. Either the destination entry exists or it doesn't. In this case we just search the entire filesystem for the destination entry. This is expensive, however the chance of a power loss during a move is relatively low.
2017-10-07 14:19:08 +00:00
}
return 0;
}
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
/// External filesystem filesystem operations ///
int lfs_fs_getattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
lfs_dir_t dir;
int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
lfs_entry_t entry = {.off = sizeof(dir.d)};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (err) {
return err;
}
entry.size = lfs_entry_size(&entry);
return lfs_dir_getattrs(lfs, &dir, &entry, attrs, count);
}
int lfs_fs_setattrs(lfs_t *lfs, const struct lfs_attr *attrs, int count) {
lfs_dir_t dir;
int err = lfs_dir_fetch(lfs, &dir, (const lfs_block_t[2]){0, 1});
if (err) {
return err;
}
lfs_entry_t entry = {.off = sizeof(dir.d)};
Fixed big-endian support again This is what I get for not runing CI on a local development branch.
2018-04-10 21:35:29 +00:00
err = lfs_dir_get(lfs, &dir, entry.off, &entry.d, 4);
Added file-level and fs-level custom attribute APIs In the form of lfs_file_setattr, lfs_file_getattr, lfs_fs_setattr, lfs_fs_getattr. This enables atomic updates of custom attributes as described in 6c754c8, and provides a custom attribute API that allows custom attributes to be stored on the filesystem itself.
2018-04-08 21:58:12 +00:00
if (err) {
return err;
}
entry.size = lfs_entry_size(&entry);
return lfs_dir_setattrs(lfs, &dir, &entry, attrs, count);
}
Added lfs_fs_size for finding a count of used blocks This has existed for some time in the form of the lfs_traverse function, through which a user could provide a simple callback that would just count the number of blocks lfs_traverse finds. However, this approach is relatively unconventional and has proven to be confusing for most users.
2018-04-09 03:25:58 +00:00
static int lfs_fs_size_count(void *p, lfs_block_t block) {
lfs_size_t *size = p;
*size += 1;
return 0;
}
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
lfs_size_t size = 0;
int err = lfs_traverse(lfs, lfs_fs_size_count, &size);
if (err) {
return err;
}
return size;
}
Reference in New Issue Copy Permalink