Adds to support below three commands to operate reserved blocks
in compressed inode.
- get_cblocks
- release_cblocks
- reserve_cblocks
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
clock_t time is per-process time, which is not wall time. For example,
if the CPU is shared by other processes, clock_t time may advance slower
than wall clock. On the other hand, if the current process is
multithreaded and more than one execution core is available, clock_t
time may advance faster than wall clock.
this CL changes it to use CLOCK_BOOTTIME (Linux-specific)
Signed-off-by: Wei Wang <wvw@google.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This is no longer a transitive include of android_filesystem_config.h
Signed-off-by: Tom Cherry <tomcherry@google.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This adds support for casefolded and encrypted directories.
Fsck cannot check the hashes of such directories because it would
require access to the encryption key to generate the siphash
Signed-off-by: Daniel Rosenberg <drosen@google.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As Ondřej Jirman <megi@xff.cz> reported:
I was trying to run: fsck.f2fs --dry-run /dev/mmcblk0p2 on a RO mounted device,
and fsck refuses to run. Strace shows that it tries to open the block device
with O_EXCL even in RO mode, which will always fail if the block device
is mounted.
fsck.f2fs --dry-run /dev/mmcblk0p2
Info: Dry run
Info: Mounted device!
Info: Check FS only on RO mounted device
Error: Failed to open the device!
I suggest not using O_EXCL for --dry-run check.
Let's change to allow --dry-run to check readonly mounted fs.
Reported-by: Ondřej Jirman <megi@xff.cz>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
resize.f2fs has already supported large_nat_bitmap feature, but has no
option to turn on it.
This change add a new '-i' option to control turning on it.
Signed-off-by: xiongping1 <xiongping1@xiaomi.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds to print all valid blkaddrs in inode's i_addr field,
besides, it also adds to print meaning of specific blkaddr(flag).
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds to support compression, introducing '-O compression'
option to enable this feature in image.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Added command line options -c <num_cache_entry> and -m <max_hash_collision>
to activate cache for fsck. It may significantly speed up fsck.
Signed-off-by: Robin Hsu <robinhsu@google.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Implemented cache options in F2FS configuration 'c' (i.e. user
options):
* use c.cache_config.num_cache_entry to set the number of
cache entries (in block), minimum 1024, or 0 to disable cache.
* use c.cache_config.max_hash_collision to set maximum hash
collision (max 16).
* use c.cavhe_config.dbg_en to enable printing of debug messages.
Cache properties:
* Per block save based (block size = f2fs block size = 4K Bytes)
* Device block indices are used to hash (simple modular hash
function) to cache indices.
* A hash collision (when a block is hashed to an already occupied
cache block) will trigger offset-based relocation to max 16
other places, at first empty slot found policy.
* A hash collision with all 17 possible relocating places occupied
will evict the oldest used one (replaced by the new read block).
A global 'use' tick is kept ticking-up per block read for this
purpose.
* On write cache hit, it will update the corresponding cached
block.
* On write cache miss, the data won't be cached.
* Write-through policy only: never delayed write.
* On read cache hit, it returns the cached block.
* On read cache miss, it issues an I/O read to the kernel, and
then caches and returns the block.
* Currently read ahead is not implemented. Yet, the read ahead
logic is kept as before, i.e. using kernel's read ahead
mechanism.
Signed-off-by: Robin Hsu <robinhsu@google.com>
[Jaegeuk Kim: Fix one bug in dcache_update_rw()]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Given this option, fsck.f2fs does not run fsck forcefully, even if kernel
is updated. Android devices will do --kernel-check by default, while others
will not.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Current Makefile.am included the 'include' directory
directly as -I./include, but this doesn't guarantee that
said directory exists when building outside of the source
directory.
By adding $(srcdir) instead of '.' this gets fixed automatically.
Signed-off-by: Santiago Hormazabal <santiagohssl@gmail.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch tries to fix memory leak problem reported in Android.
Fixed the following problems in fsck.f2fs, make_f2fs and sload_f2fs:
* reuse of same pointer without clean-up
* exit on error without clean-up
Signed-off-by: Robin Hsu <robinhsu@google.com>
[Jaegeuk Kim: add missing definition to avoid build error]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
ckpt->entries is initialized by fsck_init(), but we tried to access it during
f2fs_do_mount().
The call sequence is:
- f2fs_do_mount
- record_fsync_data
- traverse_dnodes
- do_record_fsync_data
- ADDRS_PER_PAGE
- get_node_info
- node_info_from_raw_nat(fsck->entries[nid])
- do_fsck
- fsck_init
- build_nat_area_bitmap
- fsck->entries = calloc(fsck->nr_nat_entries);
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To catch bugs in write pointer handling code for zoned block devices,
have fsck check consistency of write pointers of non-open zones, that
current segments do not point to. Check two items comparing write pointer
positions with valid block maps in SIT.
The first item is check for zones with no valid blocks. When there is no
valid blocks in a zone, the write pointer should be at the start of the
zone. If not, next write operation to the zone will cause unaligned write
error. If write pointer is not at the zone start, reset the zone to move
the write pointer to the zone start.
The second item is check between write pointer position and the last
valid block in the zone. It is unexpected that the last valid block
position is beyond the write pointer. In such a case, report as the bug.
Fix is not required for such zone, because the zone is not selected for
next write operation until the zone get discarded.
In the same manner as the consistency check for current segments, do the
check and fix twice: at the beginning of do_fsck() to avoid unaligned
write error during fsck, and at fsck_verify() to reflect meta data
updates by fsck.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
On sudden f2fs shutdown, write pointers of zoned block devices can go
further but f2fs meta data keeps current segments at positions before the
write operations. After remounting the f2fs, this inconsistency causes
write operations not at write pointers and "Unaligned write command"
error is reported.
To avoid the error, have f2fs.fsck check consistency of write pointers
of open zones that current segments point to. Compare each current
segment's position and the write pointer position of the open zone. If
inconsistency is found and 'fix_on' flag is set, assign a new zone to the
current segment and check the newly assigned zone has write pointer at
the zone start. Leave the original zone as is to keep data recorded in
it.
To care about fsync data, refer each seg_entry's ckpt_valid_map to get
the last valid block in the zone. If the last valid block is beyond the
current segments position, fsync data exits in the zone. In case fsync
data exists, do not assign a new zone to the current segment not to lose
the fsync data. It is expected that the kernel replay the fsync data and
fix the write pointer inconsistency at mount time.
Also check consistency between write pointer of the zone the current
segment points to with valid block maps of the zone. If the last valid
block is beyond the write pointer position, report to indicate a bug. If
'fix_on' flag is set, assign a new zone to the current segment.
When inconsistencies are found, turn on 'bug_on' flag in fsck_verify() to
ask users to fix them or not. When inconsistencies get fixed, turn on
'force' flag in fsck_verify() to enforce fixes in following checks.
This check and fix is done twice. The first is done at the beginning of
do_fsck() function so that other fixes can reflect the current segment
modification. The second is done in fsck_verify() to reflect updated meta
data by other fixes.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Fsck checks fsync data when UMOUNT flag is not set. When the f2fs was not
cleanly unmouted, UMOUNT flag is not recorded in meta data and fsync data
can be left in the f2fs. The first fsck run checks fsync data to reflect
it on quota status recovery. After that, fsck writes UMOUNT flag in the
f2fs meta data so that second fsck run can skip fsync data check.
However, fsck for zoned block devices need to care fsync data for all
fsck runs. The first fsck run checks fsync data, then fsck can check
write pointer consistency with fsync data. However, since second fsck run
does not check fsync data, fsck detects write pointer at fsync data end
is not consistent with f2fs meta data. This results in meta data update
by fsck and fsync data gets lost.
To have fsck check fsync data always for zoned block devices, introduce
need_fsync_data_record() helper function which returns boolean to tell
if fsck needs fsync data check or not. For zoned block devices, always
return true. Otherwise, return true if UMOUNT flag is not set in CP.
Replace UMOUNT flag check codes for fsync data with the function call.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
When fsck updates one of the current segments, update_curseg_info() is
called specifying a single current segment as its argument. However,
update_curseg_info() calls move_curseg_info() function which updates all
six current segments. Then update_curseg_info() for a single current
segment moves all current segments.
This excessive current segment move causes an issue when a new zone is
assigned to a current segment because of write pointer inconsistency.
Even when a current segment has write pointer inconsistency, all other
current segments should not be moved because they may have fsync data
at their positions.
To avoid the excessive current segment move, introduce
move_one_curseg_info() function which does same work as
move_curseg_info() only for a single current segment. Call
move_one_curseg_info() in place of move_curseg_info() from
update_curseg_info().
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
When fsck needs to assign a new area to a curreng segment, it calls
find_next_free_block() function to find a new block to assign. For zoned
block devices, fsck checks write pointer consistency with current
segments' positions. In case a curseg is inconsistent with the
write pointer of the zone it points to, fsck should assign not a new free
block but a new free zone/section with write pointer at the zone start,
so that next write to the current segment succeeds without error.
To extend find_next_free_block() function's capability to find not only
a block but also a zone/section, add new_sec flag to
find_next_free_block() function. When new_sec flag is true, skip check
for each block's availability so that the check is done with unit of
section. Note that it is ensured that one zone has one section for f2fs
on zoned block devices. Then the logic to find a new free section is good
to find a new free zone.
When fsck target devices have ZONED_HM model, set new_sec flag true to
call find_next_free_block() from move_curseg_info(). Set curseg's
alloc_type not SSR but LFS for the devices with ZONED_HM model, because
SSR block allocation is not allowed for zoned block devices. Also skip
relocate_curseg_offset() for the devices with ZONED_HM model for the
same reason.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency fix by fsck, add
f2fs_reset_zone() helper function which calls RESET ZONE command. The
function is added to lib/libf2fs_zoned which gathers zoned block device
related functions.
When f2fs-tools are built without blkzoned.h kernel header, the helper
function f2fs_reset_zone() prints an error message as other helper
functions in lib/libf2fs_zoned print. To make the message consistent
through the all helper functions, modify message strings in
f2fs_check_zones() and f2fs_reset_zones().
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency check by fsck, add
f2fs_report_zone() helper function which calls REPORT ZONE command to
get write pointer status of a single zone. The function is added to
lib/libf2fs_zoned which gathers zoned block device related functions.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency check by fsck, add
f2fs_report_zones() helper function which calls REPORT ZONE command to
get write pointer status. The function is added to lib/libf2fs_zoned
which gathers zoned block device related functions.
To check write pointer consistency with f2fs meta data, fsck needs to
refer both of reported zone information and f2fs super block structure
"f2fs_sb_info". However, libf2fs_zoned does not import f2fs_sb_info. To
keep f2fs_sb_info structure out of libf2fs_zoned, provide a callback
function in fsck to f2fs_report_zones() and call it for each zone.
Add SECTOR_SHIFT definition in include/f2fs_fs.h to avoid a magic number
to convert bytes into 512B sectors.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
For multi-device F2FS, we should check if the sum of total_segments from
all devices matches segment_count.
Signed-off-by: Qiuyang Sun <sunqiuyang@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Commit d56232bc16 hardcoded -lblkid in
libf2fs_format_la_LDFLAGS which breaks the build with --without-blkid
option so use ${libblkid_LIBS} instead
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Add a copy command to f2fs_io, to allow testing direct I/O writes where
the source page is from a different file (mmap) or an internal kernel
page (sendfile). It could be useful for other tests in the future too.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Add and use helper functions for:
- Printing an error message (optionally with errno) and exiting.
- Allocating memory, exiting on error.
- Opening a file, exiting on error.
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This release includes:
- enable VERITY by default on Android
- introduce some preen mode in fsck.f2fs
- add f2fs_io tool
- add casefolding support
And, there are lots of bug fixes.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Previously, we don't allow block allocation on unclean umounted image,
result in failing to repair quota system file.
In this patch, we port most recovery codes from kernel to userspace
tools, so that on unclean image, during fsck initialization, we will
record all data/node block address we may recover in kernel, and
then during allocation of quota file repair, we can skip those blocks
to avoid block use conflict.
Eventually, if free space is enough, we can repair the quota system
file on an unclean umounted image.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
[Jaegeuk Kim: remove unnecessary parameter]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Change logic as below:
- fix to account block/node/inode stats correctly in reserve_new_block()
- check overflow in reserve_new_block()
- move stat update from f2fs_alloc_nid() to reserve_new_block()
- adjust write_checkpoint() to update stat for sload/fsck
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
if we new node block in fsck flow, we need to update
the valid_node_cnt at the same time.
Signed-off-by: Lihong Kou <koulihong@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
inode.i_blocks includes inode, xnode and data block count, so, only
fix in below condition:
- i_blocks := 3 (inode + xnode + data_block)
- i_blocks := 2 (inode + data_block)
In addition, it recovers symlink's i_size to 4k rather than i_blocks *
4k.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We should always set c.bug_on whenever found a bug, then fix them
if c.fix_on is on, otherwise, some bugs won't be shown unless we
enable debug log.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Propagate correct error number from write_dquots() to
quota_write_inode().
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
For defrag, resize, sload tools, let's advise to mount unclean
image to replay journal first in order to not lose any fsynced
data.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
During block allocation in large free section, we need to change
all sub segments' type in it, otherwise, we will fail to allocate
block in non-first segment due to mismatch seg-type.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Just cleanup, no logic change, besides, it can be reused by latter
patch.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Introduce f2fs_ra_meta_pages() to readahead meta pages like we did
in kernel.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Like we did in kernel, allocating memory in batch will be more
efficient.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We missed to free sit_i->sentries in f2fs_do_umount(), fix it.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
In error path of build_sit_info(), start variable is unsigned int type,
it should never be less than zero, fix it.
build_sit_info()
{
...
unsigned int start;
...
free_validity_maps:
for (--start ; start >= 0; --start)
free(sit_i->sentries[start].cur_valid_map);
...
}
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
