NVM Express Zoned Namespace (ZNS) devices can have zone-capacity(zc) less
than the zone-size. ZNS defines a per zone capacity which can be equal
or less than the zone-size. Zone-capacity is the number of usable blocks
in the zone. If zone-capacity is less than zone-size, then the segments
which start at/after zone-capacity are considered unusable. Only those
segments which start before the zone-capacity are considered as usable
and added to the free_segment_count and free_segment_bitmap of the kernel.
In such cases, the filesystem should not write/read beyond the
zone-capacity.
Update the super block with the usable number of blocks and free segment
count in the ZNS device zones, if zone-capacity is less than zone-size.
Set reserved segment count and overprovision ratio based on the usable
segments in the zone.
Allow fsck to find the free_segment_count based on the zone-capacity and
compare with checkpoint values.
Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
[Jaegeuk Kim: add UNUSED to is_usable_seg()]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
generic/339 reports below assertion on image w/ compression feature
enabled.
[ASSERT] (f2fs_check_dirent_position:1366) -->
Wrong position of dirent pino:4521, name:"....", level:9, dir_level:0,
pgofs:1880, correct range:[1882, 1883]
The root cause is we calculate blkaddr number in direct node
incorrectly for directory inode, since during calculation, we only
need align blkaddr number to cluster size for regular inode rather
than directory inode, let's fix it.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As Ondřej Jirman <megi@xff.cz> reported:
I was trying to run: fsck.f2fs --dry-run /dev/mmcblk0p2 on a RO mounted device,
and fsck refuses to run. Strace shows that it tries to open the block device
with O_EXCL even in RO mode, which will always fail if the block device
is mounted.
fsck.f2fs --dry-run /dev/mmcblk0p2
Info: Dry run
Info: Mounted device!
Info: Check FS only on RO mounted device
Error: Failed to open the device!
I suggest not using O_EXCL for --dry-run check.
Let's change to allow --dry-run to check readonly mounted fs.
Reported-by: Ondřej Jirman <megi@xff.cz>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds to support compression, introducing '-O compression'
option to enable this feature in image.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Implemented cache options in F2FS configuration 'c' (i.e. user
options):
* use c.cache_config.num_cache_entry to set the number of
cache entries (in block), minimum 1024, or 0 to disable cache.
* use c.cache_config.max_hash_collision to set maximum hash
collision (max 16).
* use c.cavhe_config.dbg_en to enable printing of debug messages.
Cache properties:
* Per block save based (block size = f2fs block size = 4K Bytes)
* Device block indices are used to hash (simple modular hash
function) to cache indices.
* A hash collision (when a block is hashed to an already occupied
cache block) will trigger offset-based relocation to max 16
other places, at first empty slot found policy.
* A hash collision with all 17 possible relocating places occupied
will evict the oldest used one (replaced by the new read block).
A global 'use' tick is kept ticking-up per block read for this
purpose.
* On write cache hit, it will update the corresponding cached
block.
* On write cache miss, the data won't be cached.
* Write-through policy only: never delayed write.
* On read cache hit, it returns the cached block.
* On read cache miss, it issues an I/O read to the kernel, and
then caches and returns the block.
* Currently read ahead is not implemented. Yet, the read ahead
logic is kept as before, i.e. using kernel's read ahead
mechanism.
Signed-off-by: Robin Hsu <robinhsu@google.com>
[Jaegeuk Kim: Fix one bug in dcache_update_rw()]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Given this option, fsck.f2fs does not run fsck forcefully, even if kernel
is updated. Android devices will do --kernel-check by default, while others
will not.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch tries to fix memory leak problem reported in Android.
Fixed the following problems in fsck.f2fs, make_f2fs and sload_f2fs:
* reuse of same pointer without clean-up
* exit on error without clean-up
Signed-off-by: Robin Hsu <robinhsu@google.com>
[Jaegeuk Kim: add missing definition to avoid build error]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency fix by fsck, add
f2fs_reset_zone() helper function which calls RESET ZONE command. The
function is added to lib/libf2fs_zoned which gathers zoned block device
related functions.
When f2fs-tools are built without blkzoned.h kernel header, the helper
function f2fs_reset_zone() prints an error message as other helper
functions in lib/libf2fs_zoned print. To make the message consistent
through the all helper functions, modify message strings in
f2fs_check_zones() and f2fs_reset_zones().
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency check by fsck, add
f2fs_report_zone() helper function which calls REPORT ZONE command to
get write pointer status of a single zone. The function is added to
lib/libf2fs_zoned which gathers zoned block device related functions.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To prepare for write pointer consistency check by fsck, add
f2fs_report_zones() helper function which calls REPORT ZONE command to
get write pointer status. The function is added to lib/libf2fs_zoned
which gathers zoned block device related functions.
To check write pointer consistency with f2fs meta data, fsck needs to
refer both of reported zone information and f2fs super block structure
"f2fs_sb_info". However, libf2fs_zoned does not import f2fs_sb_info. To
keep f2fs_sb_info structure out of libf2fs_zoned, provide a callback
function in fsck to f2fs_report_zones() and call it for each zone.
Add SECTOR_SHIFT definition in include/f2fs_fs.h to avoid a magic number
to convert bytes into 512B sectors.
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This adds support for f2fs casefolding. Similarly to ext4 casefolding,
this is controlled per-folder via the +F attribute. It can be toggled on
empty directories only. It is not currently compatible with encryption,
but that will likely change.
When enabling the casefold feature, use the -C flag. The format is:
-C encoding[:flag1,flag2,etc]
Signed-off-by: Daniel Rosenberg <drosen@google.com>
[Jaegeuk Kim: print "casefold" in sb->feature]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
When the -m option is specified to format a Zoned device,
do not fall back to the non-zoned mode in case information
about the device is not found.
Explicitly specify this error to the user.
Signed-off-by: Surbhi Palande <csurbhi@gmail.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
In generic/38[3456], we use quotactl(2) to check if prjquota is
enabled on the given device (src/feature -P $dev in _require_prjquota),
and quotactl(2) requires the given device is a mounted device.
So it requires dump.f2fs to list/check enabled features on a mounted
device, let's relieve to allow such operation.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Calculate inode checksum and writeback when processing
sload or resize.
Signed-off-by: Xue Liu <liuxueliu.liu@huawei.com>
[Chao Yu: change commit message and code]
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
A partition device does not have the "zoned" nor "chunk_sectors" sysfs
attribute files. Only the owner block device of the partition has these
files. This causes the detection of the zoned model and zone size of a
partition device to fail when executing mkfs.f2fs.
Fix this problem by using the owner device sysfs directory as the base
directory for accessing the zoned and chunk_sectors files. This is done
by using the device major:minor symbolic link under the /sys/dev/block
directory, reading this link and removing the partition device name from
the link path for a partition device (which is indicated by the presence
of the "partition" file under the directory).
Also add a check for the ENOENT error when opening the device "zoned"
sysfs attribute file. The absence of this file indicates that the
kernel does not support zoned block devices. Since the device file is
already open, it exists, and so the device can safely be assumed as not
being zoned.
Changes from v2:
* Addressed Chao Yu's comment on snprintf buffer length
Changes from v1:
* Addressed Chao Yu's comment on ENOENT and return value checks
* Rewrite of sysfs file handling (simplified)
* Rebased on dev-test tree
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Host-aware zoned block devices can accept random writes anywhere and so
do not require to be handled under F2FS_ZONED_HM mode. Allow host aware
disks to be treated as regular devices if c.zoned_mode is false, that
is, if the -m option is not specified in mkfs.f2fs.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
[Jaegeuk Kim: fixed wrong spaces]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
There is no need to require conventional zones for a zoned block device
that is not the first device of a multi-device volume. As a result,
there is no need to check the number of conventional zones of the
device if the device index is not 0.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Return an error if an unknown zoned model is reported for a device or
if parsing of the device zoned model fails. Also add comments to
briefly explain the zone models and what to do in the absence of a
kernel reported zoned model for a device.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
[Jaegeuk Kim: Fix one missing function def change]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Previously, f2fs_checkpoint.checksum_offset points fixed position of
f2fs_checkpoint structure:
"#define CP_CHKSUM_OFFSET 4092"
It is unnecessary, and it breaks the consecutiveness of nat and sit
bitmap stored across checkpoint park block and payload blocks.
This patch allows f2fs-tools to handle unfixed .checksum_offset.
In addition, for the case checksum value is stored in the middle of
checkpoint park, calculating checksum value with superposition method
like we did for inode_checksum.
In addition, add below change:
- using MAX_BITMAP_SIZE_IN_CKPT to clean up codes.
- introduce verify_checksum_chksum() to verify chksum_{offset,value}
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As zhaowuyun reported:
we met one problem of f2fs, and found one issue of make_f2fs, so I write
this email to search for your help to confirm this issue.
The issue was found on one of Android projects. We use f2fs as the filesystem
of userdata, and make sparse userdata.img using following command, which
invoked in script mkf2fsuserimg.sh
make_f2fs -S $SIZE -f -O encrypt -O quota -O verity $MKFS_OPTS $OUTPUT_FILE
use fastboot to flash this userdata.img to device, and it encountered f2fs
problem and leading to the mount fail of data partition.
we can make this issue 100% persent reproduced by making the data partition
dirty before flashing userdata.img.
suspect that issue is caused by the dirty data in the data partition.
so we checked that source code of make_f2fs in f2fs-tool, found that when
making f2fs, it use dev_fill to do some process:
...
we change code to the following, and the issue is gone.
if (c.sparse_mode)
return dev_write(buf, offset, len);
Chao Yu:
>
> After checking the codes, IIUC, I guess the problem here is, unlike
> img2simg, mkfs.f2fs won't record zeroed block in sparse image, so
> during transforming to normal image, some critical region like
> NAT/SIT/CP.payload area weren't be zeroed correctly, later kernel may
> load obsoleting data from those region.
>
> Also, The way you provide will obviously increase the size of sparse
> file, since with it we need to write all zeroed blocks of
> NAT/SIT/CP.payload to sparse file, it's not needed.
>
> Not sure, maybe we should use sparse_file_add_fill() to record zeroed
> blocks, so that this will make formatted image more like img2simged one.
Jaegeuk:
> We have to call sparse_file_add_fill() for dev_fill().
This patch fixes to support writing fill chunk sparse file for those
zeroed blocks in mkfs.f2fs.
Reported-and-tested-by: zhaowuyun <zhaowuyun@wingtech.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As Jaegeuk reminded:
Once user updates f2fs-tools which support new fields in inode layout,
but do keep the kernel which can not support those fields, it will cause
old f2fs fail to mount new image due to root_inode's i_extra_isize value
sanity check.
So if f2fs-tools doesn't enable feature which will use new fields of
inode, we don't need to expand i_extra_isize to include them, let's just
let i_extra_isize point to the end of last valid extra field's position.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch fixes the "open failure" issue on ro disk, reported by Hagbard.
"
If I boot with kernel option "ro rootfstype=f2fs
I get the following halfway trough boot:
* Checking local filesystems ...
Info: Use default preen mode
Info: Mounted device!
Info: Check FS only due to RO
Error: Failed to open the device!
* Filesystems couldn't be fixed
"
Reported-by: Hagbard Celine <hagbardcelin@gmail.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As Hagbard Celine reported:
"
Referring to the output from the fsck running against a "ro"
filesystem, especially this line:
Info: Check FS only due to RO
As far as i can tell this says that opposed to other filesystems
running fsck against a "ro" mounted f2fs partition will never fix any
errors.
So I tried running fsck against the same partition mounted "rw":
- mount -o remount,rw /mnt/f2fstest/
- fsck.f2fs -f /dev/nvme0n1p7
Info: Force to fix corruption
Info: Mounted device!
Error: Not available on mounted device!
I might be misunderstanding something, but all this tells me that
unless one make a custom initramfs that runs fsck before root is
mounted (something no distributions has, as far as I know), fsck will
never fix an f2fs formatted root partition during boot.
If this is by design and not a bug/unintended behavior, it should be
documented somewhere least more people will experience system crashes
like mine.
All tests above done with kernel 5.0.5 and f2fs-tools 1.12.0 with
"fsck.f2fs: allow to fsck readonly image w/ -f option"-patch by Chao
Yu.
"
We try to make our fsck behavior keeping line with e2fsprogs, but w/
-f option, we can just check a RO mounted device rather repair it, so
let's fix this.
Reported-and-Tested-by: Hagbard Celine <hagbardcelin@gmail.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Added a missing free statement related to each device’s path.
Signed-off-by: Sotirios-Efstathios Maneas <smaneas@cs.toronto.edu>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Avoid various compilation warnings due to strncpy:
libf2fs.c:590:33: warning: ‘snprintf’ output may be truncated before
the last format character [-Wformat-truncation=]
snprintf(rootdev, ret, "/dev/%s", buf);
../include/f2fs_fs.h:1384:2: warning: ‘strncpy’ specified bound
depends on the length of the source argument [-Wstringop-overflow=]
strncpy(buf, features, strlen(features) + 1);
f2fstat.c:243:3: warning: ‘strncpy’ output truncated before
terminating nul copying as many bytes from a string as its length
[-Wstringop-truncation]
strncpy(ptr_buf, name[i], strlen(name[i]));
This patch does not change any functionality.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
To keep line with e2fsprogs, let's allow to fsck mounted image as
readonly w/ -f option.
Reported-by: Perfect Gentleman <perfect007gentleman@gmail.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We must create a sparse file first before calling stat().
Fixes: eb9d8037ed ("f2fs-tools: avoid mounting f2fs if tools already open the device")
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Malloc Failure occurs in 32bit Windows, when using fastboot.exe flash the
f2fs sparse image filling with up to 2G chunk size.
Signed-off-by: Gao Ming <gaoming20@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
In 32-bits platform, {f,}stat on a large size file during mkfs, it will
cause EOVERFLOW error, this patch fixes to add macro definition
_FILE_OFFSET_BITS to avoid that error.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This allows the development link libraries to be installed in
/usr/lib, while the run-libraries are installed in /lib, which is
required by Debian policy. This can be done via:
configure --prefix=/ --libdir=/usr/lib --with-root-libdir=/lib
The technique of working around libtool's inflexibility is borrowed
from util-linux.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds an option to mkfs.f2fs in order for user to assign uid/gid
to the target partition.
This requires when vold in android formats a sdcard partition.
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
If the block device is opened by tools, F2FS should not be mounted.
Especially when fsck is running, errors unexpected may happen. So if
tools open a block device, we give it the O_EXCL flag to make sure
the block device is opened exclusivly.
Signed-off-by: Sheng Yong <shengyong1@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Ref: drivers/scsi/scsi_scan.c scsi_add_lun()
Ie, fixed-width 16 bytes, assumed to be filled with spaces -- NOT
null-terminated; comments suggest that in some cases this field can be
truncated and filled with nulls but printf is fine with that.
The old code did read up to 64 characters, which produced garbage for
at least some USB-attached card readers. It also special-cased '`'
character which the kernel does not.
Signed-off-by: Adam Borowski <kilobyte@angband.pl>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
The libsparse in AOSP is updating the type for 'len', from 'int' to
'size_t', in the callback parameter of sparse_file_foreach_chunk(). The
value represents a chunk size, which could be legitimately larger than
INT_MAX. This patch tracks the libsparse API change.
The change is guarded with SPARSE_CALLBACK_USES_SIZE_T that's exposed as
part of the libsparse change. This allows f2fs-tools to keep working
against older libsparse versions.
Signed-off-by: Tao Bao <tbao@google.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Problem:
Function f2fs_dev_is_unmounted() and get_device_info() define local
variable "struct stat xxx". If the callstack is very deep and stack
is smaller, it will result in stack corruption.
Solution:
It is better to use pointer and memory allocation instead of defining
"structure stat" in function stack.
Signed-off-by: Iris Chang <iris.chang@mediatek.com>
[Jaegeuk Kim: fix build errors.]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
