if we new node block in fsck flow, we need to update
the valid_node_cnt at the same time.
Signed-off-by: Lihong Kou <koulihong@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
inode.i_blocks includes inode, xnode and data block count, so, only
fix in below condition:
- i_blocks := 3 (inode + xnode + data_block)
- i_blocks := 2 (inode + data_block)
In addition, it recovers symlink's i_size to 4k rather than i_blocks *
4k.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We should always set c.bug_on whenever found a bug, then fix them
if c.fix_on is on, otherwise, some bugs won't be shown unless we
enable debug log.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Propagate correct error number from write_dquots() to
quota_write_inode().
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
For defrag, resize, sload tools, let's advise to mount unclean
image to replay journal first in order to not lose any fsynced
data.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
During block allocation in large free section, we need to change
all sub segments' type in it, otherwise, we will fail to allocate
block in non-first segment due to mismatch seg-type.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Just cleanup, no logic change, besides, it can be reused by latter
patch.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Introduce f2fs_ra_meta_pages() to readahead meta pages like we did
in kernel.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Like we did in kernel, allocating memory in batch will be more
efficient.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We missed to free sit_i->sentries in f2fs_do_umount(), fix it.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
In error path of build_sit_info(), start variable is unsigned int type,
it should never be less than zero, fix it.
build_sit_info()
{
...
unsigned int start;
...
free_validity_maps:
for (--start ; start >= 0; --start)
free(sit_i->sentries[start].cur_valid_map);
...
}
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Support 'defrag_file' sub command to trigger file based defragment via
ioctl in f2fs.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This adds support for f2fs casefolding. Similarly to ext4 casefolding,
this is controlled per-folder via the +F attribute. It can be toggled on
empty directories only. It is not currently compatible with encryption,
but that will likely change.
When enabling the casefold feature, use the -C flag. The format is:
-C encoding[:flag1,flag2,etc]
Signed-off-by: Daniel Rosenberg <drosen@google.com>
[Jaegeuk Kim: print "casefold" in sb->feature]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
It tries to let fsck be noticed wrong inline size, and do the fix.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
We forgot to release memory allocated in dentry.link, fix to release
it after last use of it.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch fix bug that variable dentries going out of scope leaks
the storage it points to.
Signed-off-by: Xiaojun Wang <wangxiaojun11@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
When the -m option is specified to format a Zoned device,
do not fall back to the non-zoned mode in case information
about the device is not found.
Explicitly specify this error to the user.
Signed-off-by: Surbhi Palande <csurbhi@gmail.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Reviewed-by: Damien Le Moal <damien.lemoal@wdc.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
f2fs_fs.h locats in local directory rather than system one, fix it.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch enable to set file's temperature while loading files
to image.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
In generic/38[3456], we use quotactl(2) to check if prjquota is
enabled on the given device (src/feature -P $dev in _require_prjquota),
and quotactl(2) requires the given device is a mounted device.
So it requires dump.f2fs to list/check enabled features on a mounted
device, let's relieve to allow such operation.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This original patch was from Weichao Guo.
We may encounter both checkpoints invalid in such a case:
1. kernel writes CP A;
2. power-cut when kernel writes CP B, then CP B is corrupted;
3. fsck: load CP A, fix meta/data;
4. power-cut when fsck writes CP A in-place, then CP A is corrupted too;
To avoid both checkpoints being invalid, this patch changes to duplicate
valid checkpoint to mirror position first, and then, write fixed checkpoint
to CP #0 position.
This can make sure that, while fsck repairing, even there is sudden
power-cut, last valid checkpoint can be kept in CP #1 position.
Signed-off-by: Weichao Guo <guoweichao@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Add more sanity check logic for superblock like we did in kernel side.
This fixes bug reported by Seulbae Kim from bugzilla.
https://bugzilla.kernel.org/show_bug.cgi?id=203861
Reproted-by: Seulbae Kim <seulbae@gatech.edu>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Enables to detect corrupted sit journal
a) total count of entries.
b) segment no. in entry.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch keeps line with f2fs kernel module for checkpoint update:
1. writeback all blocks inside checkpoint except last cp_park block;
2. fsync;
3. write last cp_park block;
4. fsync;
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Calculate inode checksum and writeback when processing
sload or resize.
Signed-off-by: Xue Liu <liuxueliu.liu@huawei.com>
[Chao Yu: change commit message and code]
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Build segment manager and node manager always to catch and auto-fix
invalid NAT entries found in the NAT journal.
Signed-off-by: Sahitya Tummala <stummala@codeaurora.org>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
A partition device does not have the "zoned" nor "chunk_sectors" sysfs
attribute files. Only the owner block device of the partition has these
files. This causes the detection of the zoned model and zone size of a
partition device to fail when executing mkfs.f2fs.
Fix this problem by using the owner device sysfs directory as the base
directory for accessing the zoned and chunk_sectors files. This is done
by using the device major:minor symbolic link under the /sys/dev/block
directory, reading this link and removing the partition device name from
the link path for a partition device (which is indicated by the presence
of the "partition" file under the directory).
Also add a check for the ENOENT error when opening the device "zoned"
sysfs attribute file. The absence of this file indicates that the
kernel does not support zoned block devices. Since the device file is
already open, it exists, and so the device can safely be assumed as not
being zoned.
Changes from v2:
* Addressed Chao Yu's comment on snprintf buffer length
Changes from v1:
* Addressed Chao Yu's comment on ENOENT and return value checks
* Rewrite of sysfs file handling (simplified)
* Rebased on dev-test tree
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Now, kernel can catch incorrect total valid block count which is exceed
max user block count of image.
Then, generic/051,476 of fstest reports below message:
Apr 15 11:08:03 szvp000201624 kernel: [ 2533.515813] F2FS-fs (zram1): Wrong valid_user_blocks: 469505, user_block_count: 469504
Apr 15 11:08:03 szvp000201624 kernel: [ 2533.519166] F2FS-fs (zram1): Failed to get valid F2FS checkpoint
The reason is that when fsck repairs corrupted quota sysfile, it didn't
check max user block count when allocating new block for quota sysfile,
so ckpt.valid_block_count can exceed max user block count, result in
mount failure later.
Adding upper boundary check of block count in reserve_new_block() to
fix this issue.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Following extensions are removed:
- divx: deprecated video format and it's usually wrapped with avi
- asf: deprecated streaming format
- asx: redirecting file to asf(small)
- wmx: redirecting file to wma/wmv(small)
- rm: deprecated media container
- video: unused
- wv: unpopular audio format from 1998
The extensions list is limited to 64 and those don't deserve to be
on this space-precious list.
Common prefixes are introduced and are checked with
https://en.wikipedia.org/wiki/List_of_filename_extensions
to avoid treating possible hot files as cold:
- mp: covers mp3, mp4, mpeg, mpg
- wm: covers wma, wmb, wmv
- og: covers oga, ogg, ogm, ogv
- jp: covers jpg, jpeg, jp2
Following extensions are added:
- webm: extremely popular free media container format from Google
VP8/VP9/AV1 and Vorbis/Opus is often wrapped with this container
- wav: uncompressed audio format, commonly used with voice recorders
- svg: vector image format commonly used in web
- webp: free lossy image format commonly used in web
- jar: Java archive file
- deb: Debian software package
- iso: disk image file
- gz: gzip compressed file, unable to randomly update
- xz: xz compressed file, unable to randomly update
- zst: zstd compressed file, unable to randomly update
- pdf: PDF document
- pyc: Python bytecode automatically generated when
executing python to run .py files
- ttc, ttf: font files
- cnt: image alias files commonly used in Android apps
- exo: EXO player's cache files, commonly used in Android's YouTube app
- odex, vdex: Android RunTime files found in /data/app/*/oat
Total entries on the list changed from 34 to 36.
Signed-off-by: Park Ju Hyung <qkrwngud825@gmail.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Host-aware zoned block devices can accept random writes anywhere and so
do not require to be handled under F2FS_ZONED_HM mode. Allow host aware
disks to be treated as regular devices if c.zoned_mode is false, that
is, if the -m option is not specified in mkfs.f2fs.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
[Jaegeuk Kim: fixed wrong spaces]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
There is no need to require conventional zones for a zoned block device
that is not the first device of a multi-device volume. As a result,
there is no need to check the number of conventional zones of the
device if the device index is not 0.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Return an error if an unknown zoned model is reported for a device or
if parsing of the device zoned model fails. Also add comments to
briefly explain the zone models and what to do in the absence of a
kernel reported zoned model for a device.
Signed-off-by: Damien Le Moal <damien.lemoal@wdc.com>
[Jaegeuk Kim: Fix one missing function def change]
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
If we use later kernel having larger extra_isize, old fsck will delete
entire old files.
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
If .next_blkoff is inconsistent, we can update curseg .next_blkoff
to first unused block address, and change .alloc_type to SSR
preferential, instead of move curseg to other position.
This can help to repair fuzzed image which has no more free segment.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Let check_curseg_offset() just check one curseg, and introduce
check_curseg_offsets() to check all cursegs, no logic changes.
In addition, update message printed a bit.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
For large_nat_bitmap feature, there is a design flaw:
Previous:
struct f2fs_checkpoint layout:
+--------------------------+ 0x0000
| checkpoint_ver |
| ...... |
| checksum_offset |------+
| ...... | |
| sit_nat_version_bitmap[] |<-----|-------+
| ...... | | |
| checksum_value |<-----+ |
+--------------------------+ 0x1000 |
| | nat_bitmap + sit_bitmap
| payload blocks | |
| | |
+--------------------------|<-------------+
Obviously, if nat_bitmap size + sit_bitmap size is larger than
MAX_BITMAP_SIZE_IN_CKPT, nat_bitmap or sit_bitmap may overlap
checkpoint checksum's position, once checkpoint() is triggered
from kernel, nat or sit bitmap will be damaged by checksum field.
In order to fix this, let's relocate checksum_value's position
to the head of sit_nat_version_bitmap as below, then nat/sit
bitmap and chksum value update will become safe.
After:
struct f2fs_checkpoint layout:
+--------------------------+ 0x0000
| checkpoint_ver |
| ...... |
| checksum_offset |------+
| ...... | |
| sit_nat_version_bitmap[] |<-----+
| ...... |<-------------+
| | |
+--------------------------+ 0x1000 |
| | nat_bitmap + sit_bitmap
| payload blocks | |
| | |
+--------------------------|<-------------+
Related report and discussion:
https://sourceforge.net/p/linux-f2fs/mailman/message/36642346/
In addition, during writing checkpoint, if large_nat_bitmap feature is
enabled, we need to set CP_LARGE_NAT_BITMAP_FLAG flag in checkpoint.
Reported-by: Park Ju Hyung <qkrwngud825@gmail.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Previously, f2fs_checkpoint.checksum_offset points fixed position of
f2fs_checkpoint structure:
"#define CP_CHKSUM_OFFSET 4092"
It is unnecessary, and it breaks the consecutiveness of nat and sit
bitmap stored across checkpoint park block and payload blocks.
This patch allows f2fs-tools to handle unfixed .checksum_offset.
In addition, for the case checksum value is stored in the middle of
checkpoint park, calculating checksum value with superposition method
like we did for inode_checksum.
In addition, add below change:
- using MAX_BITMAP_SIZE_IN_CKPT to clean up codes.
- introduce verify_checksum_chksum() to verify chksum_{offset,value}
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Let's fix the wrong start offset of cp_payload blocks in
fix_checkpoint().
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch fixes to initialize NAT/SIT/CP.payload region in sparse
file mode for sload.
Reported-and-tested-by: zhaowuyun <zhaowuyun@wingtech.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As zhaowuyun reported:
we met one problem of f2fs, and found one issue of make_f2fs, so I write
this email to search for your help to confirm this issue.
The issue was found on one of Android projects. We use f2fs as the filesystem
of userdata, and make sparse userdata.img using following command, which
invoked in script mkf2fsuserimg.sh
make_f2fs -S $SIZE -f -O encrypt -O quota -O verity $MKFS_OPTS $OUTPUT_FILE
use fastboot to flash this userdata.img to device, and it encountered f2fs
problem and leading to the mount fail of data partition.
we can make this issue 100% persent reproduced by making the data partition
dirty before flashing userdata.img.
suspect that issue is caused by the dirty data in the data partition.
so we checked that source code of make_f2fs in f2fs-tool, found that when
making f2fs, it use dev_fill to do some process:
...
we change code to the following, and the issue is gone.
if (c.sparse_mode)
return dev_write(buf, offset, len);
Chao Yu:
>
> After checking the codes, IIUC, I guess the problem here is, unlike
> img2simg, mkfs.f2fs won't record zeroed block in sparse image, so
> during transforming to normal image, some critical region like
> NAT/SIT/CP.payload area weren't be zeroed correctly, later kernel may
> load obsoleting data from those region.
>
> Also, The way you provide will obviously increase the size of sparse
> file, since with it we need to write all zeroed blocks of
> NAT/SIT/CP.payload to sparse file, it's not needed.
>
> Not sure, maybe we should use sparse_file_add_fill() to record zeroed
> blocks, so that this will make formatted image more like img2simged one.
Jaegeuk:
> We have to call sparse_file_add_fill() for dev_fill().
This patch fixes to support writing fill chunk sparse file for those
zeroed blocks in mkfs.f2fs.
Reported-and-tested-by: zhaowuyun <zhaowuyun@wingtech.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
As Jaegeuk reminded:
Once user updates f2fs-tools which support new fields in inode layout,
but do keep the kernel which can not support those fields, it will cause
old f2fs fail to mount new image due to root_inode's i_extra_isize value
sanity check.
So if f2fs-tools doesn't enable feature which will use new fields of
inode, we don't need to expand i_extra_isize to include them, let's just
let i_extra_isize point to the end of last valid extra field's position.
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
