the convert process is currently completely implemented with sync operations.
That means it reads one buffer and then writes it. No parallelism and each sync
request takes as long as it takes until it is completed.
This can be a big performance hit when the convert process reads and writes
to devices which do not benefit from kernel readahead or pagecache.
In our environment we heavily have the following two use cases when using
qemu-img convert.
a) reading from NFS and writing to iSCSI for deploying templates
b) reading from iSCSI and writing to NFS for backups
In both processes we use libiscsi and libnfs so we have no kernel cache.
This patch changes the convert process to work with parallel running coroutines
which can significantly improve performance for network storage devices:
qemu-img (master)
nfs -> iscsi 22.8 secs
nfs -> ram 11.7 secs
ram -> iscsi 12.3 secs
qemu-img-async (8 coroutines, in-order write disabled)
nfs -> iscsi 11.0 secs
nfs -> ram 10.4 secs
ram -> iscsi 9.0 secs
This patches introduces 2 new cmdline parameters. The -m parameter to specify
the number of coroutines running in parallel (defaults to 8). And the -W parameter to
allow qemu-img to write to the target out of order rather than sequential. This improves
performance as the writes do not have to wait for each other to complete.
Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This adds the skip option which allows qemu-img dd to skip a number of blocks
before copying the input.
A test case was added to test the skip option.
Signed-off-by: Reda Sallahi <fullmanet@gmail.com>
Message-id: 20160810141609.32727-1-fullmanet@gmail.com
Signed-off-by: Max Reitz <mreitz@redhat.com>
This patch adds a basic dd subcommand analogous to dd(1) to qemu-img.
For the start, this implements the bs, if, of and count options and requires
both if and of to be specified (no stdin/stdout if not specified) and doesn't
support tty, pipes, etc.
The image format must be specified with -O for the output if the raw format
is not the intended one.
Two tests are added to test qemu-img dd.
Signed-off-by: Reda Sallahi <fullmanet@gmail.com>
Message-id: 20160810024312.14544-1-fullmanet@gmail.com
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[mreitz: Moved test 158 to 170]
Signed-off-by: Max Reitz <mreitz@redhat.com>
The command will work this way:
qemu-img --trace "qcow2*" create -f qcow2 1.img 64G
[Quote "qcow2*" to protect against shell globbing as suggested by Eric
Blake <eblake@redhat.com>.
--Stefan]
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1466174654-30130-8-git-send-email-den@openvz.org
Suggested by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This is necessary to enable creation of common qemu-img options which will
be specified before command.
The patch also enables '-V' alias to '--version' (exactly like in other
block utilities) and documents this change.
Signed-off-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 1466174654-30130-7-git-send-email-den@openvz.org
CC: Paolo Bonzini <pbonzini@redhat.com>
CC: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This options allows to flush the image periodically during write tests.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
With this new option, qemu-img bench can be told to advance the current
offset after each request by a different value than the buffer size.
This is useful for controlling the conditions for cluster allocation in
image formats (e.g. qcow2 cluster allocation with COW in front of the
request, or COW areas that aren't overwritten immediately).
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
This patch adds an option the specify the offset of the first request
made by qemu-img bench. This allows to benchmark misaligned requests.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
This extends qemu-img bench with an option that makes it use sequential
writes instead of reads for the test run.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
This adds a qemu-img command that allows doing some simple benchmarks
for the block layer without involving guest devices and a real VM.
For the start, this implements only a test of sequential reads.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Currently qemu-img allows an image filename to be passed on the
command line, but unless using the JSON format, it does not have
a way to set any options except the format eg
qemu-img info https://127.0.0.1/images/centos7.iso
This adds a --image-opts arg that indicates that the positional
filename should be interpreted as a full option string, not
just a filename.
qemu-img info --image-opts driver=https,url=https://127.0.0.1/images,sslverify=off
This flag is mutually exclusive with the '-f' / '-F' flags.
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Allow creation of user creatable object types with qemu-img
via a new --object command line arg. This will be used to supply
passwords and/or encryption keys to the various block driver
backends via the recently added 'secret' object type.
# printf letmein > mypasswd.txt
# qemu-img info --object secret,id=sec0,file=mypasswd.txt \
...other info args...
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Many source files have doubled words (eg "the the", "to to",
and so on). Most of these can simply be removed, but a couple
were actual mis-spellings (eg "to to" instead of "to do").
There was even one triple word score "to to to" :-)
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
Now that bdrv_amend_options() supports a status callback, use it to
display a progress report.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Benoît Canet <benoit.canet@nodalink.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 1414404776-4919-3-git-send-email-mreitz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Introduce a new parameter for qemu-img commit which may be used to
explicitly specify the backing file into which an image should be
committed if the backing chain has more than a single layer.
[Applied Eric Blake's qemu-img.texi documentation rewording
--Stefan]
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-id: 1414159063-25977-12-git-send-email-mreitz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Implement progress output for the commit command by querying the
progress of the block job.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 1414159063-25977-11-git-send-email-mreitz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
After the top image has been committed, it should be emptied unless
specified otherwise.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-id: 1414159063-25977-10-git-send-email-mreitz@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This patch removes support for the cow file format.
Normally we do not break backwards compatibility but in this case there
is no impact and it is the most logical option. Extraordinary claims
require extraordinary evidence so I will show why removing the cow block
driver is the right thing to do.
The cow file format is the disk image format for Usermode Linux, a way
of running a Linux system in userspace. The performance of UML was
never great and it was hacky, but it enjoyed some popularity before
hardware virtualization support became mainstream.
QEMU's block/cow.c is supposed to read this image file format.
Unfortunately the file format was underspecified:
1. Earlier Linux versions used the MAXPATHLEN constant for the backing
filename field. The value of MAXPATHLEN can change, so Linux
switched to a 4096 literal but QEMU has a 1024 literal.
2. Padding was not used on the header struct (both in the Linux kernel
and in QEMU) so the struct layout varied across architectures. In
particular, i386 and x86_64 were different due to int64_t alignment
differences. Linux now uses __attribute__((packed)), QEMU does not.
Therefore:
1. QEMU cow images do not conform to the Linux cow image file format.
2. cow images cannot be shared between different host architectures.
This means QEMU cow images are useless and QEMU has not had bug reports
from users actually hitting these issues.
Let's get rid of this thing, it serves no purpose and no one will be
affected.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 1410877464-20481-1-git-send-email-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
preallocation=falloc allocates disk space by posix_fallocate(),
preallocation=full allocates disk space by writing zeros to disk.
Both modes imply preallocation=metadata.
Signed-off-by: Hu Tao <hutao@cn.fujitsu.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This patch adds a new option preallocation for raw format, and implements
falloc and full preallocation.
Signed-off-by: Hu Tao <hutao@cn.fujitsu.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The src_cache option (-T) specifies the cache mode for backing files.
It applies both the image's old backing file as well as the new backing
file:
ret = bdrv_open(&bs_old_backing, backing_name, NULL, NULL, src_flags,
old_backing_drv, &local_err);
if (ret) {
...
}
if (out_baseimg[0]) {
bs_new_backing = bdrv_new("new_backing", &error_abort);
ret = bdrv_open(&bs_new_backing, out_baseimg, NULL, NULL, src_flags,
new_backing_drv, &local_err);
if (ret) {
...
}
}
The documentation only mentions the new backing file but it really
applies to both.
Suggested-by: Jeff Nelson <jenelson@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
The source cache option takes the same values as the cache option. The
documentation reads a little strange because it starts with "In contrast
the src_cache option ...". The fact that this is comparing with the
previous documented option (the 'cache' option) is implicit. Readers
may be confused, especially if they jump to src_cache without reading
cache documentation first.
Suggested-by: Jeff Nelson <jenelson@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
qemu-img amend may extensively modify the target image, depending on the
options to be amended (e.g. conversion to qcow2 compat level 0.10 from
1.1 for an image with many unallocated zero clusters). Therefore it
makes sense to allow the user to specify the cache mode to be used.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Many qemu-img subcommands only read the source file(s) once. For these
use cases, a full write-back cache is unnecessary and mainly clutters
host cache memory. Though this is generally no concern as cache memory
is freely available and can be scaled by the host OS, it may become a
concern with thin provisioning.
For these cases, it makes sense to allow users to freely specify the
source cache mode (e.g. use no cache at all).
This commit adds a new switch (-T) for the qemu-img subcommands check,
compare, convert and rebase to specify the cache to be used for source
images (the backing file in case of rebase).
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Add 'nocow' option so that users could have a chance to set NOCOW flag to
newly created files. It's useful on btrfs file system to enhance performance.
Btrfs has low performance when hosting VM images, even more when the guest
in those VM are also using btrfs as file system. One way to mitigate this bad
performance is to turn off COW attributes on VM files. Generally, there are
two ways to turn off NOCOW on btrfs: a) by mounting fs with nodatacow, then
all newly created files will be NOCOW. b) per file. Add the NOCOW file
attribute. It could only be done to empty or new files.
This patch tries the second way, according to the option, it could add NOCOW
per file.
For most block drivers, since the create file step is in raw-posix.c, so we
can do setting NOCOW flag ioctl in raw-posix.c only.
But there are some exceptions, like block/vpc.c and block/vdi.c, they are
creating file by calling qemu_open directly. For them, do the same setting
NOCOW flag ioctl work in them separately.
[Fixed up 082.out due to the new 'nocow' creation option
--Stefan]
Signed-off-by: Chunyan Liu <cyliu@suse.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The exit code 63 (check not supported by image format) was not even
documented in the comment above the check command in the source code;
add it, as it does indeed seem useful.
Also, document all of check's exit codes in the manpage.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reported-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The qemu-img.texi / qemu-doc.texi files currently describe the
qcow2/qcow2 encryption thus
"Encryption uses the AES format which is very secure (128 bit
keys). Use a long password (16 characters) to get maximum
protection."
While AES is indeed a strong encryption system, the way that
QCow/QCow2 use it results in a poor/weak encryption system.
Due to the use of predictable IVs, based on the sector number
extended to 128 bits, it is vulnerable to chosen plaintext
attacks which can reveal the existence of encrypted data.
The direct use of the user passphrase as the encryption key
also leads to an inability to change the passphrase of an
image. If passphrase is ever compromised the image data will
all be vulnerable, since it cannot be re-encrypted. The admin
has to clone the image files with a new passphrase and then
use a program like shred to secure erase all the old files.
Recommend against any use of QCow/QCow2 encryption, directing
users to dm-crypt / LUKS which can meet modern cryptography
best practices.
[Changed "Qcow" to "qcow" for consistency.
--Stefan]
Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This updates the documentation for commiting snapshot images.
Specifically, this highlights what happens when the base image
is either smaller or larger than the snapshot image being committed.
In the case of the base image being smaller, it is resized to the
larger size of the snapshot image. In the case of the base image
being larger, it is not resized automatically, but once the commit
has completed it is safe for the user to truncate the base image.
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Document the SIGUSR1 behaviour of qemu-img. Also, added compare to the
list of subcommands that support -p.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
Commit 9117b47717 ("qcow2: Change default
for new images to compat=1.1") changed the default qcow2 image format
version but forgot to update qemu-doc.texi and qemu-img.texi.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The man page for qemu-img, and the qemu-doc, did not mention VHDX
as a supported format. This adds in reference to VHDX in those
documents.
[Stefan Weil <sw@weilnetz.de> suggested s/Block Size/Block size/ for
consistency. I have made this change.
--Stefan]
Signed-off-by: Jeff Cody <jcody@redhat.com>
Reviewed-by: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Now qemu-img convert have similar options as qemu-nbd for internal
snapshot.
Signed-off-by: Wenchao Xia <xiawenc@linux.vnet.ibm.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This patch adds the "amend" option to qemu-img which allows changing
image options on existing image files. It also adds the generic bdrv
implementation which is basically just a wrapper for the image format
specific function.
Signed-off-by: Max Reitz <mreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake also requested including the output in qapi-schema.json,
so that it is published through the introspection mechanism.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Add a -n option to skip volume creation on qemu-img convert.
This is useful for targets such as rbd / ceph, where the
target volume may already exist; we cannot always rely on
qemu-img convert to create the image, as dependent on the
output format, there may be parameters which are not possible
to specify through the qemu-img convert command line.
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
Signed-off-by: Alex Bligh <alex@alex.org.uk>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This patch adds new qemu-img subcommand that compares content of two disk
images.
Signed-off-by: Miroslav Rezanina <mrezanin@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
There can be a need to turn output to stdout off. This patch adds a -q option
that enable "Quiet mode". In Quiet mode, only errors are printed out.
Signed-off-by: Miroslav Rezanina <mrezanin@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This option --output=[human|json] makes qemu-img check output a human
or JSON representation at the choice of the user.
Signed-off-by: Federico Simoncelli <fsimonce@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Document new and yet undocumented options and image formats. The
qemu-img man page contains information only for raw and qcow2 now and
references the HTML documentation for a more detailed description of
other formats.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kashyap Chamarthy <kashyap.cv@gmail.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
This patch allows an empty filename to be passed as the new base image name
for qemu-img rebase to mean base the image on no backing file (i.e.
independent of any backing file). According to Eric Blake, qemu-img rebase
already supports this when '-u' is used; this adds support when -u is not
used.
Signed-off-by: Alex Bligh <alex@alex.org.uk>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* kwolf/for-anthony:
block: Don't forget to delete temporary file
Don't require encryption password for 'qemu-img info' command
qemu-img: Add json output option to the info command.
qapi: Add SnapshotInfo and ImageInfo.
ahci: properly reset PxCMD on HBA reset
block: fix block tray status
vdi: Fix warning from clang
block/curl: Fix wrong free statement
ide: Fix error messages from static code analysis (no real error)
ATAPI: STARTSTOPUNIT only eject/load media if powercondition is 0
sheepdog: fix savevm and loadvm
These wrong spellings were detected by codespell:
* successully -> successfully
* alot -> a lot
* wanna -> want to
* infomation -> information
* occured -> occurred
["also is" -> "is also" and "ressources" -> "resources" suggested by
Peter Maydell <peter.maydell@linaro.org>]
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
People have repeatedly expected that you can do things like snapshotting
an image with qemu-img while a qemu instance is running. Maybe we need
to consider locking the files while they are in use, but having a
warning in the qemu-img manpage is doable for 1.2 and can't hurt anyway.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The qemu-img.1 man page is missing the qed format from its list of
supported formats. Document the image creation options for qed.
Suggested-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
The QED block driver already provides the functionality to not only
detect inconsistencies in images, but also fix them. However, this
functionality cannot be manually invoked with qemu-img, but the
check happens only automatically during bdrv_open().
This adds a -r switch to qemu-img check that allows manual invocation
of an image repair.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
