mirror of
https://github.com/reactos/CMake.git
synced 2024-12-11 05:14:00 +00:00
fb51d98562
This allows for a built in bzip and zip capability, so external tools will not be needed for these packagers. The cmake -E tar xf should be able to handle all compression types now as well.
2965 lines
109 KiB
XML
2965 lines
109 KiB
XML
<?xml version="1.0"?> <!-- -*- sgml -*- -->
|
|
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[
|
|
|
|
<!-- various strings, dates etc. common to all docs -->
|
|
<!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents;
|
|
]>
|
|
|
|
<book lang="en" id="userman" xreflabel="bzip2 Manual">
|
|
|
|
<bookinfo>
|
|
<title>bzip2 and libbzip2, version 1.0.5</title>
|
|
<subtitle>A program and library for data compression</subtitle>
|
|
<copyright>
|
|
<year>&bz-lifespan;</year>
|
|
<holder>Julian Seward</holder>
|
|
</copyright>
|
|
<releaseinfo>Version &bz-version; of &bz-date;</releaseinfo>
|
|
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Julian</firstname>
|
|
<surname>Seward</surname>
|
|
<affiliation>
|
|
<orgname>&bz-url;</orgname>
|
|
</affiliation>
|
|
</author>
|
|
</authorgroup>
|
|
|
|
<legalnotice>
|
|
|
|
<para>This program, <computeroutput>bzip2</computeroutput>, the
|
|
associated library <computeroutput>libbzip2</computeroutput>, and
|
|
all documentation, are copyright © &bz-lifespan; Julian Seward.
|
|
All rights reserved.</para>
|
|
|
|
<para>Redistribution and use in source and binary forms, with
|
|
or without modification, are permitted provided that the
|
|
following conditions are met:</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>Redistributions of source code must retain the
|
|
above copyright notice, this list of conditions and the
|
|
following disclaimer.</para></listitem>
|
|
|
|
<listitem><para>The origin of this software must not be
|
|
misrepresented; you must not claim that you wrote the original
|
|
software. If you use this software in a product, an
|
|
acknowledgment in the product documentation would be
|
|
appreciated but is not required.</para></listitem>
|
|
|
|
<listitem><para>Altered source versions must be plainly marked
|
|
as such, and must not be misrepresented as being the original
|
|
software.</para></listitem>
|
|
|
|
<listitem><para>The name of the author may not be used to
|
|
endorse or promote products derived from this software without
|
|
specific prior written permission.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY
|
|
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
|
|
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
|
|
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
|
|
AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
|
|
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
|
|
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
|
|
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
|
|
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
|
|
THE POSSIBILITY OF SUCH DAMAGE.</para>
|
|
|
|
<para>PATENTS: To the best of my knowledge,
|
|
<computeroutput>bzip2</computeroutput> and
|
|
<computeroutput>libbzip2</computeroutput> do not use any patented
|
|
algorithms. However, I do not have the resources to carry
|
|
out a patent search. Therefore I cannot give any guarantee of
|
|
the above statement.
|
|
</para>
|
|
|
|
</legalnotice>
|
|
|
|
</bookinfo>
|
|
|
|
|
|
|
|
<chapter id="intro" xreflabel="Introduction">
|
|
<title>Introduction</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> compresses files
|
|
using the Burrows-Wheeler block-sorting text compression
|
|
algorithm, and Huffman coding. Compression is generally
|
|
considerably better than that achieved by more conventional
|
|
LZ77/LZ78-based compressors, and approaches the performance of
|
|
the PPM family of statistical compressors.</para>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> is built on top of
|
|
<computeroutput>libbzip2</computeroutput>, a flexible library for
|
|
handling compressed data in the
|
|
<computeroutput>bzip2</computeroutput> format. This manual
|
|
describes both how to use the program and how to work with the
|
|
library interface. Most of the manual is devoted to this
|
|
library, not the program, which is good news if your interest is
|
|
only in the program.</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para><xref linkend="using"/> describes how to use
|
|
<computeroutput>bzip2</computeroutput>; this is the only part
|
|
you need to read if you just want to know how to operate the
|
|
program.</para></listitem>
|
|
|
|
<listitem><para><xref linkend="libprog"/> describes the
|
|
programming interfaces in detail, and</para></listitem>
|
|
|
|
<listitem><para><xref linkend="misc"/> records some
|
|
miscellaneous notes which I thought ought to be recorded
|
|
somewhere.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</chapter>
|
|
|
|
|
|
<chapter id="using" xreflabel="How to use bzip2">
|
|
<title>How to use bzip2</title>
|
|
|
|
<para>This chapter contains a copy of the
|
|
<computeroutput>bzip2</computeroutput> man page, and nothing
|
|
else.</para>
|
|
|
|
<sect1 id="name" xreflabel="NAME">
|
|
<title>NAME</title>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para><computeroutput>bzip2</computeroutput>,
|
|
<computeroutput>bunzip2</computeroutput> - a block-sorting file
|
|
compressor, v1.0.4</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bzcat</computeroutput> -
|
|
decompresses files to stdout</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bzip2recover</computeroutput> -
|
|
recovers data from damaged bzip2 files</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="synopsis" xreflabel="SYNOPSIS">
|
|
<title>SYNOPSIS</title>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para><computeroutput>bzip2</computeroutput> [
|
|
-cdfkqstvzVL123456789 ] [ filenames ... ]</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bunzip2</computeroutput> [
|
|
-fkvsVL ] [ filenames ... ]</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [
|
|
filenames ... ]</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bzip2recover</computeroutput>
|
|
filename</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="description" xreflabel="DESCRIPTION">
|
|
<title>DESCRIPTION</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> compresses files
|
|
using the Burrows-Wheeler block sorting text compression
|
|
algorithm, and Huffman coding. Compression is generally
|
|
considerably better than that achieved by more conventional
|
|
LZ77/LZ78-based compressors, and approaches the performance of
|
|
the PPM family of statistical compressors.</para>
|
|
|
|
<para>The command-line options are deliberately very similar to
|
|
those of GNU <computeroutput>gzip</computeroutput>, but they are
|
|
not identical.</para>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> expects a list of
|
|
file names to accompany the command-line flags. Each file is
|
|
replaced by a compressed version of itself, with the name
|
|
<computeroutput>original_name.bz2</computeroutput>. Each
|
|
compressed file has the same modification date, permissions, and,
|
|
when possible, ownership as the corresponding original, so that
|
|
these properties can be correctly restored at decompression time.
|
|
File name handling is naive in the sense that there is no
|
|
mechanism for preserving original file names, permissions,
|
|
ownerships or dates in filesystems which lack these concepts, or
|
|
have serious file name length restrictions, such as
|
|
MS-DOS.</para>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> and
|
|
<computeroutput>bunzip2</computeroutput> will by default not
|
|
overwrite existing files. If you want this to happen, specify
|
|
the <computeroutput>-f</computeroutput> flag.</para>
|
|
|
|
<para>If no file names are specified,
|
|
<computeroutput>bzip2</computeroutput> compresses from standard
|
|
input to standard output. In this case,
|
|
<computeroutput>bzip2</computeroutput> will decline to write
|
|
compressed output to a terminal, as this would be entirely
|
|
incomprehensible and therefore pointless.</para>
|
|
|
|
<para><computeroutput>bunzip2</computeroutput> (or
|
|
<computeroutput>bzip2 -d</computeroutput>) decompresses all
|
|
specified files. Files which were not created by
|
|
<computeroutput>bzip2</computeroutput> will be detected and
|
|
ignored, and a warning issued.
|
|
<computeroutput>bzip2</computeroutput> attempts to guess the
|
|
filename for the decompressed file from that of the compressed
|
|
file as follows:</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para><computeroutput>filename.bz2 </computeroutput>
|
|
becomes
|
|
<computeroutput>filename</computeroutput></para></listitem>
|
|
|
|
<listitem><para><computeroutput>filename.bz </computeroutput>
|
|
becomes
|
|
<computeroutput>filename</computeroutput></para></listitem>
|
|
|
|
<listitem><para><computeroutput>filename.tbz2</computeroutput>
|
|
becomes
|
|
<computeroutput>filename.tar</computeroutput></para></listitem>
|
|
|
|
<listitem><para><computeroutput>filename.tbz </computeroutput>
|
|
becomes
|
|
<computeroutput>filename.tar</computeroutput></para></listitem>
|
|
|
|
<listitem><para><computeroutput>anyothername </computeroutput>
|
|
becomes
|
|
<computeroutput>anyothername.out</computeroutput></para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>If the file does not end in one of the recognised endings,
|
|
<computeroutput>.bz2</computeroutput>,
|
|
<computeroutput>.bz</computeroutput>,
|
|
<computeroutput>.tbz2</computeroutput> or
|
|
<computeroutput>.tbz</computeroutput>,
|
|
<computeroutput>bzip2</computeroutput> complains that it cannot
|
|
guess the name of the original file, and uses the original name
|
|
with <computeroutput>.out</computeroutput> appended.</para>
|
|
|
|
<para>As with compression, supplying no filenames causes
|
|
decompression from standard input to standard output.</para>
|
|
|
|
<para><computeroutput>bunzip2</computeroutput> will correctly
|
|
decompress a file which is the concatenation of two or more
|
|
compressed files. The result is the concatenation of the
|
|
corresponding uncompressed files. Integrity testing
|
|
(<computeroutput>-t</computeroutput>) of concatenated compressed
|
|
files is also supported.</para>
|
|
|
|
<para>You can also compress or decompress files to the standard
|
|
output by giving the <computeroutput>-c</computeroutput> flag.
|
|
Multiple files may be compressed and decompressed like this. The
|
|
resulting outputs are fed sequentially to stdout. Compression of
|
|
multiple files in this manner generates a stream containing
|
|
multiple compressed file representations. Such a stream can be
|
|
decompressed correctly only by
|
|
<computeroutput>bzip2</computeroutput> version 0.9.0 or later.
|
|
Earlier versions of <computeroutput>bzip2</computeroutput> will
|
|
stop after decompressing the first file in the stream.</para>
|
|
|
|
<para><computeroutput>bzcat</computeroutput> (or
|
|
<computeroutput>bzip2 -dc</computeroutput>) decompresses all
|
|
specified files to the standard output.</para>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> will read arguments
|
|
from the environment variables
|
|
<computeroutput>BZIP2</computeroutput> and
|
|
<computeroutput>BZIP</computeroutput>, in that order, and will
|
|
process them before any arguments read from the command line.
|
|
This gives a convenient way to supply default arguments.</para>
|
|
|
|
<para>Compression is always performed, even if the compressed
|
|
file is slightly larger than the original. Files of less than
|
|
about one hundred bytes tend to get larger, since the compression
|
|
mechanism has a constant overhead in the region of 50 bytes.
|
|
Random data (including the output of most file compressors) is
|
|
coded at about 8.05 bits per byte, giving an expansion of around
|
|
0.5%.</para>
|
|
|
|
<para>As a self-check for your protection,
|
|
<computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make
|
|
sure that the decompressed version of a file is identical to the
|
|
original. This guards against corruption of the compressed data,
|
|
and against undetected bugs in
|
|
<computeroutput>bzip2</computeroutput> (hopefully very unlikely).
|
|
The chances of data corruption going undetected is microscopic,
|
|
about one chance in four billion for each file processed. Be
|
|
aware, though, that the check occurs upon decompression, so it
|
|
can only tell you that something is wrong. It can't help you
|
|
recover the original uncompressed data. You can use
|
|
<computeroutput>bzip2recover</computeroutput> to try to recover
|
|
data from damaged files.</para>
|
|
|
|
<para>Return values: 0 for a normal exit, 1 for environmental
|
|
problems (file not found, invalid flags, I/O errors, etc.), 2
|
|
to indicate a corrupt compressed file, 3 for an internal
|
|
consistency error (eg, bug) which caused
|
|
<computeroutput>bzip2</computeroutput> to panic.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="options" xreflabel="OPTIONS">
|
|
<title>OPTIONS</title>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-c --stdout</computeroutput></term>
|
|
<listitem><para>Compress or decompress to standard
|
|
output.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-d --decompress</computeroutput></term>
|
|
<listitem><para>Force decompression.
|
|
<computeroutput>bzip2</computeroutput>,
|
|
<computeroutput>bunzip2</computeroutput> and
|
|
<computeroutput>bzcat</computeroutput> are really the same
|
|
program, and the decision about what actions to take is done on
|
|
the basis of which name is used. This flag overrides that
|
|
mechanism, and forces bzip2 to decompress.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-z --compress</computeroutput></term>
|
|
<listitem><para>The complement to
|
|
<computeroutput>-d</computeroutput>: forces compression,
|
|
regardless of the invokation name.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-t --test</computeroutput></term>
|
|
<listitem><para>Check integrity of the specified file(s), but
|
|
don't decompress them. This really performs a trial
|
|
decompression and throws away the result.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-f --force</computeroutput></term>
|
|
<listitem><para>Force overwrite of output files. Normally,
|
|
<computeroutput>bzip2</computeroutput> will not overwrite
|
|
existing output files. Also forces
|
|
<computeroutput>bzip2</computeroutput> to break hard links to
|
|
files, which it otherwise wouldn't do.</para>
|
|
<para><computeroutput>bzip2</computeroutput> normally declines
|
|
to decompress files which don't have the correct magic header
|
|
bytes. If forced (<computeroutput>-f</computeroutput>),
|
|
however, it will pass such files through unmodified. This is
|
|
how GNU <computeroutput>gzip</computeroutput> behaves.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-k --keep</computeroutput></term>
|
|
<listitem><para>Keep (don't delete) input files during
|
|
compression or decompression.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-s --small</computeroutput></term>
|
|
<listitem><para>Reduce memory usage, for compression,
|
|
decompression and testing. Files are decompressed and tested
|
|
using a modified algorithm which only requires 2.5 bytes per
|
|
block byte. This means any file can be decompressed in 2300k
|
|
of memory, albeit at about half the normal speed.</para>
|
|
<para>During compression, <computeroutput>-s</computeroutput>
|
|
selects a block size of 200k, which limits memory use to around
|
|
the same figure, at the expense of your compression ratio. In
|
|
short, if your machine is low on memory (8 megabytes or less),
|
|
use <computeroutput>-s</computeroutput> for everything. See
|
|
<xref linkend="memory-management"/> below.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-q --quiet</computeroutput></term>
|
|
<listitem><para>Suppress non-essential warning messages.
|
|
Messages pertaining to I/O errors and other critical events
|
|
will not be suppressed.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-v --verbose</computeroutput></term>
|
|
<listitem><para>Verbose mode -- show the compression ratio for
|
|
each file processed. Further
|
|
<computeroutput>-v</computeroutput>'s increase the verbosity
|
|
level, spewing out lots of information which is primarily of
|
|
interest for diagnostic purposes.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-L --license -V --version</computeroutput></term>
|
|
<listitem><para>Display the software version, license terms and
|
|
conditions.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>-1</computeroutput> (or
|
|
<computeroutput>--fast</computeroutput>) to
|
|
<computeroutput>-9</computeroutput> (or
|
|
<computeroutput>-best</computeroutput>)</term>
|
|
<listitem><para>Set the block size to 100 k, 200 k ... 900 k
|
|
when compressing. Has no effect when decompressing. See <xref
|
|
linkend="memory-management" /> below. The
|
|
<computeroutput>--fast</computeroutput> and
|
|
<computeroutput>--best</computeroutput> aliases are primarily
|
|
for GNU <computeroutput>gzip</computeroutput> compatibility.
|
|
In particular, <computeroutput>--fast</computeroutput> doesn't
|
|
make things significantly faster. And
|
|
<computeroutput>--best</computeroutput> merely selects the
|
|
default behaviour.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>--</computeroutput></term>
|
|
<listitem><para>Treats all subsequent arguments as file names,
|
|
even if they start with a dash. This is so you can handle
|
|
files with names beginning with a dash, for example:
|
|
<computeroutput>bzip2 --
|
|
-myfilename</computeroutput>.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>--repetitive-fast</computeroutput></term>
|
|
<term><computeroutput>--repetitive-best</computeroutput></term>
|
|
<listitem><para>These flags are redundant in versions 0.9.5 and
|
|
above. They provided some coarse control over the behaviour of
|
|
the sorting algorithm in earlier versions, which was sometimes
|
|
useful. 0.9.5 and above have an improved algorithm which
|
|
renders these flags irrelevant.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT">
|
|
<title>MEMORY MANAGEMENT</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> compresses large
|
|
files in blocks. The block size affects both the compression
|
|
ratio achieved, and the amount of memory needed for compression
|
|
and decompression. The flags <computeroutput>-1</computeroutput>
|
|
through <computeroutput>-9</computeroutput> specify the block
|
|
size to be 100,000 bytes through 900,000 bytes (the default)
|
|
respectively. At decompression time, the block size used for
|
|
compression is read from the header of the compressed file, and
|
|
<computeroutput>bunzip2</computeroutput> then allocates itself
|
|
just enough memory to decompress the file. Since block sizes are
|
|
stored in compressed files, it follows that the flags
|
|
<computeroutput>-1</computeroutput> to
|
|
<computeroutput>-9</computeroutput> are irrelevant to and so
|
|
ignored during decompression.</para>
|
|
|
|
<para>Compression and decompression requirements, in bytes, can be
|
|
estimated as:</para>
|
|
<programlisting>
|
|
Compression: 400k + ( 8 x block size )
|
|
|
|
Decompression: 100k + ( 4 x block size ), or
|
|
100k + ( 2.5 x block size )
|
|
</programlisting>
|
|
|
|
<para>Larger block sizes give rapidly diminishing marginal
|
|
returns. Most of the compression comes from the first two or
|
|
three hundred k of block size, a fact worth bearing in mind when
|
|
using <computeroutput>bzip2</computeroutput> on small machines.
|
|
It is also important to appreciate that the decompression memory
|
|
requirement is set at compression time by the choice of block
|
|
size.</para>
|
|
|
|
<para>For files compressed with the default 900k block size,
|
|
<computeroutput>bunzip2</computeroutput> will require about 3700
|
|
kbytes to decompress. To support decompression of any file on a
|
|
4 megabyte machine, <computeroutput>bunzip2</computeroutput> has
|
|
an option to decompress using approximately half this amount of
|
|
memory, about 2300 kbytes. Decompression speed is also halved,
|
|
so you should use this option only where necessary. The relevant
|
|
flag is <computeroutput>-s</computeroutput>.</para>
|
|
|
|
<para>In general, try and use the largest block size memory
|
|
constraints allow, since that maximises the compression achieved.
|
|
Compression and decompression speed are virtually unaffected by
|
|
block size.</para>
|
|
|
|
<para>Another significant point applies to files which fit in a
|
|
single block -- that means most files you'd encounter using a
|
|
large block size. The amount of real memory touched is
|
|
proportional to the size of the file, since the file is smaller
|
|
than a block. For example, compressing a file 20,000 bytes long
|
|
with the flag <computeroutput>-9</computeroutput> will cause the
|
|
compressor to allocate around 7600k of memory, but only touch
|
|
400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor
|
|
will allocate 3700k but only touch 100k + 20000 * 4 = 180
|
|
kbytes.</para>
|
|
|
|
<para>Here is a table which summarises the maximum memory usage
|
|
for different block sizes. Also recorded is the total compressed
|
|
size for 14 files of the Calgary Text Compression Corpus
|
|
totalling 3,141,622 bytes. This column gives some feel for how
|
|
compression varies with block size. These figures tend to
|
|
understate the advantage of larger block sizes for larger files,
|
|
since the Corpus is dominated by smaller files.</para>
|
|
|
|
<programlisting>
|
|
Compress Decompress Decompress Corpus
|
|
Flag usage usage -s usage Size
|
|
|
|
-1 1200k 500k 350k 914704
|
|
-2 2000k 900k 600k 877703
|
|
-3 2800k 1300k 850k 860338
|
|
-4 3600k 1700k 1100k 846899
|
|
-5 4400k 2100k 1350k 845160
|
|
-6 5200k 2500k 1600k 838626
|
|
-7 6100k 2900k 1850k 834096
|
|
-8 6800k 3300k 2100k 828642
|
|
-9 7600k 3700k 2350k 828642
|
|
</programlisting>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES">
|
|
<title>RECOVERING DATA FROM DAMAGED FILES</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> compresses files in
|
|
blocks, usually 900kbytes long. Each block is handled
|
|
independently. If a media or transmission error causes a
|
|
multi-block <computeroutput>.bz2</computeroutput> file to become
|
|
damaged, it may be possible to recover data from the undamaged
|
|
blocks in the file.</para>
|
|
|
|
<para>The compressed representation of each block is delimited by
|
|
a 48-bit pattern, which makes it possible to find the block
|
|
boundaries with reasonable certainty. Each block also carries
|
|
its own 32-bit CRC, so damaged blocks can be distinguished from
|
|
undamaged ones.</para>
|
|
|
|
<para><computeroutput>bzip2recover</computeroutput> is a simple
|
|
program whose purpose is to search for blocks in
|
|
<computeroutput>.bz2</computeroutput> files, and write each block
|
|
out into its own <computeroutput>.bz2</computeroutput> file. You
|
|
can then use <computeroutput>bzip2 -t</computeroutput> to test
|
|
the integrity of the resulting files, and decompress those which
|
|
are undamaged.</para>
|
|
|
|
<para><computeroutput>bzip2recover</computeroutput> takes a
|
|
single argument, the name of the damaged file, and writes a
|
|
number of files <computeroutput>rec0001file.bz2</computeroutput>,
|
|
<computeroutput>rec0002file.bz2</computeroutput>, etc, containing
|
|
the extracted blocks. The output filenames are designed so that
|
|
the use of wildcards in subsequent processing -- for example,
|
|
<computeroutput>bzip2 -dc rec*file.bz2 >
|
|
recovered_data</computeroutput> -- lists the files in the correct
|
|
order.</para>
|
|
|
|
<para><computeroutput>bzip2recover</computeroutput> should be of
|
|
most use dealing with large <computeroutput>.bz2</computeroutput>
|
|
files, as these will contain many blocks. It is clearly futile
|
|
to use it on damaged single-block files, since a damaged block
|
|
cannot be recovered. If you wish to minimise any potential data
|
|
loss through media or transmission errors, you might consider
|
|
compressing with a smaller block size.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="performance" xreflabel="PERFORMANCE NOTES">
|
|
<title>PERFORMANCE NOTES</title>
|
|
|
|
<para>The sorting phase of compression gathers together similar
|
|
strings in the file. Because of this, files containing very long
|
|
runs of repeated symbols, like "aabaabaabaab ..." (repeated
|
|
several hundred times) may compress more slowly than normal.
|
|
Versions 0.9.5 and above fare much better than previous versions
|
|
in this respect. The ratio between worst-case and average-case
|
|
compression time is in the region of 10:1. For previous
|
|
versions, this figure was more like 100:1. You can use the
|
|
<computeroutput>-vvvv</computeroutput> option to monitor progress
|
|
in great detail, if you want.</para>
|
|
|
|
<para>Decompression speed is unaffected by these
|
|
phenomena.</para>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> usually allocates
|
|
several megabytes of memory to operate in, and then charges all
|
|
over it in a fairly random fashion. This means that performance,
|
|
both for compressing and decompressing, is largely determined by
|
|
the speed at which your machine can service cache misses.
|
|
Because of this, small changes to the code to reduce the miss
|
|
rate have been observed to give disproportionately large
|
|
performance improvements. I imagine
|
|
<computeroutput>bzip2</computeroutput> will perform best on
|
|
machines with very large caches.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="caveats" xreflabel="CAVEATS">
|
|
<title>CAVEATS</title>
|
|
|
|
<para>I/O error messages are not as helpful as they could be.
|
|
<computeroutput>bzip2</computeroutput> tries hard to detect I/O
|
|
errors and exit cleanly, but the details of what the problem is
|
|
sometimes seem rather misleading.</para>
|
|
|
|
<para>This manual page pertains to version &bz-version; of
|
|
<computeroutput>bzip2</computeroutput>. Compressed data created by
|
|
this version is entirely forwards and backwards compatible with the
|
|
previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
|
|
1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
|
|
above can correctly decompress multiple concatenated compressed files.
|
|
0.1pl2 cannot do this; it will stop after decompressing just the first
|
|
file in the stream.</para>
|
|
|
|
<para><computeroutput>bzip2recover</computeroutput> versions
|
|
prior to 1.0.2 used 32-bit integers to represent bit positions in
|
|
compressed files, so it could not handle compressed files more
|
|
than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints
|
|
on some platforms which support them (GNU supported targets, and
|
|
Windows). To establish whether or not
|
|
<computeroutput>bzip2recover</computeroutput> was built with such
|
|
a limitation, run it without arguments. In any event you can
|
|
build yourself an unlimited version if you can recompile it with
|
|
<computeroutput>MaybeUInt64</computeroutput> set to be an
|
|
unsigned 64-bit integer.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="author" xreflabel="AUTHOR">
|
|
<title>AUTHOR</title>
|
|
|
|
<para>Julian Seward,
|
|
<computeroutput>&bz-email;</computeroutput></para>
|
|
|
|
<para>The ideas embodied in
|
|
<computeroutput>bzip2</computeroutput> are due to (at least) the
|
|
following people: Michael Burrows and David Wheeler (for the
|
|
block sorting transformation), David Wheeler (again, for the
|
|
Huffman coder), Peter Fenwick (for the structured coding model in
|
|
the original <computeroutput>bzip</computeroutput>, and many
|
|
refinements), and Alistair Moffat, Radford Neal and Ian Witten
|
|
(for the arithmetic coder in the original
|
|
<computeroutput>bzip</computeroutput>). I am much indebted for
|
|
their help, support and advice. See the manual in the source
|
|
distribution for pointers to sources of documentation. Christian
|
|
von Roques encouraged me to look for faster sorting algorithms,
|
|
so as to speed up compression. Bela Lubkin encouraged me to
|
|
improve the worst-case compression performance.
|
|
Donna Robinson XMLised the documentation.
|
|
Many people sent
|
|
patches, helped with portability problems, lent machines, gave
|
|
advice and were generally helpful.</para>
|
|
|
|
</sect1>
|
|
|
|
</chapter>
|
|
|
|
|
|
|
|
<chapter id="libprog" xreflabel="Programming with libbzip2">
|
|
<title>
|
|
Programming with <computeroutput>libbzip2</computeroutput>
|
|
</title>
|
|
|
|
<para>This chapter describes the programming interface to
|
|
<computeroutput>libbzip2</computeroutput>.</para>
|
|
|
|
<para>For general background information, particularly about
|
|
memory use and performance aspects, you'd be well advised to read
|
|
<xref linkend="using"/> as well.</para>
|
|
|
|
|
|
<sect1 id="top-level" xreflabel="Top-level structure">
|
|
<title>Top-level structure</title>
|
|
|
|
<para><computeroutput>libbzip2</computeroutput> is a flexible
|
|
library for compressing and decompressing data in the
|
|
<computeroutput>bzip2</computeroutput> data format. Although
|
|
packaged as a single entity, it helps to regard the library as
|
|
three separate parts: the low level interface, and the high level
|
|
interface, and some utility functions.</para>
|
|
|
|
<para>The structure of
|
|
<computeroutput>libbzip2</computeroutput>'s interfaces is similar
|
|
to that of Jean-loup Gailly's and Mark Adler's excellent
|
|
<computeroutput>zlib</computeroutput> library.</para>
|
|
|
|
<para>All externally visible symbols have names beginning
|
|
<computeroutput>BZ2_</computeroutput>. This is new in version
|
|
1.0. The intention is to minimise pollution of the namespaces of
|
|
library clients.</para>
|
|
|
|
<para>To use any part of the library, you need to
|
|
<computeroutput>#include <bzlib.h></computeroutput>
|
|
into your sources.</para>
|
|
|
|
|
|
|
|
<sect2 id="ll-summary" xreflabel="Low-level summary">
|
|
<title>Low-level summary</title>
|
|
|
|
<para>This interface provides services for compressing and
|
|
decompressing data in memory. There's no provision for dealing
|
|
with files, streams or any other I/O mechanisms, just straight
|
|
memory-to-memory work. In fact, this part of the library can be
|
|
compiled without inclusion of
|
|
<computeroutput>stdio.h</computeroutput>, which may be helpful
|
|
for embedded applications.</para>
|
|
|
|
<para>The low-level part of the library has no global variables
|
|
and is therefore thread-safe.</para>
|
|
|
|
<para>Six routines make up the low level interface:
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>,
|
|
<computeroutput>BZ2_bzCompress</computeroutput>, and
|
|
<computeroutput>BZ2_bzCompressEnd</computeroutput> for
|
|
compression, and a corresponding trio
|
|
<computeroutput>BZ2_bzDecompressInit</computeroutput>,
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> and
|
|
<computeroutput>BZ2_bzDecompressEnd</computeroutput> for
|
|
decompression. The <computeroutput>*Init</computeroutput>
|
|
functions allocate memory for compression/decompression and do
|
|
other initialisations, whilst the
|
|
<computeroutput>*End</computeroutput> functions close down
|
|
operations and release memory.</para>
|
|
|
|
<para>The real work is done by
|
|
<computeroutput>BZ2_bzCompress</computeroutput> and
|
|
<computeroutput>BZ2_bzDecompress</computeroutput>. These
|
|
compress and decompress data from a user-supplied input buffer to
|
|
a user-supplied output buffer. These buffers can be any size;
|
|
arbitrary quantities of data are handled by making repeated calls
|
|
to these functions. This is a flexible mechanism allowing a
|
|
consumer-pull style of activity, or producer-push, or a mixture
|
|
of both.</para>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="hl-summary" xreflabel="High-level summary">
|
|
<title>High-level summary</title>
|
|
|
|
<para>This interface provides some handy wrappers around the
|
|
low-level interface to facilitate reading and writing
|
|
<computeroutput>bzip2</computeroutput> format files
|
|
(<computeroutput>.bz2</computeroutput> files). The routines
|
|
provide hooks to facilitate reading files in which the
|
|
<computeroutput>bzip2</computeroutput> data stream is embedded
|
|
within some larger-scale file structure, or where there are
|
|
multiple <computeroutput>bzip2</computeroutput> data streams
|
|
concatenated end-to-end.</para>
|
|
|
|
<para>For reading files,
|
|
<computeroutput>BZ2_bzReadOpen</computeroutput>,
|
|
<computeroutput>BZ2_bzRead</computeroutput>,
|
|
<computeroutput>BZ2_bzReadClose</computeroutput> and
|
|
<computeroutput>BZ2_bzReadGetUnused</computeroutput> are
|
|
supplied. For writing files,
|
|
<computeroutput>BZ2_bzWriteOpen</computeroutput>,
|
|
<computeroutput>BZ2_bzWrite</computeroutput> and
|
|
<computeroutput>BZ2_bzWriteFinish</computeroutput> are
|
|
available.</para>
|
|
|
|
<para>As with the low-level library, no global variables are used
|
|
so the library is per se thread-safe. However, if I/O errors
|
|
occur whilst reading or writing the underlying compressed files,
|
|
you may have to consult <computeroutput>errno</computeroutput> to
|
|
determine the cause of the error. In that case, you'd need a C
|
|
library which correctly supports
|
|
<computeroutput>errno</computeroutput> in a multithreaded
|
|
environment.</para>
|
|
|
|
<para>To make the library a little simpler and more portable,
|
|
<computeroutput>BZ2_bzReadOpen</computeroutput> and
|
|
<computeroutput>BZ2_bzWriteOpen</computeroutput> require you to
|
|
pass them file handles (<computeroutput>FILE*</computeroutput>s)
|
|
which have previously been opened for reading or writing
|
|
respectively. That avoids portability problems associated with
|
|
file operations and file attributes, whilst not being much of an
|
|
imposition on the programmer.</para>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="util-fns-summary" xreflabel="Utility functions summary">
|
|
<title>Utility functions summary</title>
|
|
|
|
<para>For very simple needs,
|
|
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
|
|
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are
|
|
provided. These compress data in memory from one buffer to
|
|
another buffer in a single function call. You should assess
|
|
whether these functions fulfill your memory-to-memory
|
|
compression/decompression requirements before investing effort in
|
|
understanding the more general but more complex low-level
|
|
interface.</para>
|
|
|
|
<para>Yoshioka Tsuneo
|
|
(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has
|
|
contributed some functions to give better
|
|
<computeroutput>zlib</computeroutput> compatibility. These
|
|
functions are <computeroutput>BZ2_bzopen</computeroutput>,
|
|
<computeroutput>BZ2_bzread</computeroutput>,
|
|
<computeroutput>BZ2_bzwrite</computeroutput>,
|
|
<computeroutput>BZ2_bzflush</computeroutput>,
|
|
<computeroutput>BZ2_bzclose</computeroutput>,
|
|
<computeroutput>BZ2_bzerror</computeroutput> and
|
|
<computeroutput>BZ2_bzlibVersion</computeroutput>. You may find
|
|
these functions more convenient for simple file reading and
|
|
writing, than those in the high-level interface. These functions
|
|
are not (yet) officially part of the library, and are minimally
|
|
documented here. If they break, you get to keep all the pieces.
|
|
I hope to document them properly when time permits.</para>
|
|
|
|
<para>Yoshioka also contributed modifications to allow the
|
|
library to be built as a Windows DLL.</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="err-handling" xreflabel="Error handling">
|
|
<title>Error handling</title>
|
|
|
|
<para>The library is designed to recover cleanly in all
|
|
situations, including the worst-case situation of decompressing
|
|
random data. I'm not 100% sure that it can always do this, so
|
|
you might want to add a signal handler to catch segmentation
|
|
violations during decompression if you are feeling especially
|
|
paranoid. I would be interested in hearing more about the
|
|
robustness of the library to corrupted compressed data.</para>
|
|
|
|
<para>Version 1.0.3 more robust in this respect than any
|
|
previous version. Investigations with Valgrind (a tool for detecting
|
|
problems with memory management) indicate
|
|
that, at least for the few files I tested, all single-bit errors
|
|
in the decompressed data are caught properly, with no
|
|
segmentation faults, no uses of uninitialised data, no out of
|
|
range reads or writes, and no infinite looping in the decompressor.
|
|
So it's certainly pretty robust, although
|
|
I wouldn't claim it to be totally bombproof.</para>
|
|
|
|
<para>The file <computeroutput>bzlib.h</computeroutput> contains
|
|
all definitions needed to use the library. In particular, you
|
|
should definitely not include
|
|
<computeroutput>bzlib_private.h</computeroutput>.</para>
|
|
|
|
<para>In <computeroutput>bzlib.h</computeroutput>, the various
|
|
return values are defined. The following list is not intended as
|
|
an exhaustive description of the circumstances in which a given
|
|
value may be returned -- those descriptions are given later.
|
|
Rather, it is intended to convey the rough meaning of each return
|
|
value. The first five actions are normal and not intended to
|
|
denote an error situation.</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_OK</computeroutput></term>
|
|
<listitem><para>The requested action was completed
|
|
successfully.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK,
|
|
BZ_FINISH_OK</computeroutput></term>
|
|
<listitem><para>In
|
|
<computeroutput>BZ2_bzCompress</computeroutput>, the requested
|
|
flush/finish/nothing-special action was completed
|
|
successfully.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_STREAM_END</computeroutput></term>
|
|
<listitem><para>Compression of data was completed, or the
|
|
logical stream end was detected during
|
|
decompression.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
<para>The following return values indicate an error of some
|
|
kind.</para>
|
|
|
|
<variablelist>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term>
|
|
<listitem><para>Indicates that the library has been improperly
|
|
compiled on your platform -- a major configuration error.
|
|
Specifically, it means that
|
|
<computeroutput>sizeof(char)</computeroutput>,
|
|
<computeroutput>sizeof(short)</computeroutput> and
|
|
<computeroutput>sizeof(int)</computeroutput> are not 1, 2 and
|
|
4 respectively, as they should be. Note that the library
|
|
should still work properly on 64-bit platforms which follow
|
|
the LP64 programming model -- that is, where
|
|
<computeroutput>sizeof(long)</computeroutput> and
|
|
<computeroutput>sizeof(void*)</computeroutput> are 8. Under
|
|
LP64, <computeroutput>sizeof(int)</computeroutput> is still 4,
|
|
so <computeroutput>libbzip2</computeroutput>, which doesn't
|
|
use the <computeroutput>long</computeroutput> type, is
|
|
OK.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term>
|
|
<listitem><para>When using the library, it is important to call
|
|
the functions in the correct sequence and with data structures
|
|
(buffers etc) in the correct states.
|
|
<computeroutput>libbzip2</computeroutput> checks as much as it
|
|
can to ensure this is happening, and returns
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not.
|
|
Code which complies precisely with the function semantics, as
|
|
detailed below, should never receive this value; such an event
|
|
denotes buggy code which you should
|
|
investigate.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_PARAM_ERROR</computeroutput></term>
|
|
<listitem><para>Returned when a parameter to a function call is
|
|
out of range or otherwise manifestly incorrect. As with
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this
|
|
denotes a bug in the client code. The distinction between
|
|
<computeroutput>BZ_PARAM_ERROR</computeroutput> and
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit
|
|
hazy, but still worth making.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_MEM_ERROR</computeroutput></term>
|
|
<listitem><para>Returned when a request to allocate memory
|
|
failed. Note that the quantity of memory needed to decompress
|
|
a stream cannot be determined until the stream's header has
|
|
been read. So
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> and
|
|
<computeroutput>BZ2_bzRead</computeroutput> may return
|
|
<computeroutput>BZ_MEM_ERROR</computeroutput> even though some
|
|
of the compressed data has been read. The same is not true
|
|
for compression; once
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput> or
|
|
<computeroutput>BZ2_bzWriteOpen</computeroutput> have
|
|
successfully completed,
|
|
<computeroutput>BZ_MEM_ERROR</computeroutput> cannot
|
|
occur.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_DATA_ERROR</computeroutput></term>
|
|
<listitem><para>Returned when a data integrity error is
|
|
detected during decompression. Most importantly, this means
|
|
when stored and computed CRCs for the data do not match. This
|
|
value is also returned upon detection of any other anomaly in
|
|
the compressed data.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term>
|
|
<listitem><para>As a special case of
|
|
<computeroutput>BZ_DATA_ERROR</computeroutput>, it is
|
|
sometimes useful to know when the compressed stream does not
|
|
start with the correct magic bytes (<computeroutput>'B' 'Z'
|
|
'h'</computeroutput>).</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_IO_ERROR</computeroutput></term>
|
|
<listitem><para>Returned by
|
|
<computeroutput>BZ2_bzRead</computeroutput> and
|
|
<computeroutput>BZ2_bzWrite</computeroutput> when there is an
|
|
error reading or writing in the compressed file, and by
|
|
<computeroutput>BZ2_bzReadOpen</computeroutput> and
|
|
<computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts
|
|
to use a file for which the error indicator (viz,
|
|
<computeroutput>ferror(f)</computeroutput>) is set. On
|
|
receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the
|
|
caller should consult <computeroutput>errno</computeroutput>
|
|
and/or <computeroutput>perror</computeroutput> to acquire
|
|
operating-system specific information about the
|
|
problem.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term>
|
|
<listitem><para>Returned by
|
|
<computeroutput>BZ2_bzRead</computeroutput> when the
|
|
compressed file finishes before the logical end of stream is
|
|
detected.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term>
|
|
<listitem><para>Returned by
|
|
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
|
|
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to
|
|
indicate that the output data will not fit into the output
|
|
buffer provided.</para></listitem>
|
|
</varlistentry>
|
|
|
|
</variablelist>
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="low-level" xreflabel=">Low-level interface">
|
|
<title>Low-level interface</title>
|
|
|
|
|
|
<sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit">
|
|
<title><computeroutput>BZ2_bzCompressInit</computeroutput></title>
|
|
|
|
<programlisting>
|
|
typedef struct {
|
|
char *next_in;
|
|
unsigned int avail_in;
|
|
unsigned int total_in_lo32;
|
|
unsigned int total_in_hi32;
|
|
|
|
char *next_out;
|
|
unsigned int avail_out;
|
|
unsigned int total_out_lo32;
|
|
unsigned int total_out_hi32;
|
|
|
|
void *state;
|
|
|
|
void *(*bzalloc)(void *,int,int);
|
|
void (*bzfree)(void *,void *);
|
|
void *opaque;
|
|
} bz_stream;
|
|
|
|
int BZ2_bzCompressInit ( bz_stream *strm,
|
|
int blockSize100k,
|
|
int verbosity,
|
|
int workFactor );
|
|
</programlisting>
|
|
|
|
<para>Prepares for compression. The
|
|
<computeroutput>bz_stream</computeroutput> structure holds all
|
|
data pertaining to the compression activity. A
|
|
<computeroutput>bz_stream</computeroutput> structure should be
|
|
allocated and initialised prior to the call. The fields of
|
|
<computeroutput>bz_stream</computeroutput> comprise the entirety
|
|
of the user-visible data. <computeroutput>state</computeroutput>
|
|
is a pointer to the private data structures required for
|
|
compression.</para>
|
|
|
|
<para>Custom memory allocators are supported, via fields
|
|
<computeroutput>bzalloc</computeroutput>,
|
|
<computeroutput>bzfree</computeroutput>, and
|
|
<computeroutput>opaque</computeroutput>. The value
|
|
<computeroutput>opaque</computeroutput> is passed to as the first
|
|
argument to all calls to <computeroutput>bzalloc</computeroutput>
|
|
and <computeroutput>bzfree</computeroutput>, but is otherwise
|
|
ignored by the library. The call <computeroutput>bzalloc (
|
|
opaque, n, m )</computeroutput> is expected to return a pointer
|
|
<computeroutput>p</computeroutput> to <computeroutput>n *
|
|
m</computeroutput> bytes of memory, and <computeroutput>bzfree (
|
|
opaque, p )</computeroutput> should free that memory.</para>
|
|
|
|
<para>If you don't want to use a custom memory allocator, set
|
|
<computeroutput>bzalloc</computeroutput>,
|
|
<computeroutput>bzfree</computeroutput> and
|
|
<computeroutput>opaque</computeroutput> to
|
|
<computeroutput>NULL</computeroutput>, and the library will then
|
|
use the standard <computeroutput>malloc</computeroutput> /
|
|
<computeroutput>free</computeroutput> routines.</para>
|
|
|
|
<para>Before calling
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>, fields
|
|
<computeroutput>bzalloc</computeroutput>,
|
|
<computeroutput>bzfree</computeroutput> and
|
|
<computeroutput>opaque</computeroutput> should be filled
|
|
appropriately, as just described. Upon return, the internal
|
|
state will have been allocated and initialised, and
|
|
<computeroutput>total_in_lo32</computeroutput>,
|
|
<computeroutput>total_in_hi32</computeroutput>,
|
|
<computeroutput>total_out_lo32</computeroutput> and
|
|
<computeroutput>total_out_hi32</computeroutput> will have been
|
|
set to zero. These four fields are used by the library to inform
|
|
the caller of the total amount of data passed into and out of the
|
|
library, respectively. You should not try to change them. As of
|
|
version 1.0, 64-bit counts are maintained, even on 32-bit
|
|
platforms, using the <computeroutput>_hi32</computeroutput>
|
|
fields to store the upper 32 bits of the count. So, for example,
|
|
the total amount of data in is <computeroutput>(total_in_hi32
|
|
<< 32) + total_in_lo32</computeroutput>.</para>
|
|
|
|
<para>Parameter <computeroutput>blockSize100k</computeroutput>
|
|
specifies the block size to be used for compression. It should
|
|
be a value between 1 and 9 inclusive, and the actual block size
|
|
used is 100000 x this figure. 9 gives the best compression but
|
|
takes most memory.</para>
|
|
|
|
<para>Parameter <computeroutput>verbosity</computeroutput> should
|
|
be set to a number between 0 and 4 inclusive. 0 is silent, and
|
|
greater numbers give increasingly verbose monitoring/debugging
|
|
output. If the library has been compiled with
|
|
<computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output
|
|
will appear for any verbosity setting.</para>
|
|
|
|
<para>Parameter <computeroutput>workFactor</computeroutput>
|
|
controls how the compression phase behaves when presented with
|
|
worst case, highly repetitive, input data. If compression runs
|
|
into difficulties caused by repetitive data, the library switches
|
|
from the standard sorting algorithm to a fallback algorithm. The
|
|
fallback is slower than the standard algorithm by perhaps a
|
|
factor of three, but always behaves reasonably, no matter how bad
|
|
the input.</para>
|
|
|
|
<para>Lower values of <computeroutput>workFactor</computeroutput>
|
|
reduce the amount of effort the standard algorithm will expend
|
|
before resorting to the fallback. You should set this parameter
|
|
carefully; too low, and many inputs will be handled by the
|
|
fallback algorithm and so compress rather slowly, too high, and
|
|
your average-to-worst case compression times can become very
|
|
large. The default value of 30 gives reasonable behaviour over a
|
|
wide range of circumstances.</para>
|
|
|
|
<para>Allowable values range from 0 to 250 inclusive. 0 is a
|
|
special case, equivalent to using the default value of 30.</para>
|
|
|
|
<para>Note that the compressed output generated is the same
|
|
regardless of whether or not the fallback algorithm is
|
|
used.</para>
|
|
|
|
<para>Be aware also that this parameter may disappear entirely in
|
|
future versions of the library. In principle it should be
|
|
possible to devise a good way to automatically choose which
|
|
algorithm to use. Such a mechanism would render the parameter
|
|
obsolete.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if strm is NULL
|
|
or blockSize < 1 or blockSize > 9
|
|
or verbosity < 0 or verbosity > 4
|
|
or workFactor < 0 or workFactor > 250
|
|
BZ_MEM_ERROR
|
|
if not enough memory is available
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzCompress
|
|
if BZ_OK is returned
|
|
no specific action needed in case of error
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzCompress" xreflabel="BZ2_bzCompress">
|
|
<title><computeroutput>BZ2_bzCompress</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzCompress ( bz_stream *strm, int action );
|
|
</programlisting>
|
|
|
|
<para>Provides more input and/or output buffer space for the
|
|
library. The caller maintains input and output buffers, and
|
|
calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer
|
|
data between them.</para>
|
|
|
|
<para>Before each call to
|
|
<computeroutput>BZ2_bzCompress</computeroutput>,
|
|
<computeroutput>next_in</computeroutput> should point at the data
|
|
to be compressed, and <computeroutput>avail_in</computeroutput>
|
|
should indicate how many bytes the library may read.
|
|
<computeroutput>BZ2_bzCompress</computeroutput> updates
|
|
<computeroutput>next_in</computeroutput>,
|
|
<computeroutput>avail_in</computeroutput> and
|
|
<computeroutput>total_in</computeroutput> to reflect the number
|
|
of bytes it has read.</para>
|
|
|
|
<para>Similarly, <computeroutput>next_out</computeroutput> should
|
|
point to a buffer in which the compressed data is to be placed,
|
|
with <computeroutput>avail_out</computeroutput> indicating how
|
|
much output space is available.
|
|
<computeroutput>BZ2_bzCompress</computeroutput> updates
|
|
<computeroutput>next_out</computeroutput>,
|
|
<computeroutput>avail_out</computeroutput> and
|
|
<computeroutput>total_out</computeroutput> to reflect the number
|
|
of bytes output.</para>
|
|
|
|
<para>You may provide and remove as little or as much data as you
|
|
like on each call of
|
|
<computeroutput>BZ2_bzCompress</computeroutput>. In the limit,
|
|
it is acceptable to supply and remove data one byte at a time,
|
|
although this would be terribly inefficient. You should always
|
|
ensure that at least one byte of output space is available at
|
|
each call.</para>
|
|
|
|
<para>A second purpose of
|
|
<computeroutput>BZ2_bzCompress</computeroutput> is to request a
|
|
change of mode of the compressed stream.</para>
|
|
|
|
<para>Conceptually, a compressed stream can be in one of four
|
|
states: IDLE, RUNNING, FLUSHING and FINISHING. Before
|
|
initialisation
|
|
(<computeroutput>BZ2_bzCompressInit</computeroutput>) and after
|
|
termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>),
|
|
a stream is regarded as IDLE.</para>
|
|
|
|
<para>Upon initialisation
|
|
(<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream
|
|
is placed in the RUNNING state. Subsequent calls to
|
|
<computeroutput>BZ2_bzCompress</computeroutput> should pass
|
|
<computeroutput>BZ_RUN</computeroutput> as the requested action;
|
|
other actions are illegal and will result in
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.</para>
|
|
|
|
<para>At some point, the calling program will have provided all
|
|
the input data it wants to. It will then want to finish up -- in
|
|
effect, asking the library to process any data it might have
|
|
buffered internally. In this state,
|
|
<computeroutput>BZ2_bzCompress</computeroutput> will no longer
|
|
attempt to read data from
|
|
<computeroutput>next_in</computeroutput>, but it will want to
|
|
write data to <computeroutput>next_out</computeroutput>. Because
|
|
the output buffer supplied by the user can be arbitrarily small,
|
|
the finishing-up operation cannot necessarily be done with a
|
|
single call of
|
|
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
|
|
|
|
<para>Instead, the calling program passes
|
|
<computeroutput>BZ_FINISH</computeroutput> as an action to
|
|
<computeroutput>BZ2_bzCompress</computeroutput>. This changes
|
|
the stream's state to FINISHING. Any remaining input (ie,
|
|
<computeroutput>next_in[0 .. avail_in-1]</computeroutput>) is
|
|
compressed and transferred to the output buffer. To do this,
|
|
<computeroutput>BZ2_bzCompress</computeroutput> must be called
|
|
repeatedly until all the output has been consumed. At that
|
|
point, <computeroutput>BZ2_bzCompress</computeroutput> returns
|
|
<computeroutput>BZ_STREAM_END</computeroutput>, and the stream's
|
|
state is set back to IDLE.
|
|
<computeroutput>BZ2_bzCompressEnd</computeroutput> should then be
|
|
called.</para>
|
|
|
|
<para>Just to make sure the calling program does not cheat, the
|
|
library makes a note of <computeroutput>avail_in</computeroutput>
|
|
at the time of the first call to
|
|
<computeroutput>BZ2_bzCompress</computeroutput> which has
|
|
<computeroutput>BZ_FINISH</computeroutput> as an action (ie, at
|
|
the time the program has announced its intention to not supply
|
|
any more input). By comparing this value with that of
|
|
<computeroutput>avail_in</computeroutput> over subsequent calls
|
|
to <computeroutput>BZ2_bzCompress</computeroutput>, the library
|
|
can detect any attempts to slip in more data to compress. Any
|
|
calls for which this is detected will return
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>. This
|
|
indicates a programming mistake which should be corrected.</para>
|
|
|
|
<para>Instead of asking to finish, the calling program may ask
|
|
<computeroutput>BZ2_bzCompress</computeroutput> to take all the
|
|
remaining input, compress it and terminate the current
|
|
(Burrows-Wheeler) compression block. This could be useful for
|
|
error control purposes. The mechanism is analogous to that for
|
|
finishing: call <computeroutput>BZ2_bzCompress</computeroutput>
|
|
with an action of <computeroutput>BZ_FLUSH</computeroutput>,
|
|
remove output data, and persist with the
|
|
<computeroutput>BZ_FLUSH</computeroutput> action until the value
|
|
<computeroutput>BZ_RUN</computeroutput> is returned. As with
|
|
finishing, <computeroutput>BZ2_bzCompress</computeroutput>
|
|
detects any attempt to provide more input data once the flush has
|
|
begun.</para>
|
|
|
|
<para>Once the flush is complete, the stream returns to the
|
|
normal RUNNING state.</para>
|
|
|
|
<para>This all sounds pretty complex, but isn't really. Here's a
|
|
table which shows which actions are allowable in each state, what
|
|
action will be taken, what the next state is, and what the
|
|
non-error return values are. Note that you can't explicitly ask
|
|
what state the stream is in, but nor do you need to -- it can be
|
|
inferred from the values returned by
|
|
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
|
|
|
|
<programlisting>
|
|
IDLE/any
|
|
Illegal. IDLE state only exists after BZ2_bzCompressEnd or
|
|
before BZ2_bzCompressInit.
|
|
Return value = BZ_SEQUENCE_ERROR
|
|
|
|
RUNNING/BZ_RUN
|
|
Compress from next_in to next_out as much as possible.
|
|
Next state = RUNNING
|
|
Return value = BZ_RUN_OK
|
|
|
|
RUNNING/BZ_FLUSH
|
|
Remember current value of next_in. Compress from next_in
|
|
to next_out as much as possible, but do not accept any more input.
|
|
Next state = FLUSHING
|
|
Return value = BZ_FLUSH_OK
|
|
|
|
RUNNING/BZ_FINISH
|
|
Remember current value of next_in. Compress from next_in
|
|
to next_out as much as possible, but do not accept any more input.
|
|
Next state = FINISHING
|
|
Return value = BZ_FINISH_OK
|
|
|
|
FLUSHING/BZ_FLUSH
|
|
Compress from next_in to next_out as much as possible,
|
|
but do not accept any more input.
|
|
If all the existing input has been used up and all compressed
|
|
output has been removed
|
|
Next state = RUNNING; Return value = BZ_RUN_OK
|
|
else
|
|
Next state = FLUSHING; Return value = BZ_FLUSH_OK
|
|
|
|
FLUSHING/other
|
|
Illegal.
|
|
Return value = BZ_SEQUENCE_ERROR
|
|
|
|
FINISHING/BZ_FINISH
|
|
Compress from next_in to next_out as much as possible,
|
|
but to not accept any more input.
|
|
If all the existing input has been used up and all compressed
|
|
output has been removed
|
|
Next state = IDLE; Return value = BZ_STREAM_END
|
|
else
|
|
Next state = FINISHING; Return value = BZ_FINISH_OK
|
|
|
|
FINISHING/other
|
|
Illegal.
|
|
Return value = BZ_SEQUENCE_ERROR
|
|
</programlisting>
|
|
|
|
|
|
<para>That still looks complicated? Well, fair enough. The
|
|
usual sequence of calls for compressing a load of data is:</para>
|
|
|
|
<orderedlist>
|
|
|
|
<listitem><para>Get started with
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para></listitem>
|
|
|
|
<listitem><para>Shovel data in and shlurp out its compressed form
|
|
using zero or more calls of
|
|
<computeroutput>BZ2_bzCompress</computeroutput> with action =
|
|
<computeroutput>BZ_RUN</computeroutput>.</para></listitem>
|
|
|
|
<listitem><para>Finish up. Repeatedly call
|
|
<computeroutput>BZ2_bzCompress</computeroutput> with action =
|
|
<computeroutput>BZ_FINISH</computeroutput>, copying out the
|
|
compressed output, until
|
|
<computeroutput>BZ_STREAM_END</computeroutput> is
|
|
returned.</para></listitem> <listitem><para>Close up and go home. Call
|
|
<computeroutput>BZ2_bzCompressEnd</computeroutput>.</para></listitem>
|
|
|
|
</orderedlist>
|
|
|
|
<para>If the data you want to compress fits into your input
|
|
buffer all at once, you can skip the calls of
|
|
<computeroutput>BZ2_bzCompress ( ..., BZ_RUN )</computeroutput>
|
|
and just do the <computeroutput>BZ2_bzCompress ( ..., BZ_FINISH
|
|
)</computeroutput> calls.</para>
|
|
|
|
<para>All required memory is allocated by
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>. The
|
|
compression library can accept any data at all (obviously). So
|
|
you shouldn't get any error return values from the
|
|
<computeroutput>BZ2_bzCompress</computeroutput> calls. If you
|
|
do, they will be
|
|
<computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, and indicate
|
|
a bug in your programming.</para>
|
|
|
|
<para>Trivial other possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if strm is NULL, or strm->s is NULL
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzCompress-end" xreflabel="BZ2_bzCompressEnd">
|
|
<title><computeroutput>BZ2_bzCompressEnd</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzCompressEnd ( bz_stream *strm );
|
|
</programlisting>
|
|
|
|
<para>Releases all memory associated with a compression
|
|
stream.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR if strm is NULL or strm->s is NULL
|
|
BZ_OK otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzDecompress-init" xreflabel="BZ2_bzDecompressInit">
|
|
<title><computeroutput>BZ2_bzDecompressInit</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );
|
|
</programlisting>
|
|
|
|
<para>Prepares for decompression. As with
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>, a
|
|
<computeroutput>bz_stream</computeroutput> record should be
|
|
allocated and initialised before the call. Fields
|
|
<computeroutput>bzalloc</computeroutput>,
|
|
<computeroutput>bzfree</computeroutput> and
|
|
<computeroutput>opaque</computeroutput> should be set if a custom
|
|
memory allocator is required, or made
|
|
<computeroutput>NULL</computeroutput> for the normal
|
|
<computeroutput>malloc</computeroutput> /
|
|
<computeroutput>free</computeroutput> routines. Upon return, the
|
|
internal state will have been initialised, and
|
|
<computeroutput>total_in</computeroutput> and
|
|
<computeroutput>total_out</computeroutput> will be zero.</para>
|
|
|
|
<para>For the meaning of parameter
|
|
<computeroutput>verbosity</computeroutput>, see
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
|
|
|
|
<para>If <computeroutput>small</computeroutput> is nonzero, the
|
|
library will use an alternative decompression algorithm which
|
|
uses less memory but at the cost of decompressing more slowly
|
|
(roughly speaking, half the speed, but the maximum memory
|
|
requirement drops to around 2300k). See <xref linkend="using"/>
|
|
for more information on memory management.</para>
|
|
|
|
<para>Note that the amount of memory needed to decompress a
|
|
stream cannot be determined until the stream's header has been
|
|
read, so even if
|
|
<computeroutput>BZ2_bzDecompressInit</computeroutput> succeeds, a
|
|
subsequent <computeroutput>BZ2_bzDecompress</computeroutput>
|
|
could fail with
|
|
<computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if ( small != 0 && small != 1 )
|
|
or (verbosity <; 0 || verbosity > 4)
|
|
BZ_MEM_ERROR
|
|
if insufficient memory is available
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzDecompress
|
|
if BZ_OK was returned
|
|
no specific action required in case of error
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzDecompress" xreflabel="BZ2_bzDecompress">
|
|
<title><computeroutput>BZ2_bzDecompress</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzDecompress ( bz_stream *strm );
|
|
</programlisting>
|
|
|
|
<para>Provides more input and/out output buffer space for the
|
|
library. The caller maintains input and output buffers, and uses
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> to transfer
|
|
data between them.</para>
|
|
|
|
<para>Before each call to
|
|
<computeroutput>BZ2_bzDecompress</computeroutput>,
|
|
<computeroutput>next_in</computeroutput> should point at the
|
|
compressed data, and <computeroutput>avail_in</computeroutput>
|
|
should indicate how many bytes the library may read.
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> updates
|
|
<computeroutput>next_in</computeroutput>,
|
|
<computeroutput>avail_in</computeroutput> and
|
|
<computeroutput>total_in</computeroutput> to reflect the number
|
|
of bytes it has read.</para>
|
|
|
|
<para>Similarly, <computeroutput>next_out</computeroutput> should
|
|
point to a buffer in which the uncompressed output is to be
|
|
placed, with <computeroutput>avail_out</computeroutput>
|
|
indicating how much output space is available.
|
|
<computeroutput>BZ2_bzCompress</computeroutput> updates
|
|
<computeroutput>next_out</computeroutput>,
|
|
<computeroutput>avail_out</computeroutput> and
|
|
<computeroutput>total_out</computeroutput> to reflect the number
|
|
of bytes output.</para>
|
|
|
|
<para>You may provide and remove as little or as much data as you
|
|
like on each call of
|
|
<computeroutput>BZ2_bzDecompress</computeroutput>. In the limit,
|
|
it is acceptable to supply and remove data one byte at a time,
|
|
although this would be terribly inefficient. You should always
|
|
ensure that at least one byte of output space is available at
|
|
each call.</para>
|
|
|
|
<para>Use of <computeroutput>BZ2_bzDecompress</computeroutput> is
|
|
simpler than
|
|
<computeroutput>BZ2_bzCompress</computeroutput>.</para>
|
|
|
|
<para>You should provide input and remove output as described
|
|
above, and repeatedly call
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> until
|
|
<computeroutput>BZ_STREAM_END</computeroutput> is returned.
|
|
Appearance of <computeroutput>BZ_STREAM_END</computeroutput>
|
|
denotes that <computeroutput>BZ2_bzDecompress</computeroutput>
|
|
has detected the logical end of the compressed stream.
|
|
<computeroutput>BZ2_bzDecompress</computeroutput> will not
|
|
produce <computeroutput>BZ_STREAM_END</computeroutput> until all
|
|
output data has been placed into the output buffer, so once
|
|
<computeroutput>BZ_STREAM_END</computeroutput> appears, you are
|
|
guaranteed to have available all the decompressed output, and
|
|
<computeroutput>BZ2_bzDecompressEnd</computeroutput> can safely
|
|
be called.</para>
|
|
|
|
<para>If case of an error return value, you should call
|
|
<computeroutput>BZ2_bzDecompressEnd</computeroutput> to clean up
|
|
and release memory.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if strm is NULL or strm->s is NULL
|
|
or strm->avail_out < 1
|
|
BZ_DATA_ERROR
|
|
if a data integrity error is detected in the compressed stream
|
|
BZ_DATA_ERROR_MAGIC
|
|
if the compressed stream doesn't begin with the right magic bytes
|
|
BZ_MEM_ERROR
|
|
if there wasn't enough memory available
|
|
BZ_STREAM_END
|
|
if the logical end of the data stream was detected and all
|
|
output in has been consumed, eg s-->avail_out > 0
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzDecompress
|
|
if BZ_OK was returned
|
|
BZ2_bzDecompressEnd
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzDecompress-end" xreflabel="BZ2_bzDecompressEnd">
|
|
<title><computeroutput>BZ2_bzDecompressEnd</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzDecompressEnd ( bz_stream *strm );
|
|
</programlisting>
|
|
|
|
<para>Releases all memory associated with a decompression
|
|
stream.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if strm is NULL or strm->s is NULL
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
None.
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="hl-interface" xreflabel="High-level interface">
|
|
<title>High-level interface</title>
|
|
|
|
<para>This interface provides functions for reading and writing
|
|
<computeroutput>bzip2</computeroutput> format files. First, some
|
|
general points.</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>All of the functions take an
|
|
<computeroutput>int*</computeroutput> first argument,
|
|
<computeroutput>bzerror</computeroutput>. After each call,
|
|
<computeroutput>bzerror</computeroutput> should be consulted
|
|
first to determine the outcome of the call. If
|
|
<computeroutput>bzerror</computeroutput> is
|
|
<computeroutput>BZ_OK</computeroutput>, the call completed
|
|
successfully, and only then should the return value of the
|
|
function (if any) be consulted. If
|
|
<computeroutput>bzerror</computeroutput> is
|
|
<computeroutput>BZ_IO_ERROR</computeroutput>, there was an
|
|
error reading/writing the underlying compressed file, and you
|
|
should then consult <computeroutput>errno</computeroutput> /
|
|
<computeroutput>perror</computeroutput> to determine the cause
|
|
of the difficulty. <computeroutput>bzerror</computeroutput>
|
|
may also be set to various other values; precise details are
|
|
given on a per-function basis below.</para></listitem>
|
|
|
|
<listitem><para>If <computeroutput>bzerror</computeroutput> indicates
|
|
an error (ie, anything except
|
|
<computeroutput>BZ_OK</computeroutput> and
|
|
<computeroutput>BZ_STREAM_END</computeroutput>), you should
|
|
immediately call
|
|
<computeroutput>BZ2_bzReadClose</computeroutput> (or
|
|
<computeroutput>BZ2_bzWriteClose</computeroutput>, depending on
|
|
whether you are attempting to read or to write) to free up all
|
|
resources associated with the stream. Once an error has been
|
|
indicated, behaviour of all calls except
|
|
<computeroutput>BZ2_bzReadClose</computeroutput>
|
|
(<computeroutput>BZ2_bzWriteClose</computeroutput>) is
|
|
undefined. The implication is that (1)
|
|
<computeroutput>bzerror</computeroutput> should be checked
|
|
after each call, and (2) if
|
|
<computeroutput>bzerror</computeroutput> indicates an error,
|
|
<computeroutput>BZ2_bzReadClose</computeroutput>
|
|
(<computeroutput>BZ2_bzWriteClose</computeroutput>) should then
|
|
be called to clean up.</para></listitem>
|
|
|
|
<listitem><para>The <computeroutput>FILE*</computeroutput> arguments
|
|
passed to <computeroutput>BZ2_bzReadOpen</computeroutput> /
|
|
<computeroutput>BZ2_bzWriteOpen</computeroutput> should be set
|
|
to binary mode. Most Unix systems will do this by default, but
|
|
other platforms, including Windows and Mac, will not. If you
|
|
omit this, you may encounter problems when moving code to new
|
|
platforms.</para></listitem>
|
|
|
|
<listitem><para>Memory allocation requests are handled by
|
|
<computeroutput>malloc</computeroutput> /
|
|
<computeroutput>free</computeroutput>. At present there is no
|
|
facility for user-defined memory allocators in the file I/O
|
|
functions (could easily be added, though).</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
|
|
|
|
<sect2 id="bzreadopen" xreflabel="BZ2_bzReadOpen">
|
|
<title><computeroutput>BZ2_bzReadOpen</computeroutput></title>
|
|
|
|
<programlisting>
|
|
typedef void BZFILE;
|
|
|
|
BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f,
|
|
int verbosity, int small,
|
|
void *unused, int nUnused );
|
|
</programlisting>
|
|
|
|
<para>Prepare to read compressed data from file handle
|
|
<computeroutput>f</computeroutput>.
|
|
<computeroutput>f</computeroutput> should refer to a file which
|
|
has been opened for reading, and for which the error indicator
|
|
(<computeroutput>ferror(f)</computeroutput>)is not set. If
|
|
<computeroutput>small</computeroutput> is 1, the library will try
|
|
to decompress using less memory, at the expense of speed.</para>
|
|
|
|
<para>For reasons explained below,
|
|
<computeroutput>BZ2_bzRead</computeroutput> will decompress the
|
|
<computeroutput>nUnused</computeroutput> bytes starting at
|
|
<computeroutput>unused</computeroutput>, before starting to read
|
|
from the file <computeroutput>f</computeroutput>. At most
|
|
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes may be
|
|
supplied like this. If this facility is not required, you should
|
|
pass <computeroutput>NULL</computeroutput> and
|
|
<computeroutput>0</computeroutput> for
|
|
<computeroutput>unused</computeroutput> and
|
|
n<computeroutput>Unused</computeroutput> respectively.</para>
|
|
|
|
<para>For the meaning of parameters
|
|
<computeroutput>small</computeroutput> and
|
|
<computeroutput>verbosity</computeroutput>, see
|
|
<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
|
|
|
|
<para>The amount of memory needed to decompress a file cannot be
|
|
determined until the file's header has been read. So it is
|
|
possible that <computeroutput>BZ2_bzReadOpen</computeroutput>
|
|
returns <computeroutput>BZ_OK</computeroutput> but a subsequent
|
|
call of <computeroutput>BZ2_bzRead</computeroutput> will return
|
|
<computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if f is NULL
|
|
or small is neither 0 nor 1
|
|
or ( unused == NULL && nUnused != 0 )
|
|
or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) )
|
|
BZ_IO_ERROR
|
|
if ferror(f) is nonzero
|
|
BZ_MEM_ERROR
|
|
if insufficient memory is available
|
|
BZ_OK
|
|
otherwise.
|
|
</programlisting>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
Pointer to an abstract BZFILE
|
|
if bzerror is BZ_OK
|
|
NULL
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzRead
|
|
if bzerror is BZ_OK
|
|
BZ2_bzClose
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzread" xreflabel="BZ2_bzRead">
|
|
<title><computeroutput>BZ2_bzRead</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );
|
|
</programlisting>
|
|
|
|
<para>Reads up to <computeroutput>len</computeroutput>
|
|
(uncompressed) bytes from the compressed file
|
|
<computeroutput>b</computeroutput> into the buffer
|
|
<computeroutput>buf</computeroutput>. If the read was
|
|
successful, <computeroutput>bzerror</computeroutput> is set to
|
|
<computeroutput>BZ_OK</computeroutput> and the number of bytes
|
|
read is returned. If the logical end-of-stream was detected,
|
|
<computeroutput>bzerror</computeroutput> will be set to
|
|
<computeroutput>BZ_STREAM_END</computeroutput>, and the number of
|
|
bytes read is returned. All other
|
|
<computeroutput>bzerror</computeroutput> values denote an
|
|
error.</para>
|
|
|
|
<para><computeroutput>BZ2_bzRead</computeroutput> will supply
|
|
<computeroutput>len</computeroutput> bytes, unless the logical
|
|
stream end is detected or an error occurs. Because of this, it
|
|
is possible to detect the stream end by observing when the number
|
|
of bytes returned is less than the number requested.
|
|
Nevertheless, this is regarded as inadvisable; you should instead
|
|
check <computeroutput>bzerror</computeroutput> after every call
|
|
and watch out for
|
|
<computeroutput>BZ_STREAM_END</computeroutput>.</para>
|
|
|
|
<para>Internally, <computeroutput>BZ2_bzRead</computeroutput>
|
|
copies data from the compressed file in chunks of size
|
|
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes before
|
|
decompressing it. If the file contains more bytes than strictly
|
|
needed to reach the logical end-of-stream,
|
|
<computeroutput>BZ2_bzRead</computeroutput> will almost certainly
|
|
read some of the trailing data before signalling
|
|
<computeroutput>BZ_SEQUENCE_END</computeroutput>. To collect the
|
|
read but unused data once
|
|
<computeroutput>BZ_SEQUENCE_END</computeroutput> has appeared,
|
|
call <computeroutput>BZ2_bzReadGetUnused</computeroutput>
|
|
immediately before
|
|
<computeroutput>BZ2_bzReadClose</computeroutput>.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if b is NULL or buf is NULL or len < 0
|
|
BZ_SEQUENCE_ERROR
|
|
if b was opened with BZ2_bzWriteOpen
|
|
BZ_IO_ERROR
|
|
if there is an error reading from the compressed file
|
|
BZ_UNEXPECTED_EOF
|
|
if the compressed file ended before
|
|
the logical end-of-stream was detected
|
|
BZ_DATA_ERROR
|
|
if a data integrity error was detected in the compressed stream
|
|
BZ_DATA_ERROR_MAGIC
|
|
if the stream does not begin with the requisite header bytes
|
|
(ie, is not a bzip2 data file). This is really
|
|
a special case of BZ_DATA_ERROR.
|
|
BZ_MEM_ERROR
|
|
if insufficient memory was available
|
|
BZ_STREAM_END
|
|
if the logical end of stream was detected.
|
|
BZ_OK
|
|
otherwise.
|
|
</programlisting>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
number of bytes read
|
|
if bzerror is BZ_OK or BZ_STREAM_END
|
|
undefined
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
collect data from buf, then BZ2_bzRead or BZ2_bzReadClose
|
|
if bzerror is BZ_OK
|
|
collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused
|
|
if bzerror is BZ_SEQUENCE_END
|
|
BZ2_bzReadClose
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzreadgetunused" xreflabel="BZ2_bzReadGetUnused">
|
|
<title><computeroutput>BZ2_bzReadGetUnused</computeroutput></title>
|
|
|
|
<programlisting>
|
|
void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b,
|
|
void** unused, int* nUnused );
|
|
</programlisting>
|
|
|
|
<para>Returns data which was read from the compressed file but
|
|
was not needed to get to the logical end-of-stream.
|
|
<computeroutput>*unused</computeroutput> is set to the address of
|
|
the data, and <computeroutput>*nUnused</computeroutput> to the
|
|
number of bytes. <computeroutput>*nUnused</computeroutput> will
|
|
be set to a value between <computeroutput>0</computeroutput> and
|
|
<computeroutput>BZ_MAX_UNUSED</computeroutput> inclusive.</para>
|
|
|
|
<para>This function may only be called once
|
|
<computeroutput>BZ2_bzRead</computeroutput> has signalled
|
|
<computeroutput>BZ_STREAM_END</computeroutput> but before
|
|
<computeroutput>BZ2_bzReadClose</computeroutput>.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if b is NULL
|
|
or unused is NULL or nUnused is NULL
|
|
BZ_SEQUENCE_ERROR
|
|
if BZ_STREAM_END has not been signalled
|
|
or if b was opened with BZ2_bzWriteOpen
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzReadClose
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzreadclose" xreflabel="BZ2_bzReadClose">
|
|
<title><computeroutput>BZ2_bzReadClose</computeroutput></title>
|
|
|
|
<programlisting>
|
|
void BZ2_bzReadClose ( int *bzerror, BZFILE *b );
|
|
</programlisting>
|
|
|
|
<para>Releases all memory pertaining to the compressed file
|
|
<computeroutput>b</computeroutput>.
|
|
<computeroutput>BZ2_bzReadClose</computeroutput> does not call
|
|
<computeroutput>fclose</computeroutput> on the underlying file
|
|
handle, so you should do that yourself if appropriate.
|
|
<computeroutput>BZ2_bzReadClose</computeroutput> should be called
|
|
to clean up after all error situations.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_SEQUENCE_ERROR
|
|
if b was opened with BZ2_bzOpenWrite
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
none
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzwriteopen" xreflabel="BZ2_bzWriteOpen">
|
|
<title><computeroutput>BZ2_bzWriteOpen</computeroutput></title>
|
|
|
|
<programlisting>
|
|
BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f,
|
|
int blockSize100k, int verbosity,
|
|
int workFactor );
|
|
</programlisting>
|
|
|
|
<para>Prepare to write compressed data to file handle
|
|
<computeroutput>f</computeroutput>.
|
|
<computeroutput>f</computeroutput> should refer to a file which
|
|
has been opened for writing, and for which the error indicator
|
|
(<computeroutput>ferror(f)</computeroutput>)is not set.</para>
|
|
|
|
<para>For the meaning of parameters
|
|
<computeroutput>blockSize100k</computeroutput>,
|
|
<computeroutput>verbosity</computeroutput> and
|
|
<computeroutput>workFactor</computeroutput>, see
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
|
|
|
|
<para>All required memory is allocated at this stage, so if the
|
|
call completes successfully,
|
|
<computeroutput>BZ_MEM_ERROR</computeroutput> cannot be signalled
|
|
by a subsequent call to
|
|
<computeroutput>BZ2_bzWrite</computeroutput>.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if f is NULL
|
|
or blockSize100k < 1 or blockSize100k > 9
|
|
BZ_IO_ERROR
|
|
if ferror(f) is nonzero
|
|
BZ_MEM_ERROR
|
|
if insufficient memory is available
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
Pointer to an abstract BZFILE
|
|
if bzerror is BZ_OK
|
|
NULL
|
|
otherwise
|
|
</programlisting>
|
|
|
|
<para>Allowable next actions:</para>
|
|
|
|
<programlisting>
|
|
BZ2_bzWrite
|
|
if bzerror is BZ_OK
|
|
(you could go directly to BZ2_bzWriteClose, but this would be pretty pointless)
|
|
BZ2_bzWriteClose
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzwrite" xreflabel="BZ2_bzWrite">
|
|
<title><computeroutput>BZ2_bzWrite</computeroutput></title>
|
|
|
|
<programlisting>
|
|
void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );
|
|
</programlisting>
|
|
|
|
<para>Absorbs <computeroutput>len</computeroutput> bytes from the
|
|
buffer <computeroutput>buf</computeroutput>, eventually to be
|
|
compressed and written to the file.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_PARAM_ERROR
|
|
if b is NULL or buf is NULL or len < 0
|
|
BZ_SEQUENCE_ERROR
|
|
if b was opened with BZ2_bzReadOpen
|
|
BZ_IO_ERROR
|
|
if there is an error writing the compressed file.
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzwriteclose" xreflabel="BZ2_bzWriteClose">
|
|
<title><computeroutput>BZ2_bzWriteClose</computeroutput></title>
|
|
|
|
<programlisting>
|
|
void BZ2_bzWriteClose( int *bzerror, BZFILE* f,
|
|
int abandon,
|
|
unsigned int* nbytes_in,
|
|
unsigned int* nbytes_out );
|
|
|
|
void BZ2_bzWriteClose64( int *bzerror, BZFILE* f,
|
|
int abandon,
|
|
unsigned int* nbytes_in_lo32,
|
|
unsigned int* nbytes_in_hi32,
|
|
unsigned int* nbytes_out_lo32,
|
|
unsigned int* nbytes_out_hi32 );
|
|
</programlisting>
|
|
|
|
<para>Compresses and flushes to the compressed file all data so
|
|
far supplied by <computeroutput>BZ2_bzWrite</computeroutput>.
|
|
The logical end-of-stream markers are also written, so subsequent
|
|
calls to <computeroutput>BZ2_bzWrite</computeroutput> are
|
|
illegal. All memory associated with the compressed file
|
|
<computeroutput>b</computeroutput> is released.
|
|
<computeroutput>fflush</computeroutput> is called on the
|
|
compressed file, but it is not
|
|
<computeroutput>fclose</computeroutput>'d.</para>
|
|
|
|
<para>If <computeroutput>BZ2_bzWriteClose</computeroutput> is
|
|
called to clean up after an error, the only action is to release
|
|
the memory. The library records the error codes issued by
|
|
previous calls, so this situation will be detected automatically.
|
|
There is no attempt to complete the compression operation, nor to
|
|
<computeroutput>fflush</computeroutput> the compressed file. You
|
|
can force this behaviour to happen even in the case of no error,
|
|
by passing a nonzero value to
|
|
<computeroutput>abandon</computeroutput>.</para>
|
|
|
|
<para>If <computeroutput>nbytes_in</computeroutput> is non-null,
|
|
<computeroutput>*nbytes_in</computeroutput> will be set to be the
|
|
total volume of uncompressed data handled. Similarly,
|
|
<computeroutput>nbytes_out</computeroutput> will be set to the
|
|
total volume of compressed data written. For compatibility with
|
|
older versions of the library,
|
|
<computeroutput>BZ2_bzWriteClose</computeroutput> only yields the
|
|
lower 32 bits of these counts. Use
|
|
<computeroutput>BZ2_bzWriteClose64</computeroutput> if you want
|
|
the full 64 bit counts. These two functions are otherwise
|
|
absolutely identical.</para>
|
|
|
|
<para>Possible assignments to
|
|
<computeroutput>bzerror</computeroutput>:</para>
|
|
|
|
<programlisting>
|
|
BZ_SEQUENCE_ERROR
|
|
if b was opened with BZ2_bzReadOpen
|
|
BZ_IO_ERROR
|
|
if there is an error writing the compressed file
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="embed" xreflabel="Handling embedded compressed data streams">
|
|
<title>Handling embedded compressed data streams</title>
|
|
|
|
<para>The high-level library facilitates use of
|
|
<computeroutput>bzip2</computeroutput> data streams which form
|
|
some part of a surrounding, larger data stream.</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>For writing, the library takes an open file handle,
|
|
writes compressed data to it,
|
|
<computeroutput>fflush</computeroutput>es it but does not
|
|
<computeroutput>fclose</computeroutput> it. The calling
|
|
application can write its own data before and after the
|
|
compressed data stream, using that same file handle.</para></listitem>
|
|
|
|
<listitem><para>Reading is more complex, and the facilities are not as
|
|
general as they could be since generality is hard to reconcile
|
|
with efficiency. <computeroutput>BZ2_bzRead</computeroutput>
|
|
reads from the compressed file in blocks of size
|
|
<computeroutput>BZ_MAX_UNUSED</computeroutput> bytes, and in
|
|
doing so probably will overshoot the logical end of compressed
|
|
stream. To recover this data once decompression has ended,
|
|
call <computeroutput>BZ2_bzReadGetUnused</computeroutput> after
|
|
the last call of <computeroutput>BZ2_bzRead</computeroutput>
|
|
(the one returning
|
|
<computeroutput>BZ_STREAM_END</computeroutput>) but before
|
|
calling
|
|
<computeroutput>BZ2_bzReadClose</computeroutput>.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>This mechanism makes it easy to decompress multiple
|
|
<computeroutput>bzip2</computeroutput> streams placed end-to-end.
|
|
As the end of one stream, when
|
|
<computeroutput>BZ2_bzRead</computeroutput> returns
|
|
<computeroutput>BZ_STREAM_END</computeroutput>, call
|
|
<computeroutput>BZ2_bzReadGetUnused</computeroutput> to collect
|
|
the unused data (copy it into your own buffer somewhere). That
|
|
data forms the start of the next compressed stream. To start
|
|
uncompressing that next stream, call
|
|
<computeroutput>BZ2_bzReadOpen</computeroutput> again, feeding in
|
|
the unused data via the <computeroutput>unused</computeroutput> /
|
|
<computeroutput>nUnused</computeroutput> parameters. Keep doing
|
|
this until <computeroutput>BZ_STREAM_END</computeroutput> return
|
|
coincides with the physical end of file
|
|
(<computeroutput>feof(f)</computeroutput>). In this situation
|
|
<computeroutput>BZ2_bzReadGetUnused</computeroutput> will of
|
|
course return no data.</para>
|
|
|
|
<para>This should give some feel for how the high-level interface
|
|
can be used. If you require extra flexibility, you'll have to
|
|
bite the bullet and get to grips with the low-level
|
|
interface.</para>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="std-rdwr" xreflabel="Standard file-reading/writing code">
|
|
<title>Standard file-reading/writing code</title>
|
|
|
|
<para>Here's how you'd write data to a compressed file:</para>
|
|
|
|
<programlisting>
|
|
FILE* f;
|
|
BZFILE* b;
|
|
int nBuf;
|
|
char buf[ /* whatever size you like */ ];
|
|
int bzerror;
|
|
int nWritten;
|
|
|
|
f = fopen ( "myfile.bz2", "w" );
|
|
if ( !f ) {
|
|
/* handle error */
|
|
}
|
|
b = BZ2_bzWriteOpen( &bzerror, f, 9 );
|
|
if (bzerror != BZ_OK) {
|
|
BZ2_bzWriteClose ( b );
|
|
/* handle error */
|
|
}
|
|
|
|
while ( /* condition */ ) {
|
|
/* get data to write into buf, and set nBuf appropriately */
|
|
nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf );
|
|
if (bzerror == BZ_IO_ERROR) {
|
|
BZ2_bzWriteClose ( &bzerror, b );
|
|
/* handle error */
|
|
}
|
|
}
|
|
|
|
BZ2_bzWriteClose( &bzerror, b );
|
|
if (bzerror == BZ_IO_ERROR) {
|
|
/* handle error */
|
|
}
|
|
</programlisting>
|
|
|
|
<para>And to read from a compressed file:</para>
|
|
|
|
<programlisting>
|
|
FILE* f;
|
|
BZFILE* b;
|
|
int nBuf;
|
|
char buf[ /* whatever size you like */ ];
|
|
int bzerror;
|
|
int nWritten;
|
|
|
|
f = fopen ( "myfile.bz2", "r" );
|
|
if ( !f ) {
|
|
/* handle error */
|
|
}
|
|
b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 );
|
|
if ( bzerror != BZ_OK ) {
|
|
BZ2_bzReadClose ( &bzerror, b );
|
|
/* handle error */
|
|
}
|
|
|
|
bzerror = BZ_OK;
|
|
while ( bzerror == BZ_OK && /* arbitrary other conditions */) {
|
|
nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ );
|
|
if ( bzerror == BZ_OK ) {
|
|
/* do something with buf[0 .. nBuf-1] */
|
|
}
|
|
}
|
|
if ( bzerror != BZ_STREAM_END ) {
|
|
BZ2_bzReadClose ( &bzerror, b );
|
|
/* handle error */
|
|
} else {
|
|
BZ2_bzReadClose ( &bzerror, b );
|
|
}
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="util-fns" xreflabel="Utility functions">
|
|
<title>Utility functions</title>
|
|
|
|
|
|
<sect2 id="bzbufftobuffcompress" xreflabel="BZ2_bzBuffToBuffCompress">
|
|
<title><computeroutput>BZ2_bzBuffToBuffCompress</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzBuffToBuffCompress( char* dest,
|
|
unsigned int* destLen,
|
|
char* source,
|
|
unsigned int sourceLen,
|
|
int blockSize100k,
|
|
int verbosity,
|
|
int workFactor );
|
|
</programlisting>
|
|
|
|
<para>Attempts to compress the data in <computeroutput>source[0
|
|
.. sourceLen-1]</computeroutput> into the destination buffer,
|
|
<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the
|
|
destination buffer is big enough,
|
|
<computeroutput>*destLen</computeroutput> is set to the size of
|
|
the compressed data, and <computeroutput>BZ_OK</computeroutput>
|
|
is returned. If the compressed data won't fit,
|
|
<computeroutput>*destLen</computeroutput> is unchanged, and
|
|
<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
|
|
returned.</para>
|
|
|
|
<para>Compression in this manner is a one-shot event, done with a
|
|
single call to this function. The resulting compressed data is a
|
|
complete <computeroutput>bzip2</computeroutput> format data
|
|
stream. There is no mechanism for making additional calls to
|
|
provide extra input data. If you want that kind of mechanism,
|
|
use the low-level interface.</para>
|
|
|
|
<para>For the meaning of parameters
|
|
<computeroutput>blockSize100k</computeroutput>,
|
|
<computeroutput>verbosity</computeroutput> and
|
|
<computeroutput>workFactor</computeroutput>, see
|
|
<computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
|
|
|
|
<para>To guarantee that the compressed data will fit in its
|
|
buffer, allocate an output buffer of size 1% larger than the
|
|
uncompressed data, plus six hundred extra bytes.</para>
|
|
|
|
<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
|
|
will not write data at or beyond
|
|
<computeroutput>dest[*destLen]</computeroutput>, even in case of
|
|
buffer overflow.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if dest is NULL or destLen is NULL
|
|
or blockSize100k < 1 or blockSize100k > 9
|
|
or verbosity < 0 or verbosity > 4
|
|
or workFactor < 0 or workFactor > 250
|
|
BZ_MEM_ERROR
|
|
if insufficient memory is available
|
|
BZ_OUTBUFF_FULL
|
|
if the size of the compressed data exceeds *destLen
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="bzbufftobuffdecompress" xreflabel="BZ2_bzBuffToBuffDecompress">
|
|
<title><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></title>
|
|
|
|
<programlisting>
|
|
int BZ2_bzBuffToBuffDecompress( char* dest,
|
|
unsigned int* destLen,
|
|
char* source,
|
|
unsigned int sourceLen,
|
|
int small,
|
|
int verbosity );
|
|
</programlisting>
|
|
|
|
<para>Attempts to decompress the data in <computeroutput>source[0
|
|
.. sourceLen-1]</computeroutput> into the destination buffer,
|
|
<computeroutput>dest[0 .. *destLen-1]</computeroutput>. If the
|
|
destination buffer is big enough,
|
|
<computeroutput>*destLen</computeroutput> is set to the size of
|
|
the uncompressed data, and <computeroutput>BZ_OK</computeroutput>
|
|
is returned. If the compressed data won't fit,
|
|
<computeroutput>*destLen</computeroutput> is unchanged, and
|
|
<computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
|
|
returned.</para>
|
|
|
|
<para><computeroutput>source</computeroutput> is assumed to hold
|
|
a complete <computeroutput>bzip2</computeroutput> format data
|
|
stream.
|
|
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> tries
|
|
to decompress the entirety of the stream into the output
|
|
buffer.</para>
|
|
|
|
<para>For the meaning of parameters
|
|
<computeroutput>small</computeroutput> and
|
|
<computeroutput>verbosity</computeroutput>, see
|
|
<computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
|
|
|
|
<para>Because the compression ratio of the compressed data cannot
|
|
be known in advance, there is no easy way to guarantee that the
|
|
output buffer will be big enough. You may of course make
|
|
arrangements in your code to record the size of the uncompressed
|
|
data, but such a mechanism is beyond the scope of this
|
|
library.</para>
|
|
|
|
<para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
|
|
will not write data at or beyond
|
|
<computeroutput>dest[*destLen]</computeroutput>, even in case of
|
|
buffer overflow.</para>
|
|
|
|
<para>Possible return values:</para>
|
|
|
|
<programlisting>
|
|
BZ_CONFIG_ERROR
|
|
if the library has been mis-compiled
|
|
BZ_PARAM_ERROR
|
|
if dest is NULL or destLen is NULL
|
|
or small != 0 && small != 1
|
|
or verbosity < 0 or verbosity > 4
|
|
BZ_MEM_ERROR
|
|
if insufficient memory is available
|
|
BZ_OUTBUFF_FULL
|
|
if the size of the compressed data exceeds *destLen
|
|
BZ_DATA_ERROR
|
|
if a data integrity error was detected in the compressed data
|
|
BZ_DATA_ERROR_MAGIC
|
|
if the compressed data doesn't begin with the right magic bytes
|
|
BZ_UNEXPECTED_EOF
|
|
if the compressed data ends unexpectedly
|
|
BZ_OK
|
|
otherwise
|
|
</programlisting>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="zlib-compat" xreflabel="zlib compatibility functions">
|
|
<title><computeroutput>zlib</computeroutput> compatibility functions</title>
|
|
|
|
<para>Yoshioka Tsuneo has contributed some functions to give
|
|
better <computeroutput>zlib</computeroutput> compatibility.
|
|
These functions are <computeroutput>BZ2_bzopen</computeroutput>,
|
|
<computeroutput>BZ2_bzread</computeroutput>,
|
|
<computeroutput>BZ2_bzwrite</computeroutput>,
|
|
<computeroutput>BZ2_bzflush</computeroutput>,
|
|
<computeroutput>BZ2_bzclose</computeroutput>,
|
|
<computeroutput>BZ2_bzerror</computeroutput> and
|
|
<computeroutput>BZ2_bzlibVersion</computeroutput>. These
|
|
functions are not (yet) officially part of the library. If they
|
|
break, you get to keep all the pieces. Nevertheless, I think
|
|
they work ok.</para>
|
|
|
|
<programlisting>
|
|
typedef void BZFILE;
|
|
|
|
const char * BZ2_bzlibVersion ( void );
|
|
</programlisting>
|
|
|
|
<para>Returns a string indicating the library version.</para>
|
|
|
|
<programlisting>
|
|
BZFILE * BZ2_bzopen ( const char *path, const char *mode );
|
|
BZFILE * BZ2_bzdopen ( int fd, const char *mode );
|
|
</programlisting>
|
|
|
|
<para>Opens a <computeroutput>.bz2</computeroutput> file for
|
|
reading or writing, using either its name or a pre-existing file
|
|
descriptor. Analogous to <computeroutput>fopen</computeroutput>
|
|
and <computeroutput>fdopen</computeroutput>.</para>
|
|
|
|
<programlisting>
|
|
int BZ2_bzread ( BZFILE* b, void* buf, int len );
|
|
int BZ2_bzwrite ( BZFILE* b, void* buf, int len );
|
|
</programlisting>
|
|
|
|
<para>Reads/writes data from/to a previously opened
|
|
<computeroutput>BZFILE</computeroutput>. Analogous to
|
|
<computeroutput>fread</computeroutput> and
|
|
<computeroutput>fwrite</computeroutput>.</para>
|
|
|
|
<programlisting>
|
|
int BZ2_bzflush ( BZFILE* b );
|
|
void BZ2_bzclose ( BZFILE* b );
|
|
</programlisting>
|
|
|
|
<para>Flushes/closes a <computeroutput>BZFILE</computeroutput>.
|
|
<computeroutput>BZ2_bzflush</computeroutput> doesn't actually do
|
|
anything. Analogous to <computeroutput>fflush</computeroutput>
|
|
and <computeroutput>fclose</computeroutput>.</para>
|
|
|
|
<programlisting>
|
|
const char * BZ2_bzerror ( BZFILE *b, int *errnum )
|
|
</programlisting>
|
|
|
|
<para>Returns a string describing the more recent error status of
|
|
<computeroutput>b</computeroutput>, and also sets
|
|
<computeroutput>*errnum</computeroutput> to its numerical
|
|
value.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="stdio-free"
|
|
xreflabel="Using the library in a stdio-free environment">
|
|
<title>Using the library in a <computeroutput>stdio</computeroutput>-free environment</title>
|
|
|
|
|
|
<sect2 id="stdio-bye" xreflabel="Getting rid of stdio">
|
|
<title>Getting rid of <computeroutput>stdio</computeroutput></title>
|
|
|
|
<para>In a deeply embedded application, you might want to use
|
|
just the memory-to-memory functions. You can do this
|
|
conveniently by compiling the library with preprocessor symbol
|
|
<computeroutput>BZ_NO_STDIO</computeroutput> defined. Doing this
|
|
gives you a library containing only the following eight
|
|
functions:</para>
|
|
|
|
<para><computeroutput>BZ2_bzCompressInit</computeroutput>,
|
|
<computeroutput>BZ2_bzCompress</computeroutput>,
|
|
<computeroutput>BZ2_bzCompressEnd</computeroutput>
|
|
<computeroutput>BZ2_bzDecompressInit</computeroutput>,
|
|
<computeroutput>BZ2_bzDecompress</computeroutput>,
|
|
<computeroutput>BZ2_bzDecompressEnd</computeroutput>
|
|
<computeroutput>BZ2_bzBuffToBuffCompress</computeroutput>,
|
|
<computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></para>
|
|
|
|
<para>When compiled like this, all functions will ignore
|
|
<computeroutput>verbosity</computeroutput> settings.</para>
|
|
|
|
</sect2>
|
|
|
|
|
|
<sect2 id="critical-error" xreflabel="Critical error handling">
|
|
<title>Critical error handling</title>
|
|
|
|
<para><computeroutput>libbzip2</computeroutput> contains a number
|
|
of internal assertion checks which should, needless to say, never
|
|
be activated. Nevertheless, if an assertion should fail,
|
|
behaviour depends on whether or not the library was compiled with
|
|
<computeroutput>BZ_NO_STDIO</computeroutput> set.</para>
|
|
|
|
<para>For a normal compile, an assertion failure yields the
|
|
message:</para>
|
|
|
|
<blockquote>
|
|
<para>bzip2/libbzip2: internal error number N.</para>
|
|
<para>This is a bug in bzip2/libbzip2, &bz-version; of &bz-date;.
|
|
Please report it to me at: &bz-email;. If this happened
|
|
when you were using some program which uses libbzip2 as a
|
|
component, you should also report this bug to the author(s)
|
|
of that program. Please make an effort to report this bug;
|
|
timely and accurate bug reports eventually lead to higher
|
|
quality software. Thanks. Julian Seward, &bz-date;.
|
|
</para></blockquote>
|
|
|
|
<para>where <computeroutput>N</computeroutput> is some error code
|
|
number. If <computeroutput>N == 1007</computeroutput>, it also
|
|
prints some extra text advising the reader that unreliable memory
|
|
is often associated with internal error 1007. (This is a
|
|
frequently-observed-phenomenon with versions 1.0.0/1.0.1).</para>
|
|
|
|
<para><computeroutput>exit(3)</computeroutput> is then
|
|
called.</para>
|
|
|
|
<para>For a <computeroutput>stdio</computeroutput>-free library,
|
|
assertion failures result in a call to a function declared
|
|
as:</para>
|
|
|
|
<programlisting>
|
|
extern void bz_internal_error ( int errcode );
|
|
</programlisting>
|
|
|
|
<para>The relevant code is passed as a parameter. You should
|
|
supply such a function.</para>
|
|
|
|
<para>In either case, once an assertion failure has occurred, any
|
|
<computeroutput>bz_stream</computeroutput> records involved can
|
|
be regarded as invalid. You should not attempt to resume normal
|
|
operation with them.</para>
|
|
|
|
<para>You may, of course, change critical error handling to suit
|
|
your needs. As I said above, critical errors indicate bugs in
|
|
the library and should not occur. All "normal" error situations
|
|
are indicated via error return codes from functions, and can be
|
|
recovered from.</para>
|
|
|
|
</sect2>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="win-dll" xreflabel="Making a Windows DLL">
|
|
<title>Making a Windows DLL</title>
|
|
|
|
<para>Everything related to Windows has been contributed by
|
|
Yoshioka Tsuneo
|
|
(<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>), so
|
|
you should send your queries to him (but perhaps Cc: me,
|
|
<computeroutput>&bz-email;</computeroutput>).</para>
|
|
|
|
<para>My vague understanding of what to do is: using Visual C++
|
|
5.0, open the project file
|
|
<computeroutput>libbz2.dsp</computeroutput>, and build. That's
|
|
all.</para>
|
|
|
|
<para>If you can't open the project file for some reason, make a
|
|
new one, naming these files:
|
|
<computeroutput>blocksort.c</computeroutput>,
|
|
<computeroutput>bzlib.c</computeroutput>,
|
|
<computeroutput>compress.c</computeroutput>,
|
|
<computeroutput>crctable.c</computeroutput>,
|
|
<computeroutput>decompress.c</computeroutput>,
|
|
<computeroutput>huffman.c</computeroutput>,
|
|
<computeroutput>randtable.c</computeroutput> and
|
|
<computeroutput>libbz2.def</computeroutput>. You will also need
|
|
to name the header files <computeroutput>bzlib.h</computeroutput>
|
|
and <computeroutput>bzlib_private.h</computeroutput>.</para>
|
|
|
|
<para>If you don't use VC++, you may need to define the
|
|
proprocessor symbol
|
|
<computeroutput>_WIN32</computeroutput>.</para>
|
|
|
|
<para>Finally, <computeroutput>dlltest.c</computeroutput> is a
|
|
sample program using the DLL. It has a project file,
|
|
<computeroutput>dlltest.dsp</computeroutput>.</para>
|
|
|
|
<para>If you just want a makefile for Visual C, have a look at
|
|
<computeroutput>makefile.msc</computeroutput>.</para>
|
|
|
|
<para>Be aware that if you compile
|
|
<computeroutput>bzip2</computeroutput> itself on Win32, you must
|
|
set <computeroutput>BZ_UNIX</computeroutput> to 0 and
|
|
<computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the file
|
|
<computeroutput>bzip2.c</computeroutput>, before compiling.
|
|
Otherwise the resulting binary won't work correctly.</para>
|
|
|
|
<para>I haven't tried any of this stuff myself, but it all looks
|
|
plausible.</para>
|
|
|
|
</sect1>
|
|
|
|
</chapter>
|
|
|
|
|
|
|
|
<chapter id="misc" xreflabel="Miscellanea">
|
|
<title>Miscellanea</title>
|
|
|
|
<para>These are just some random thoughts of mine. Your mileage
|
|
may vary.</para>
|
|
|
|
|
|
<sect1 id="limits" xreflabel="Limitations of the compressed file format">
|
|
<title>Limitations of the compressed file format</title>
|
|
|
|
<para><computeroutput>bzip2-1.0.X</computeroutput>,
|
|
<computeroutput>0.9.5</computeroutput> and
|
|
<computeroutput>0.9.0</computeroutput> use exactly the same file
|
|
format as the original version,
|
|
<computeroutput>bzip2-0.1</computeroutput>. This decision was
|
|
made in the interests of stability. Creating yet another
|
|
incompatible compressed file format would create further
|
|
confusion and disruption for users.</para>
|
|
|
|
<para>Nevertheless, this is not a painless decision. Development
|
|
work since the release of
|
|
<computeroutput>bzip2-0.1</computeroutput> in August 1997 has
|
|
shown complexities in the file format which slow down
|
|
decompression and, in retrospect, are unnecessary. These
|
|
are:</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>The run-length encoder, which is the first of the
|
|
compression transformations, is entirely irrelevant. The
|
|
original purpose was to protect the sorting algorithm from the
|
|
very worst case input: a string of repeated symbols. But
|
|
algorithm steps Q6a and Q6b in the original Burrows-Wheeler
|
|
technical report (SRC-124) show how repeats can be handled
|
|
without difficulty in block sorting.</para></listitem>
|
|
|
|
<listitem><para>The randomisation mechanism doesn't really need to be
|
|
there. Udi Manber and Gene Myers published a suffix array
|
|
construction algorithm a few years back, which can be employed
|
|
to sort any block, no matter how repetitive, in O(N log N)
|
|
time. Subsequent work by Kunihiko Sadakane has produced a
|
|
derivative O(N (log N)^2) algorithm which usually outperforms
|
|
the Manber-Myers algorithm.</para>
|
|
|
|
<para>I could have changed to Sadakane's algorithm, but I find
|
|
it to be slower than <computeroutput>bzip2</computeroutput>'s
|
|
existing algorithm for most inputs, and the randomisation
|
|
mechanism protects adequately against bad cases. I didn't
|
|
think it was a good tradeoff to make. Partly this is due to
|
|
the fact that I was not flooded with email complaints about
|
|
<computeroutput>bzip2-0.1</computeroutput>'s performance on
|
|
repetitive data, so perhaps it isn't a problem for real
|
|
inputs.</para>
|
|
|
|
<para>Probably the best long-term solution, and the one I have
|
|
incorporated into 0.9.5 and above, is to use the existing
|
|
sorting algorithm initially, and fall back to a O(N (log N)^2)
|
|
algorithm if the standard algorithm gets into
|
|
difficulties.</para></listitem>
|
|
|
|
<listitem><para>The compressed file format was never designed to be
|
|
handled by a library, and I have had to jump though some hoops
|
|
to produce an efficient implementation of decompression. It's
|
|
a bit hairy. Try passing
|
|
<computeroutput>decompress.c</computeroutput> through the C
|
|
preprocessor and you'll see what I mean. Much of this
|
|
complexity could have been avoided if the compressed size of
|
|
each block of data was recorded in the data stream.</para></listitem>
|
|
|
|
<listitem><para>An Adler-32 checksum, rather than a CRC32 checksum,
|
|
would be faster to compute.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>It would be fair to say that the
|
|
<computeroutput>bzip2</computeroutput> format was frozen before I
|
|
properly and fully understood the performance consequences of
|
|
doing so.</para>
|
|
|
|
<para>Improvements which I was able to incorporate into 0.9.0,
|
|
despite using the same file format, are:</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>Single array implementation of the inverse BWT. This
|
|
significantly speeds up decompression, presumably because it
|
|
reduces the number of cache misses.</para></listitem>
|
|
|
|
<listitem><para>Faster inverse MTF transform for large MTF values.
|
|
The new implementation is based on the notion of sliding blocks
|
|
of values.</para></listitem>
|
|
|
|
<listitem><para><computeroutput>bzip2-0.9.0</computeroutput> now reads
|
|
and writes files with <computeroutput>fread</computeroutput>
|
|
and <computeroutput>fwrite</computeroutput>; version 0.1 used
|
|
<computeroutput>putc</computeroutput> and
|
|
<computeroutput>getc</computeroutput>. Duh! Well, you live
|
|
and learn.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>Further ahead, it would be nice to be able to do random
|
|
access into files. This will require some careful design of
|
|
compressed file formats.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="port-issues" xreflabel="Portability issues">
|
|
<title>Portability issues</title>
|
|
|
|
<para>After some consideration, I have decided not to use GNU
|
|
<computeroutput>autoconf</computeroutput> to configure 0.9.5 or
|
|
1.0.</para>
|
|
|
|
<para><computeroutput>autoconf</computeroutput>, admirable and
|
|
wonderful though it is, mainly assists with portability problems
|
|
between Unix-like platforms. But
|
|
<computeroutput>bzip2</computeroutput> doesn't have much in the
|
|
way of portability problems on Unix; most of the difficulties
|
|
appear when porting to the Mac, or to Microsoft's operating
|
|
systems. <computeroutput>autoconf</computeroutput> doesn't help
|
|
in those cases, and brings in a whole load of new
|
|
complexity.</para>
|
|
|
|
<para>Most people should be able to compile the library and
|
|
program under Unix straight out-of-the-box, so to speak,
|
|
especially if you have a version of GNU C available.</para>
|
|
|
|
<para>There are a couple of
|
|
<computeroutput>__inline__</computeroutput> directives in the
|
|
code. GNU C (<computeroutput>gcc</computeroutput>) should be
|
|
able to handle them. If you're not using GNU C, your C compiler
|
|
shouldn't see them at all. If your compiler does, for some
|
|
reason, see them and doesn't like them, just
|
|
<computeroutput>#define</computeroutput>
|
|
<computeroutput>__inline__</computeroutput> to be
|
|
<computeroutput>/* */</computeroutput>. One easy way to do this
|
|
is to compile with the flag
|
|
<computeroutput>-D__inline__=</computeroutput>, which should be
|
|
understood by most Unix compilers.</para>
|
|
|
|
<para>If you still have difficulties, try compiling with the
|
|
macro <computeroutput>BZ_STRICT_ANSI</computeroutput> defined.
|
|
This should enable you to build the library in a strictly ANSI
|
|
compliant environment. Building the program itself like this is
|
|
dangerous and not supported, since you remove
|
|
<computeroutput>bzip2</computeroutput>'s checks against
|
|
compressing directories, symbolic links, devices, and other
|
|
not-really-a-file entities. This could cause filesystem
|
|
corruption!</para>
|
|
|
|
<para>One other thing: if you create a
|
|
<computeroutput>bzip2</computeroutput> binary for public distribution,
|
|
please consider linking it statically (<computeroutput>gcc
|
|
-static</computeroutput>). This avoids all sorts of library-version
|
|
issues that others may encounter later on.</para>
|
|
|
|
<para>If you build <computeroutput>bzip2</computeroutput> on
|
|
Win32, you must set <computeroutput>BZ_UNIX</computeroutput> to 0
|
|
and <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the
|
|
file <computeroutput>bzip2.c</computeroutput>, before compiling.
|
|
Otherwise the resulting binary won't work correctly.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="bugs" xreflabel="Reporting bugs">
|
|
<title>Reporting bugs</title>
|
|
|
|
<para>I tried pretty hard to make sure
|
|
<computeroutput>bzip2</computeroutput> is bug free, both by
|
|
design and by testing. Hopefully you'll never need to read this
|
|
section for real.</para>
|
|
|
|
<para>Nevertheless, if <computeroutput>bzip2</computeroutput> dies
|
|
with a segmentation fault, a bus error or an internal assertion
|
|
failure, it will ask you to email me a bug report. Experience from
|
|
years of feedback of bzip2 users indicates that almost all these
|
|
problems can be traced to either compiler bugs or hardware
|
|
problems.</para>
|
|
|
|
<itemizedlist mark='bullet'>
|
|
|
|
<listitem><para>Recompile the program with no optimisation, and
|
|
see if it works. And/or try a different compiler. I heard all
|
|
sorts of stories about various flavours of GNU C (and other
|
|
compilers) generating bad code for
|
|
<computeroutput>bzip2</computeroutput>, and I've run across two
|
|
such examples myself.</para>
|
|
|
|
<para>2.7.X versions of GNU C are known to generate bad code
|
|
from time to time, at high optimisation levels. If you get
|
|
problems, try using the flags
|
|
<computeroutput>-O2</computeroutput>
|
|
<computeroutput>-fomit-frame-pointer</computeroutput>
|
|
<computeroutput>-fno-strength-reduce</computeroutput>. You
|
|
should specifically <emphasis>not</emphasis> use
|
|
<computeroutput>-funroll-loops</computeroutput>.</para>
|
|
|
|
<para>You may notice that the Makefile runs six tests as part
|
|
of the build process. If the program passes all of these, it's
|
|
a pretty good (but not 100%) indication that the compiler has
|
|
done its job correctly.</para></listitem>
|
|
|
|
<listitem><para>If <computeroutput>bzip2</computeroutput>
|
|
crashes randomly, and the crashes are not repeatable, you may
|
|
have a flaky memory subsystem.
|
|
<computeroutput>bzip2</computeroutput> really hammers your
|
|
memory hierarchy, and if it's a bit marginal, you may get these
|
|
problems. Ditto if your disk or I/O subsystem is slowly
|
|
failing. Yup, this really does happen.</para>
|
|
|
|
<para>Try using a different machine of the same type, and see
|
|
if you can repeat the problem.</para></listitem>
|
|
|
|
<listitem><para>This isn't really a bug, but ... If
|
|
<computeroutput>bzip2</computeroutput> tells you your file is
|
|
corrupted on decompression, and you obtained the file via FTP,
|
|
there is a possibility that you forgot to tell FTP to do a
|
|
binary mode transfer. That absolutely will cause the file to
|
|
be non-decompressible. You'll have to transfer it
|
|
again.</para></listitem>
|
|
|
|
</itemizedlist>
|
|
|
|
<para>If you've incorporated
|
|
<computeroutput>libbzip2</computeroutput> into your own program
|
|
and are getting problems, please, please, please, check that the
|
|
parameters you are passing in calls to the library, are correct,
|
|
and in accordance with what the documentation says is allowable.
|
|
I have tried to make the library robust against such problems,
|
|
but I'm sure I haven't succeeded.</para>
|
|
|
|
<para>Finally, if the above comments don't help, you'll have to
|
|
send me a bug report. Now, it's just amazing how many people
|
|
will send me a bug report saying something like:</para>
|
|
|
|
<programlisting>
|
|
bzip2 crashed with segmentation fault on my machine
|
|
</programlisting>
|
|
|
|
<para>and absolutely nothing else. Needless to say, a such a
|
|
report is <emphasis>totally, utterly, completely and
|
|
comprehensively 100% useless; a waste of your time, my time, and
|
|
net bandwidth</emphasis>. With no details at all, there's no way
|
|
I can possibly begin to figure out what the problem is.</para>
|
|
|
|
<para>The rules of the game are: facts, facts, facts. Don't omit
|
|
them because "oh, they won't be relevant". At the bare
|
|
minimum:</para>
|
|
|
|
<programlisting>
|
|
Machine type. Operating system version.
|
|
Exact version of bzip2 (do bzip2 -V).
|
|
Exact version of the compiler used.
|
|
Flags passed to the compiler.
|
|
</programlisting>
|
|
|
|
<para>However, the most important single thing that will help me
|
|
is the file that you were trying to compress or decompress at the
|
|
time the problem happened. Without that, my ability to do
|
|
anything more than speculate about the cause, is limited.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
<sect1 id="package" xreflabel="Did you get the right package?">
|
|
<title>Did you get the right package?</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> is a resource hog.
|
|
It soaks up large amounts of CPU cycles and memory. Also, it
|
|
gives very large latencies. In the worst case, you can feed many
|
|
megabytes of uncompressed data into the library before getting
|
|
any compressed output, so this probably rules out applications
|
|
requiring interactive behaviour.</para>
|
|
|
|
<para>These aren't faults of my implementation, I hope, but more
|
|
an intrinsic property of the Burrows-Wheeler transform
|
|
(unfortunately). Maybe this isn't what you want.</para>
|
|
|
|
<para>If you want a compressor and/or library which is faster,
|
|
uses less memory but gets pretty good compression, and has
|
|
minimal latency, consider Jean-loup Gailly's and Mark Adler's
|
|
work, <computeroutput>zlib-1.2.1</computeroutput> and
|
|
<computeroutput>gzip-1.2.4</computeroutput>. Look for them at
|
|
<ulink url="http://www.zlib.org">http://www.zlib.org</ulink> and
|
|
<ulink url="http://www.gzip.org">http://www.gzip.org</ulink>
|
|
respectively.</para>
|
|
|
|
<para>For something faster and lighter still, you might try Markus F
|
|
X J Oberhumer's <computeroutput>LZO</computeroutput> real-time
|
|
compression/decompression library, at
|
|
<ulink url="http://www.oberhumer.com/opensource">http://www.oberhumer.com/opensource</ulink>.</para>
|
|
|
|
</sect1>
|
|
|
|
|
|
|
|
<sect1 id="reading" xreflabel="Further Reading">
|
|
<title>Further Reading</title>
|
|
|
|
<para><computeroutput>bzip2</computeroutput> is not research
|
|
work, in the sense that it doesn't present any new ideas.
|
|
Rather, it's an engineering exercise based on existing
|
|
ideas.</para>
|
|
|
|
<para>Four documents describe essentially all the ideas behind
|
|
<computeroutput>bzip2</computeroutput>:</para>
|
|
|
|
<literallayout>Michael Burrows and D. J. Wheeler:
|
|
"A block-sorting lossless data compression algorithm"
|
|
10th May 1994.
|
|
Digital SRC Research Report 124.
|
|
ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz
|
|
If you have trouble finding it, try searching at the
|
|
New Zealand Digital Library, http://www.nzdl.org.
|
|
|
|
Daniel S. Hirschberg and Debra A. LeLewer
|
|
"Efficient Decoding of Prefix Codes"
|
|
Communications of the ACM, April 1990, Vol 33, Number 4.
|
|
You might be able to get an electronic copy of this
|
|
from the ACM Digital Library.
|
|
|
|
David J. Wheeler
|
|
Program bred3.c and accompanying document bred3.ps.
|
|
This contains the idea behind the multi-table Huffman coding scheme.
|
|
ftp://ftp.cl.cam.ac.uk/users/djw3/
|
|
|
|
Jon L. Bentley and Robert Sedgewick
|
|
"Fast Algorithms for Sorting and Searching Strings"
|
|
Available from Sedgewick's web page,
|
|
www.cs.princeton.edu/~rs
|
|
</literallayout>
|
|
|
|
<para>The following paper gives valuable additional insights into
|
|
the algorithm, but is not immediately the basis of any code used
|
|
in bzip2.</para>
|
|
|
|
<literallayout>Peter Fenwick:
|
|
Block Sorting Text Compression
|
|
Proceedings of the 19th Australasian Computer Science Conference,
|
|
Melbourne, Australia. Jan 31 - Feb 2, 1996.
|
|
ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</literallayout>
|
|
|
|
<para>Kunihiko Sadakane's sorting algorithm, mentioned above, is
|
|
available from:</para>
|
|
|
|
<literallayout>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
|
|
</literallayout>
|
|
|
|
<para>The Manber-Myers suffix array construction algorithm is
|
|
described in a paper available from:</para>
|
|
|
|
<literallayout>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
|
|
</literallayout>
|
|
|
|
<para>Finally, the following papers document some
|
|
investigations I made into the performance of sorting
|
|
and decompression algorithms:</para>
|
|
|
|
<literallayout>Julian Seward
|
|
On the Performance of BWT Sorting Algorithms
|
|
Proceedings of the IEEE Data Compression Conference 2000
|
|
Snowbird, Utah. 28-30 March 2000.
|
|
|
|
Julian Seward
|
|
Space-time Tradeoffs in the Inverse B-W Transform
|
|
Proceedings of the IEEE Data Compression Conference 2001
|
|
Snowbird, Utah. 27-29 March 2001.
|
|
</literallayout>
|
|
|
|
</sect1>
|
|
|
|
</chapter>
|
|
|
|
</book>
|