Some-real world files have JBIG2 files with many entries in a
symbol dictionary. For example,
https://www.sciencemadness.org/library/books/Mellor_ACTITC_11.pdf
has 173,132 symbols in the global jbig2 stream's symbol dictionary.
This adds a test that has many entries in the symbol dictionary.
Since putting hundreds of thousands of symbols in the json file is a bit
wordy, this adds a new mode "all_symbols_of_size" to
"symbol_dictionary". This mode takes width and height, and then adds all
possible bitmaps with these dimensions to the symbol dictionary. For
width x height, it will add 2 ** (width * height) symbols.
It also adds a new "strips_for_all_symbols" mode to the "text_region"
segments. This gets width and height and a target bitmap. It then tiles
that target bitmap and picks the symbol dict entry for the pattern at
each tile.
It then adds an arithmetically coded and a huffman coded file using this
construct to add a all_symbols_of_size 9x2 symbol dict. In other words,
it has 2 ** 18 == 256K entries.
(All symbols have height 2, and in huffman mode, all symbols with the
same height get concatenated into a single bitmap, so this internally
creates a 9 * 2 ** 18 ~= 2.3 M wide bitmap. This is a bit artificial;
real-life files will have more than one height class. But this doesn't
seem to be a problem for other viewers, so maybe it's fine.)
That's the gist of it. To make the text region symbol IDs a bit more
interesting, we write bitmap-template0-all-ones.bmp instead of
bitmap.bmp in the text region segment and then xor bitmap-xor.bmp on top
of it to recover bitmap.bmp. This is because otherwise most tiles end up
with symbol id 0. bitmap-template0-all-ones.bmp is roughly noise, and
bitmap-xor.bmp is the same bitmap with bitmap.bmp xor'd on top.
In addition to testing symbol dicts with many symbols, it's also a bit
of a perf test as this is slow with the huffman implementation in the
spec (which we currently use, and decoding the huffman version of this
file needs over 0.5s to decode at the moment).
...as long as fewer than 32K entries are referenced in a single
text region segment.
Having more than 32K entries in a symbol dictionary happens in some
real-world PDFs where the "global" JBIG2 stream that's shared over
all pages has lots of letters. But the text region segment on every
single page just has a couple thousand letters (160 characters per line
at 100 lines and a couple thousand dust speckles is well under 32K
symbol references per page).
I'd like to add a test that's similar to such a file, so we need to
support writing text regions referring to symbol dictionaries with
more than 32K symbols.
A test jbig2 image I want to create will have a height class bitmap
that's 9 * 2 ** 18 = 2,359,296 pixels wide (and 2 high).
So bump up our limit on bitmap width and height from 64 ki to 4 Mi.
That way, we don't need all the specialized ones.
Pointless golfing, or exciting independent rediscovery of a subset
of monads? You decide!
No behavior change.
Generally, the design of jbig2-from-json is that it leaves a
roughly correct jbig2 file and then relies on parsing it to do
detailed validation of fields. That means just using
parse_u32_in_range(..., 0, 3, ...) would be good enough for most
uses in this file. But seeing that next to the error message was
confusing, so in this case we do slightly too detailed validation
in jbig2-from-json.cpp directly.
No behavior change.
Technically not needed as parse_bool + implicit conversion from
bool to int does the same thing, but relying on implicit conversion
from bool feels a bit yucky.
No behavior change.
All client code only calls set_i32(), but implement set_i32() in
terms of parse_i32() for symmetry with parse_bool() / set_bool().
set_i32() is templated to set Optional<i32>s.
No behavior change.
If the current text rendering mode isn't a solid-color fill
(or a solid-color fill with a non-scaling matrix), request
vector glyphs, and use regular path rendering.
Like in regular path rendering, for fill-and-stroke, we're supposed
to not draw the stroke on top of the fill. We don't get that right
yet, see the FIXME in Renderer::fill_and_stroke_path().
Removes some duplication from SimpleFont and Type0Font, and makes
it easier to make self-consistent decisions about this with
patterns and text rendering modes.
This will make it possible to implement text rendering mode support
in a single place.
Both SimpleFont and Type0Font will have to know to call the path
variant if the text rendering modes isn't `Fill`.
Needs a pretty sus const_cast since draw_string() receives
a Renderer const& but a non-const painter (and we need the
renderer's painter in Renderer::paint_text_glyphs(). We could
pass in the painter to paint_text_glyphs(), but it'd be the
same painter really, so that'd be silly too.)
Another option would be to give PDFFont a method to return a
Path and have Renderer decide if it wants to call this. That's
a bit janky with Type3Font though, which cannot return a vector
path, but it does support rotation already. But maybe we want
to reorganize things some more later.
No behavior change.
With this, we can draw rotated or skewed Type0 fonts with
subtype CIDFontType2. These use TrueType data, so this is
similar to TrueTypePainter::append_glyph_path().
We can now draw rotated or skewed fonts for all font types :^)
With this, we can draw rotated or skewed Type0 fonts with
subtype CIDFontType0 (using CIDFontType0C in the descriptor's
subtype). These use CFF data, so the implementation here
is very similar to the Type1 implementation in
0ac7f5228f from #26954.
It delegates to CIDFontType::append_glyph_path(), which no
CIDFontType subclass implements yet, so no behavior change yet.
Type0Font::draw_transformed_glyphs() is virtually identical to
the SimpleFont version, so we'll want to unify code here in the future.
This seems to do the right thing for horizontal text, but not for
vertical text. So the vertical text doesn't use the append_glyph_path()
codepath yet; it's for a future PR.
This does for Type0 what a subset of commits 2 and 3 of #26292
(1892b09b9f / 4bd3d4bf95) did for SimpleFont:
The core logic of Type0Font::draw_string() moves into a new
method Type0Font::for_each_glyph_position(), and the existing
code calls that from a new `Type0Font::draw_axis_aligned_glyphs()`.
for_each_glyph_position() no longer works in prescaled units,
since SimpleFont no longer does that after #26292 either.
(SimpleFont had to keep `font_scale` for fallback widths or
built-in fonts, but it rescaled units to limit that to a small
corner case, and that corner case does not exist in Type0 fonts,
so we get to simplify more.)
Type0Font::draw_transformed_glyph() is called but doesn't do anything
yet. That will be in the next commit. (Since this policy code is
duplicated in SimpleFont and Type0Font, we might want to revert
parts of 4bd3d4bf95 again in the future.)
No behavior change.
With this, we can draw rotated type1 text.
The `glyph_width` multiplication in SimpleFont::append_text_path
corresponds to the same in SimpleFont::draw_axis_aligned_glyphs.
Type1FontProgram::append_glyph_to() uses AppendRelativeToLastPoint::Yes
like ScaledFont::append_glyph_path_to().
Type1FontProgram::append_glyph_to() is like build_char() in the same
class, except it does no subpixel translation, and it doesn't move
the origin to the top left corner.
We now require required keys.
Also rename types and names to match the spec better.
(Eventually, we'll want to add "required_foo()" methods that check that
they actually exist instead of VERIFY()ing ne day.)
Makes sense now that it's a Variant<Color, Pattern>. Also,
Gfx::StrokeStyle is something else already (style for stroking paths),
and Renderer::stroke_style() returned that. So this makes things
less confusing.
No behavior change.
This moves tiled pattern painting to work like gradient
patterns work now:
- PatternColorSpace::style() is no longer called
- Pattern::style() is split into TilingPattern::create() and
TilingPattern::draw(). This is as close to a pure code move
for now as possible. (OffsetPaintStyle moves from Renderer
into TilingPattern::draw() for this. This is a slight
progression in that filing a path with a translation from
the CTM now works, see upper rect in patter-type1.pdf.)
Other than that, this is a pure reorganization that retains bugs such as
having no pattern rotation support, no support for /PainType 2, and so
on.
One way to look at this commit is that previously, ColorSpace returned a
ColorOrStyle (a Variant<Color, PaintStyle>). That's now no longer used
(a future commit will make ColorSpace return just a Color isntead), but
instead GraphicsState now has a Variant<Color, Pattern>. Pattern has an
explicit painting API instead of the implicit PaintStyle API, as that
works better for some of the gradients.
(Maybe we can even remove PatternColorSpace in the future by making
GraphicState's color spaces a Variant<ColorSpace, PatternTag>. That
way, areas in the code that don't support pattern color spaces will
have an easier time.)
Since this now uses get_resource(), it also makes files work that use
indirect object references for the pattern's resource dictionary
(e.g. 0000450.pdf p5).
Since we now paint tiling patterns by drawing paths to alpha and the
pattern to the path's bounding box, this approach is a bit slower (not
much, and we'll get some of that back when we make TilingPattern::draw()
no longer use a PaintStyle internally). But it'll allow us to handle
gradient and tiling patterns uniformly, and makes it easier to
eventually fill text and masks with patterns.
It's based on pattern-type1.pdf, but with type 2 patterns instead
of type 1 patterns. The shading object inside the pattern is
from group.pdf and slightly tweaked.
Stroking is implemented by converting the stroke to a path and
filling it, as usual.
Filling a path with a type 2 pattern is implemented by applying
the path as clip, and then drawing the shading inside the type 2
pattern.
state().stroke_style and state().paint_style are now a
`Variant<ColorOrStyle, NonnullRefPtr<Pattern>>` insteaad of just
a ColorOrStyle. Update all use sites to fail with an error if
it's a NonnullRefPtr<Pattern>.
In the future, we can hopefully move type 1 patterns to this design
as well, and then change ColorOrStyle to just Color. Then
ColorSpace will no longer have to know about patterns (or Renderer).
In the meantime, this is a bit awkward.
Future commits will fill in code to allow actually painting
objects with pattern fills or clips, starting with paths, and
eventually image masks.
(Text depends on some more text-to-path code that we also need
for implementing more text rendering modes.)
Less code, and we no longer assert on documents that use a shading but
don't have a /Shading entry in the resource dict, nor on documents that
load a graphics context and either don't have a /ExtGState entry in the
resources dict, or an entry for the name of the loaded graphics context.
The current design of PatternColorSpace returning a PaintStyle
doesn't neatly work for type 2 patterns: PaintStyle has to return
a Color for each coordinate, and shading types 4-7 do not have a
neat implicit representation.
Instead, type 2 shadings will handle their painting outside the
PDF::ColorSpace system, painting the shading to a bitmap and clipping
it to the path to be filled (or the stroke-to-path for stroking),
similar to how the `sh` operator is already implemented.
This is a pure code move at this point, the only change is
s/m_renderer/renderer/ in the new location.
Later, the API here will change a bit, and the internal
organization of Pattern.cpp will too.
No behavior change.
This file was based on Example 4.23 in the PDF 1.7 spec, but
a few changes were made that broke it:
- ZapfDingbats was changed to Helvevtica (a misspelling of Helvetica)
- But it still used an /Encoding array for ZapfDingbats
- The operator stream *was* updated to draw glyphs a, b, c, d
- And pattern had no /Resources dict even though it's required per spec
Make it more like the example from the spec:
- Change the font to ZapfDingbats, and draw \001, \002, \003, \004
in the operator stream, to match the /Encoding entry
- Give the pattern a /Resources entry. Like in the example, make it
an indirect object reference to object 16, and add an object 16
- Update file offsets with `mutool clean` as usual
With this, the file shows card suits instead of a, b, c, d, and it
now displays fine in Preview.app. And it'll keep displaying fine
if we decide to make LibPDF stricter and more spec-compliant by
requiring /Resources too, like Preview.app seems to do.
Previously, we would emit a "bitmap empty backing store" if a tiling's
/Matrix had a negative scale factor.
Happens e.g. for all four tiling patterns in 2025-Laurenz-Thesis.pdf
--page 34.
`/CS0 cs` in a form where the form has
`/Resources<</ColorSpace<</CS0 [/Pattern]>>>>` now works.
This happens e.g. in 2025-Laurenz-Thesis.pdf --page 34
(which has page number 29 written on it).
Real-life PDFs don't adhere to the spec, so we need to be lax about
this by default. But it's nice to optionally be more strict.
The main motivation is that we'll have to relax more things, and
this way they're easy to find.
For now, this adds just a handful checks. There will be more later.
This is a follow-up to #26905: If an unknown operator had arguments,
we used to not clear them, causing the next known operator to
receive them.
To test, call an unknown operator with a non-number arg,
followed by `re`.
We used to crash /Tf with an nonexistent font name,
on /CS / /cs with a nonexistent colorspace name,
on color operators with the wrong number of operators,
and on color operators with the wrong type of operators.
(We already handle /Do with nonexistent image or form name fine,
but slightly reorder lines there to match font and colorspace
handling)
Add a test for this, hand-written and fixed up by `mutool clean`.
Lets us successfully render page 2 of
BCM2837-ARM-Peripherals.-.Revised.-.V2-1.pdf
This is a separate commit from the previous one as it's technically
not strictly spec compliant.
Lets us no longer crash rendering page 2 of
BCM2837-ARM-Peripherals.-.Revised.-.V2-1.pdf.
Test file hand-written (based on text.pdf) and offsets fixed up
with `mutool clean` as usual.
This is a slight behavior change in some places. Some places did
(r.size().to_type<float>() * max(scale_x, scale_y)).to_type<int>();
After this commit, the semantics are now like `(...).to_rounded<int>()`,
that is we no longer sometimes omit a column or row in the fitted
direction.
scaled_to_fit_within() is like CSS's `object-fit: contain`: It returns
the largest rect that completely fits into the given rect, with an exact
fit in one dimension, and centered in the other dimension.
scaled_to_cover() is like `object-fit: cover`: It returns the smallest
rect that completely covers the given rect, with an exact fit in one
dimension, and centered in the other dimension.
Takes `Build/lagom/bin/image -o out.tiff ~/Downloads/balloon.jpf` from
531.4 ms ± 5.4 ms to 515.8 ms ± 4.7 ms ms under hyperfine on my system
(2.7% less). When run under `/usr/bin/time -l`, the changes are:
Before:
13202665108 instructions retired
2281080551 cycles elapsed
After:
12821908788 instructions retired
2213264986 cycles elapsed
Only affects lossy files.
Takes `Build/lagom/bin/image -o out.tiff ~/Downloads/balloon.jpf` from
542.3 ms ± 7.0 ms to 527.2 ms ± 4.7 ms under hyperfine on my system
(2.7% less). When run under `/usr/bin/time -l`, the changes are:
Before:
13956342230 instructions retired
2322276362 cycles elapsed
After:
13251294364 instructions retired
2270195534 cycles elapsed
The absolute win on the lossless file is again a bit smaller
(due to smaller image dimensions), and the relative win is tiny
there (due to it spending much more time in bitplane decoding and
less in IDWT).
Takes `Build/lagom/bin/image -o out.tiff ~/Downloads/balloon.jpf` from
586.0 ms ± 5.6 ms to 547.7 ms ± 5.1 ms under hyperfine on my system
(6.6% less). When run under `/usr/bin/time -l`, the changes are:
Before:
14557698316 instructions retired
2500654210 cycles elapsed
After:
13958498350 instructions retired
2318667968 cycles elapsed
The lossless files goes from 1.025 s to 1.000 s (likely smaller win
because it's half the dimensions).
(Note that the "from" hyperfine number is pretty different from the
previous commit, probably due to my system being under different load,
at different battery level, or what. But the "instructions retired"
are very comparable, so that's probably a better number to look at.)
Slight subtlety: Forming the pointer
`data() + (buffers.scanline_start - i0)` is usually technically invalid
in C++: scanline_start is usually 4 and i0 can be up to the current
rect's left or top, so it'll usually point to before the image.
I haven't seen this cause actual problems in practice, but make the
buffers max(left,top) longe, even if that leading data ine is never
touched.
Takes `Build/lagom/bin/image -o out.tiff ~/Downloads/balloon.jpf` from
704.3 ms ± 5.4 ms to 613.2 ms ± 3.9 ms under hyperfine on my system
(12.9% less). When run under `/usr/bin/time -l`, the changes are:
Before:
18035577126 instructions retired
2862272914 cycles elapsed
After:
14521350138 instructions retired
2492032301 cycles elapsed
balloon.jps uses the lossy 9-7 transform. For lossless files, this helps
too, but less, as the 5-3 transform uses fewer loops in this function.
Also, lossless files spend relatively much more time in bitplane
decoding and less in IDWT, so the relative win for them is smaller too:
For ballon_rev_half.jp2, which is balloon.jpf resized to 50% and
reencoded in lossless, from 1.058 s to 1.035 s.
I'd like to make changes to our JPEG2000 IDWT implementation to speed
it up. At the moment, it's very close to the spec, which seems useful
for correctness and understandability.
So copy the existing file, put the code in there in a Reference
namespace, and in JPEG2000_DEBUG builds, compare the output of the
two implementations. (At the moment, both implementations are
identical, but that will change more and more over time.)
That way, we can run `Meta/test_pdf.py` with a large test corpus with
the comparison turned on to make sure IDWT optimizations aren't
completely broken.
Run `git show` with -C / --find-copies on this commit to make it display
the new file as copy of the old one, along with changes I made to it.
(Namely: I removed the input structs from the reference file as they're
shared with the production file, and I qualified a few calls with
`Reference::`.)
No behavior change.
step_size is identical for all coefficients. No behavior change.
Takes
hyperfine \
'Build/lagom/bin/image --no-output ~/Downloads/balloon.jpf'
from 741.7 ms ± 8.0 ms to 648.0 ms ± 1.9 ms on my system,
a 12.5% reduction.
File created by copying fonts.pdf, and then manually slightly
reformatting the content stream, adding a 30 degree rotation matrix
at the top, and fixing the file up with `mutool clean` as usual.
We handle them all correctly, but we didn't have a checked-in test.
Now we do. See the PR adding this file for how it was created.
Uses SerenitySans transcoded into various formats for the vector fonts,
and KaticaRegular for the type 3 font :^)
This reverts the 2nd commit from #23265. It fires for example for
(valid!) CFF files that use a built-in encoding, such as the file
mentioned in #21469.
Nico Weber