Having the Some() constructor in each match arm body puts extras stress
on rustc's item_bodies_checking pass. We can work around that by moving
the Some() constructor around the match, and directly returning None
from the default arm, which is the only one that generates a None value.
On my box, this almost cuts the time spent in item_bodies_checking in
half, going from about 8.7s to about 4.6s, and reduces the complete
compile time from about 13s to about 9s, so about a third less.
There are no changes in performance in `cargo bench`.
Note that doing the same for the composition_table() function does not
yield any compile time wins, but would cause a performance regression.
cc #29
* In `scripts/unicode.py`, the data used to generate `is_combining_mark()`
was being passed to the emit function incorrectly, resulting in the
table containing some other data instead. The script is fixed and new
`tables.rs` is generated.
* Add test for `is_combining_mark()` for ASCII chars, as well as a
couple of random chars based on the reported issue.
Fix https://github.com/unicode-rs/unicode-normalization/issues/16
Rust's default slices are convenient, but for tables like:
const f: &'static [(char, &'static [char])]
they take up far too much space. An element of the above array consumes
24 bytes on 64-bit platforms, and unicode-normalization contains about
6000 such array elements.
A better approach is to manually store a smaller slice type:
struct Slice {
offset: u16,
length: u16,
}
const f: &'static [(char, Slice)]
and store the actual character data in a separate array on the side.
The `Slice` structures then point in to this separate array, but at a
much smaller space cost: elements of the modified `f` take up only 8
bytes on 64-bit platforms, which implies a space savings of ~96K on
64-bit platforms. On some systems, this strategy also eliminates the
necessity of run-time relocations, which can be a further, significant
savings in binary size and runtime cost.
This change is strictly local to the library; it does not affect the
public API.
This change avoids formatting table entries into a string, only to split
them apart again. The new format is also slightly easier to read and
compare when changes are made to how the tables are organized.
Fix#11.
The algorithm for composition of Hangul Jamo is:
- L (choseong jamo) + V (jungseong jamo) = LV (syllable block)
- LV (syllable block) + T (jongseong jamo) = LVT (syllable block)
However, the LV and LVT syllable blocks are intermingled in the unicode
block. In particular, for each pair LV, you will first see the syllable block
LV, followed by syllable blocks for LVT for each T. The LV+T
composition was a simple addition of offsets.
Our algorithm did not ignore the LVT syllable blocks, which meant that
LVT+T would just offset further and produce an unrelated syllable block.
By ensuring that the `S_index` is a multiple of `T_count`, we filter
for only LV syllable blocks (which occur every `T_count` codepoints in
the S block)
Manish Goregaokar