* In `scripts/unicode.py`, the data used to generate `is_combining_mark()`
was being passed to the emit function incorrectly, resulting in the
table containing some other data instead. The script is fixed and new
`tables.rs` is generated.
* Add test for `is_combining_mark()` for ASCII chars, as well as a
couple of random chars based on the reported issue.
Fix https://github.com/unicode-rs/unicode-normalization/issues/16
Rust's default slices are convenient, but for tables like:
const f: &'static [(char, &'static [char])]
they take up far too much space. An element of the above array consumes
24 bytes on 64-bit platforms, and unicode-normalization contains about
6000 such array elements.
A better approach is to manually store a smaller slice type:
struct Slice {
offset: u16,
length: u16,
}
const f: &'static [(char, Slice)]
and store the actual character data in a separate array on the side.
The `Slice` structures then point in to this separate array, but at a
much smaller space cost: elements of the modified `f` take up only 8
bytes on 64-bit platforms, which implies a space savings of ~96K on
64-bit platforms. On some systems, this strategy also eliminates the
necessity of run-time relocations, which can be a further, significant
savings in binary size and runtime cost.
This change is strictly local to the library; it does not affect the
public API.
This change avoids formatting table entries into a string, only to split
them apart again. The new format is also slightly easier to read and
compare when changes are made to how the tables are organized.
Fix#11.
The algorithm for composition of Hangul Jamo is:
- L (choseong jamo) + V (jungseong jamo) = LV (syllable block)
- LV (syllable block) + T (jongseong jamo) = LVT (syllable block)
However, the LV and LVT syllable blocks are intermingled in the unicode
block. In particular, for each pair LV, you will first see the syllable block
LV, followed by syllable blocks for LVT for each T. The LV+T
composition was a simple addition of offsets.
Our algorithm did not ignore the LVT syllable blocks, which meant that
LVT+T would just offset further and produce an unrelated syllable block.
By ensuring that the `S_index` is a multiple of `T_count`, we filter
for only LV syllable blocks (which occur every `T_count` codepoints in
the S block)
David Judd