* In `scripts/unicode.py`, the data used to generate `is_combining_mark()`
was being passed to the emit function incorrectly, resulting in the
table containing some other data instead. The script is fixed and new
`tables.rs` is generated.
* Add test for `is_combining_mark()` for ASCII chars, as well as a
couple of random chars based on the reported issue.
Fix https://github.com/unicode-rs/unicode-normalization/issues/16
Rust's default slices are convenient, but for tables like:
const f: &'static [(char, &'static [char])]
they take up far too much space. An element of the above array consumes
24 bytes on 64-bit platforms, and unicode-normalization contains about
6000 such array elements.
A better approach is to manually store a smaller slice type:
struct Slice {
offset: u16,
length: u16,
}
const f: &'static [(char, Slice)]
and store the actual character data in a separate array on the side.
The `Slice` structures then point in to this separate array, but at a
much smaller space cost: elements of the modified `f` take up only 8
bytes on 64-bit platforms, which implies a space savings of ~96K on
64-bit platforms. On some systems, this strategy also eliminates the
necessity of run-time relocations, which can be a further, significant
savings in binary size and runtime cost.
This change is strictly local to the library; it does not affect the
public API.
This change avoids formatting table entries into a string, only to split
them apart again. The new format is also slightly easier to read and
compare when changes are made to how the tables are organized.
This commit imports update to regexes in `load_properties()`,
d25c39f86568a147f9b7080c25711fb1f98f056a in rust-lang/regex
Does not result in any changes to tables.rs, but could if
published tables are updated in the future.
Sujay Jayakar