Hangul Syllables and several other ranges are defined in UnicodeData.txt
as just their first and last values:
```
AC00;<Hangul Syllable, First>;Lo;0;L;;;;;N;;;;;
D7A3;<Hangul Syllable, Last>;Lo;0;L;;;;;N;;;;;
```
Teach the unicode.py script how to recognize these, so that it correctly
classifies them as assigned ranges, for the `is_public_assigned`
predicate.
Add an `is_public_assigned` predicate, which tests whether a given
`char` is assigned (`General_Category` != `Unassigned`) in the currently
supported version of Unicode, and not Private-Use (`General_Category`
!= `Private_Use`).
This comes up in some use cases sensitive to the stability of NFC over
Unicode version changes. An unassigned codepoint could become assigned in
the future, and new normalizations could apply to it.
For further details, see
- <https://unicode.org/reports/tr15/#Versioning>
Switch to a dedicated `svar()` iterator function, which just does
standardized variation sequences, rather than framing this functionality
as an open-ended "extended" version of the standard normalization
algorithms. This makes for a more factored API, gives users more control
over exactly what transformations are done, and has less impact on users
that don't need this new functionality.
The standard normalization algorithm decomposes CJK compatibility ideographs
into nominally equivalent codepoints, but which traditionally look different,
and is one of the main reasons normalization is considered destructive in
practice.
[Unicode 6.3] introduced a solution for this, by providing
[standardized variation sequences] for these codepoints. For example, while
U+2F8A6 "CJK COMPATIBILITY-IDEOGRAPH-2F8A6" canonically decomposes to U+6148
with a different appearance, in Unicode 6.3 and later the standardized variation
sequences in the StandardizedVariants.txt file include the following:
> 6148 FE00; CJK COMPATIBILITY IDEOGRAPH-2F8A6;
which says that "CJK COMPATIBILITY IDEOGRAPH-2F8A6" corresponds to
U+6148 U+FE00, where U+FE00 is "VARIATION SELECTOR-1".
U+6148 and U+FE00 are both normalized codepoints, so we can transform text
containing U+2F8A6 into normal form without losing information about the
distinct appearance. At this time, many popular implementations ignore these
variation selectors, however this technique at least preserves the information
in a standardized way, so implementations could use it if they chose.
This PR adds "ext" versions of the `nfd`, `nfc`, `nfkd`, and `nkfd`
iterators, which perform the standard algorithms extended with this technique.
They don't match the standard decompositions, and don't guarantee stability,
but they do produce appropriately normalized output.
I used the generic term "ext" to reflect that other extensions could
theoretically be added in the future. The standard decomposition tables are
limited by their stability requirements, but these "ext" versions could be
free to adopt new useful rules.
I'm not an expert in any of these topics, so please correct me if I'm mistaken
in any of this. Also, I'm open to ideas about how to best present this
functionality in the API.
[Unicode 6.3]: https://www.unicode.org/versions/Unicode6.3.0/#Summary
[standardized variation sequences]: http://unicode.org/faq/vs.html
Having the Some() constructor in each match arm body puts extras stress
on rustc's item_bodies_checking pass. We can work around that by moving
the Some() constructor around the match, and directly returning None
from the default arm, which is the only one that generates a None value.
On my box, this almost cuts the time spent in item_bodies_checking in
half, going from about 8.7s to about 4.6s, and reduces the complete
compile time from about 13s to about 9s, so about a third less.
There are no changes in performance in `cargo bench`.
Note that doing the same for the composition_table() function does not
yield any compile time wins, but would cause a performance regression.
cc #29
* In `scripts/unicode.py`, the data used to generate `is_combining_mark()`
was being passed to the emit function incorrectly, resulting in the
table containing some other data instead. The script is fixed and new
`tables.rs` is generated.
* Add test for `is_combining_mark()` for ASCII chars, as well as a
couple of random chars based on the reported issue.
Fix https://github.com/unicode-rs/unicode-normalization/issues/16
Rust's default slices are convenient, but for tables like:
const f: &'static [(char, &'static [char])]
they take up far too much space. An element of the above array consumes
24 bytes on 64-bit platforms, and unicode-normalization contains about
6000 such array elements.
A better approach is to manually store a smaller slice type:
struct Slice {
offset: u16,
length: u16,
}
const f: &'static [(char, Slice)]
and store the actual character data in a separate array on the side.
The `Slice` structures then point in to this separate array, but at a
much smaller space cost: elements of the modified `f` take up only 8
bytes on 64-bit platforms, which implies a space savings of ~96K on
64-bit platforms. On some systems, this strategy also eliminates the
necessity of run-time relocations, which can be a further, significant
savings in binary size and runtime cost.
This change is strictly local to the library; it does not affect the
public API.
This change avoids formatting table entries into a string, only to split
them apart again. The new format is also slightly easier to read and
compare when changes are made to how the tables are organized.
This commit imports update to regexes in `load_properties()`,
d25c39f86568a147f9b7080c25711fb1f98f056a in rust-lang/regex
Does not result in any changes to tables.rs, but could if
published tables are updated in the future.
Dan Gohman