Commit Graph

15 Commits

Author SHA1 Message Date
Sujay Jayakar e03d61cce8 Merge pull request #36 from djudd/compatibility-quick-checks
Add quick check implementations for NFKC & NFKD
2019-01-14 22:40:37 +05:30
David Judd 5685166427 Add quick check implementations for NFKC & NFKD 2019-01-12 14:44:53 -08:00
Björn Steinbrink 41914c5d30 Generate code that is friendlier to rustc's item_bodies_checking
Having the Some() constructor in each match arm body puts extras stress
on rustc's item_bodies_checking pass. We can work around that by moving
the Some() constructor around the match, and directly returning None
from the default arm, which is the only one that generates a None value.

On my box, this almost cuts the time spent in item_bodies_checking in
half, going from about 8.7s to about 4.6s, and reduces the complete
compile time from about 13s to about 9s, so about a third less.

There are no changes in performance in `cargo bench`.

Note that doing the same for the composition_table() function does not
yield any compile time wins, but would cause a performance regression.

cc #29
2019-01-08 17:02:41 +01:00
Sujay Jayakar 4d7e81e0ab Add Stream-Safe Text Process (for interop with golang's unicode/norm package) 2018-05-01 21:43:04 -07:00
Sujay Jayakar 6a7e38e376 Refactor Unicode script, emit new tables, and add QuickCheck algorithm. 2018-05-01 21:00:38 -07:00
Behnam Esfahbod 1e74c347b6 Fix is_combining_mark table data
* In `scripts/unicode.py`, the data used to generate `is_combining_mark()`
was being passed to the emit function incorrectly, resulting in the
table containing some other data instead. The script is fixed and new
`tables.rs` is generated.

* Add test for `is_combining_mark()` for ASCII chars, as well as a
couple of random chars based on the reported issue.

Fix https://github.com/unicode-rs/unicode-normalization/issues/16
2017-06-01 15:10:29 -06:00
Nathan Froyd 8d01bc5e88 store smaller slices in unicode data tables
Rust's default slices are convenient, but for tables like:

  const f: &'static [(char, &'static [char])]

they take up far too much space.  An element of the above array consumes
24 bytes on 64-bit platforms, and unicode-normalization contains about
6000 such array elements.

A better approach is to manually store a smaller slice type:

  struct Slice {
    offset: u16,
    length: u16,
  }

  const f: &'static [(char, Slice)]

and store the actual character data in a separate array on the side.
The `Slice` structures then point in to this separate array, but at a
much smaller space cost: elements of the modified `f` take up only 8
bytes on 64-bit platforms, which implies a space savings of ~96K on
64-bit platforms.  On some systems, this strategy also eliminates the
necessity of run-time relocations, which can be a further, significant
savings in binary size and runtime cost.

This change is strictly local to the library; it does not affect the
public API.
2017-03-27 16:58:31 -04:00
Nathan Froyd 7552e6cd2b format tables with one entry per line
This change avoids formatting table entries into a string, only to split
them apart again.  The new format is also slightly easier to read and
compare when changes are made to how the tables are organized.
2017-03-27 15:13:36 -04:00
Nathan Froyd 6190241cc9 use more idiomatic python in unicode.py
STR.join(...) is better than manual `for` loops that remember whether
we've processed the first element.
2017-03-27 14:26:35 -04:00
Simon Sapin 5d0443bd92 Add a char::is_combining_mark function.
This is used for international domain name validation:
http://unicode.org/reports/tr46/#Validity_Criteria

> The label must not begin with a combining mark, that is: General_Category=Mark.
2016-01-15 17:49:39 +01:00
kwantam e0d33a8661 make load_properties() tolerate whitespace
This commit imports update to regexes in `load_properties()`,
d25c39f86568a147f9b7080c25711fb1f98f056a in rust-lang/regex

Does not result in any changes to tables.rs, but could if
published tables are updated in the future.
2015-04-16 16:11:04 -04:00
kwantam c7a8bcfdc5 make sure unicode.py puts out optimized tables 2015-04-15 20:04:41 -04:00
kwantam 3d02ca9e05 tiny doc touch 2015-04-15 16:03:36 -04:00
kwantam 7a49d257eb add Unicode norms test suite; bump to ver 0.0.3 2015-04-15 15:40:06 -04:00
kwantam 933f988530 add tablegen script and tables.rs 2015-04-14 21:22:32 -04:00