mirror of
https://github.com/BepInEx/mono.git
synced 2026-11-18 17:02:10 -10:00
Note: This revision was originally supplied in six parts. Notes
accompanying each of the parts are reproduced below.
[PATCH v5 1/1] System.Char icall: New Unicode category tables compatible with MS .NET
These new category tables, generated using create-category-table(.cs)
available in the mcs sources, make the Mono runtime understand the
same Unicode category codes as recent versions of Microsoft's runtime.
CategoryData_v2 matches versions v2.0.50727 and v3.5.21022 of .NET.
CategoryData_v4 matches version v4.0.30319 of .NET. The v4 tables can
be disabled by defining the DISABLE_NET_4_0 preprocessor symbol.
Similarly, the astral portions of both tables can be disabled by
defining the DISABLE_ASTRAL preprocessor symbol.
CAUTION: Mcs has to be updated with the corresponding patches, or Mono
will suffer a low-level (internal call) crash when initializing
System.Char.
[PATCH v5 1/5] create-category-table: Utility to generate reasonably-packed Unicode tables
This program generates (partially) bi-level tables encoding the
contents of the Unicode character category database.
Mono embeds a linear table with category codes for the Unicode
BMP (first 65536 codepoints), and lacks information about characters
in the astral planes--leading to requests such as bug 480178.
Extending the linear table to cover the full codespace is not an ideal
solution, as that would expand the embedded "blob" by a factor of 17.
The new tables generated by this program can be used to support the
full range of characters. An additional level of indirection used for
characters outside the U+0000..U+FFFF range enables "page" sharing, so
that the total amount of embedded data only grows by 13.5kB.
Cf. in-file comments for usage instructions.
[PATCH v5 2/5] System.Char: Handle astral planes in (string,int) overload
If the string element at index starts a surrogate pair, we decode the
full codepoint and "query" the higher planes of the database.
This fixes bug 480178.
CAUTION: This commit depends on the following runtime change:
System.Char icall: New Unicode category tables compatible with
MS .NET.
Without it, Mono will suffer a low-level (internal call) crash when
initializing System.Char.
COMPATIBILITY
The updated Mono runtime as been verified to produce the same results
as Microsoft's; here are the MD5 sums of their Unicode category
database dumps (generated via create-category-table --dump, compiled
and executed under the relevant runtime):
eba45e00acdc82f9a08873465110aef4 v2.0.50727.dump
eba45e00acdc82f9a08873465110aef4 v3.5.21022.dump
56fd5c828fbb9083693835680667fd2c v4.0.30319.dump
eba45e00acdc82f9a08873465110aef4 gmcs.dump
56fd5c828fbb9083693835680667fd2c dmcs.dump
These results will naturally differ if support for .NET v4 or astral
planes is disabled when building the runtime. Note, also, that these
are different from the results produced by Mono prior to the
introduction of these changes, even for the BMP.
Other versions of the internal database can be easily generated using
create-category-table(.cs), but this currently requires a rebuild.
PERFORMANCE
Direct array indexing is mandatory for code points in the
U+0000..U+FFFF range; as pointed out by Andreas Nahr, performing
bi-level lookups in the Char.Is*(char) predicates cause the JIT to
stop inlining them and results in an important performance drop.
The simple data access pattern used for higher planes, suggested by
Paolo Molaro, is fairly efficient but currenly only used by this
non-optimized method.
[PATCH v5 3/5] Globalization.IdnMappingTest: U+0221 is not unassigned anymore, use U+18B0
U+0221 is LATIN SMALL LETTER D WITH CURL in Unicode 4.0, and is recognized as a
valid letter in recent versions of the .NET framework. This causes
GetAsciiInvalid to (unexpectedly) succeed, and the test to fail.
Use U+18B0 instead, which is unassigned as of Unicode 5.2 and .NET 3.5 SP1.
[PATCH v5 4/5] CharCategoryTest: Update category dumps to match MS .NET
The category codes in newer versions of Microsoft's framework are not
compatible with the ones encoded in this class, causing tests to fail.
Fix that by using new sets of dumps generated under v3.5.21022 and
v4.0.30319.
Also, drop CharTest.IsWhiteSpace_Complete as the same checks are
performed by CharCategoryTest.IsWhiteSpace.
While we are at it, improve the generator so that:
- common code is shared between the generator and the test;
- a number of test cases can be generated at once via
command-line arguments;
- the dumps are word-wrapped, enhancing legibility and avoiding
line length limitations in some compilers.
[PATCH v5 5/5] CharTest: Add tests for astral plane characters/surrogate pairs
- Show that Char.GetUnicodeCategory(string,int) results in a
lookup into the higher planes of the Unicode database when invoked
on a valid surrogate pair, and returns the "normal" category code
otherwise.
- Perform a few sanity checks on the results of ConvertFromUtf32
and GetUnicodeCategory when invoked on codepoints from the
higher planes.
(cherry picked from commit 70dff8b4272dba83bed23e6502499646526efd59)
198 lines
6.1 KiB
C#
198 lines
6.1 KiB
C#
//
|
|
// IdnMappingTest.cs
|
|
//
|
|
// Author:
|
|
// Atsushi Enomoto <[email protected]>
|
|
//
|
|
// Copyright (C) 2007 Novell, Inc. (http://www.novell.com)
|
|
//
|
|
// Permission is hereby granted, free of charge, to any person obtaining
|
|
// a copy of this software and associated documentation files (the
|
|
// "Software"), to deal in the Software without restriction, including
|
|
// without limitation the rights to use, copy, modify, merge, publish,
|
|
// distribute, sublicense, and/or sell copies of the Software, and to
|
|
// permit persons to whom the Software is furnished to do so, subject to
|
|
// the following conditions:
|
|
//
|
|
// The above copyright notice and this permission notice shall be
|
|
// included in all copies or substantial portions of the Software.
|
|
//
|
|
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
|
// LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|
// OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
|
// WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
|
//
|
|
|
|
#if NET_2_0
|
|
|
|
using NUnit.Framework;
|
|
using System.IO;
|
|
using System;
|
|
using System.Globalization;
|
|
using System.Threading;
|
|
using System.Text;
|
|
|
|
namespace MonoTests.System.Globalization
|
|
{
|
|
[TestFixture]
|
|
public class IdnMappingTest
|
|
{
|
|
void GetAscii (IdnMapping m, string source, string expected, object label)
|
|
{
|
|
Assert.AreEqual (expected, m.GetAscii (source), label != null ? label.ToString () : expected);
|
|
}
|
|
|
|
void GetAsciiInvalid (IdnMapping m, string s, object label)
|
|
{
|
|
try {
|
|
m.GetAscii (s);
|
|
Assert.Fail (label != null ? label.ToString () + ":" + s : s);
|
|
} catch (ArgumentException) {
|
|
}
|
|
}
|
|
|
|
void GetUnicode (IdnMapping m, string source, string expected, object label)
|
|
{
|
|
Assert.AreEqual (expected, m.GetUnicode (source), label != null ? label.ToString () : expected);
|
|
}
|
|
|
|
void GetUnicodeInvalid (IdnMapping m, string s, object label)
|
|
{
|
|
try {
|
|
m.GetUnicode (s);
|
|
Assert.Fail (label != null ? label.ToString () + ":" + s : s);
|
|
} catch (ArgumentException) {
|
|
}
|
|
}
|
|
|
|
[Test]
|
|
public void GetAsciiInvalid ()
|
|
{
|
|
// hmm... according to RFC 3490, there are couple of
|
|
// invalid characters in RFC 3491 [Nameprop] ...
|
|
// comment out them for now.
|
|
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAsciiInvalid (m, "\x0", 1);
|
|
GetAsciiInvalid (m, "\xD\xA", 2);
|
|
GetAsciiInvalid (m, "\x1F", 3);
|
|
GetAsciiInvalid (m, "\x7F", 4);
|
|
GetAsciiInvalid (m, "\x80", 5);
|
|
// GetAsciiInvalid (m, "\xA0", 6);
|
|
// GetAsciiInvalid (m, "\u1680", 7);
|
|
GetAsciiInvalid (m, "\u200E", 8);
|
|
// GetAsciiInvalid (m, "\u0341", 9);
|
|
GetAsciiInvalid (m, "\uE000", 10);
|
|
GetAsciiInvalid (m, "\uFFFF", 11);
|
|
GetAsciiInvalid (m, "\u2028", 12);
|
|
GetAsciiInvalid (m, "\uD800", 13);
|
|
// GetAsciiInvalid (m, "\u3000", 14);
|
|
}
|
|
|
|
[Test]
|
|
public void GetAscii ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAscii (m, "www.mono-project.com", "www.mono-project.com", 1);
|
|
GetAscii (m, "a\uFF61b", "a.b", 2);
|
|
|
|
// umm... does it reject anything larger than U+FFFF ??
|
|
// GetAscii (m, "\uD800\uDC00", "", 3);
|
|
|
|
// hmm, according to RFC 3490, ToASCII never changes
|
|
// the input whose codepoints are all within ASCII.
|
|
GetAscii (m, "A\uFF61B", "a.b", 4);
|
|
GetAscii (m, "A.B", "A.B", 5); // no lowercasing
|
|
}
|
|
|
|
[Test]
|
|
public void Dots ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAsciiInvalid (m, ".", 1);
|
|
GetAsciiInvalid (m, "\uFF61", 2);
|
|
GetAscii (m, "Am running now.", "Am running now.", 3);
|
|
GetAsciiInvalid (m, ".bashrc", 4);
|
|
}
|
|
|
|
[Test]
|
|
public void UnassignedCharacter ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAsciiInvalid (m, "\u18b0.com", 1);
|
|
m.AllowUnassigned = true;
|
|
GetAscii (m, "\u18b0.com", "xn--6bf.com", 2);
|
|
}
|
|
|
|
[Test]
|
|
public void UseStd3AsciiRules ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAscii (m, "-_-.com", "-_-.com", 1);
|
|
m.UseStd3AsciiRules = true;
|
|
GetAsciiInvalid (m, "a b.com", 1.5);
|
|
GetAsciiInvalid (m, "_.com", 2);
|
|
GetAsciiInvalid (m, "-abc.com", 3);
|
|
GetAscii (m, "abc-def.com", "abc-def.com", 4); // hyphen in the middle is okay
|
|
GetAsciiInvalid (m, "abc-.def.com", 5); // hyphen is *not* in the middle.
|
|
}
|
|
|
|
[Test]
|
|
public void AcePrefix ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAscii (m, "bl--.com", "bl--.com", 1); // only ascii
|
|
GetAscii (m, "bl--\xC0.com", "xn--bl---3na.com", 2);
|
|
GetAscii (m, "xn--.com", "xn--.com", 3); // only ascii
|
|
GetAsciiInvalid (m, "xn--\xC0.com", 4);
|
|
GetAsciiInvalid (m, "xN--\xC0.com", 5);
|
|
GetAsciiInvalid (m, "Xn--\xC0.com", 6);
|
|
GetAsciiInvalid (m, "XN--\xC0.com", 7);
|
|
GetAscii (m, "xN\xC0.com", "xn--xn-kia.com", 8);
|
|
GetAscii (m, "bl--\xC0.wl--\xC0.com", "xn--bl---3na.xn--wl---3na.com", 9);
|
|
}
|
|
|
|
[Test]
|
|
public void GetAsciiRFC3492Examples ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
// 3<nen>B<gumi><kin><pachi><sen><sei>
|
|
GetAscii (m, "3\u5E74B\u7D44\u91D1\u516B\u5148\u751F", "xn--3b-ww4c5e180e575a65lsy2b", "(L)");
|
|
// Maji<de>Koi<suru>5<byoumae>
|
|
GetAscii (m, "Maji\u3067Koi\u3059\u308B5\u79D2\u524D", "xn--majikoi5-783gue6qz075azm5e", "(P)");
|
|
}
|
|
|
|
[Test]
|
|
public void GetUnicode ()
|
|
{
|
|
IdnMapping m = new IdnMapping ();
|
|
GetAscii (m, "www.mono-project.com", "www.mono-project.com", 1);
|
|
GetAscii (m, "a\uFF61b", "a.b", 2);
|
|
|
|
// umm... does it reject anything larger than U+FFFF ??
|
|
// GetAscii (m, "\uD800\uDC00", "", 3);
|
|
|
|
// hmm, according to RFC 3490, ToASCII never changes
|
|
// the input whose codepoints are all within ASCII.
|
|
GetAscii (m, "A\uFF61B", "a.b", 4);
|
|
GetAscii (m, "A.B", "A.B", 5); // no lowercasing
|
|
}
|
|
|
|
[Test]
|
|
public void GetUnicodeRFC3492Examples ()
|
|
{
|
|
// uppercases in the ASCII strings are ignored.
|
|
IdnMapping m = new IdnMapping ();
|
|
// 3<nen>B<gumi><kin><pachi><sen><sei>
|
|
GetUnicode (m, "xn--3B-ww4c5e180e575a65lsy2b", "3\u5E74b\u7D44\u91D1\u516B\u5148\u751F", "(L)");
|
|
// Maji<de>Koi<suru>5<byoumae>
|
|
GetUnicode (m, "xn--MajiKoi5-783gue6qz075azm5e", "maji\u3067koi\u3059\u308B5\u79D2\u524D", "(P)");
|
|
}
|
|
}
|
|
}
|
|
|
|
#endif
|