Note: This revision was originally supplied in six parts. Notes
accompanying each of the parts are reproduced below.
[PATCH v5 1/1] System.Char icall: New Unicode category tables compatible with MS .NET
These new category tables, generated using create-category-table(.cs)
available in the mcs sources, make the Mono runtime understand the
same Unicode category codes as recent versions of Microsoft's runtime.
CategoryData_v2 matches versions v2.0.50727 and v3.5.21022 of .NET.
CategoryData_v4 matches version v4.0.30319 of .NET. The v4 tables can
be disabled by defining the DISABLE_NET_4_0 preprocessor symbol.
Similarly, the astral portions of both tables can be disabled by
defining the DISABLE_ASTRAL preprocessor symbol.
CAUTION: Mcs has to be updated with the corresponding patches, or Mono
will suffer a low-level (internal call) crash when initializing
System.Char.
[PATCH v5 1/5] create-category-table: Utility to generate reasonably-packed Unicode tables
This program generates (partially) bi-level tables encoding the
contents of the Unicode character category database.
Mono embeds a linear table with category codes for the Unicode
BMP (first 65536 codepoints), and lacks information about characters
in the astral planes--leading to requests such as bug 480178.
Extending the linear table to cover the full codespace is not an ideal
solution, as that would expand the embedded "blob" by a factor of 17.
The new tables generated by this program can be used to support the
full range of characters. An additional level of indirection used for
characters outside the U+0000..U+FFFF range enables "page" sharing, so
that the total amount of embedded data only grows by 13.5kB.
Cf. in-file comments for usage instructions.
[PATCH v5 2/5] System.Char: Handle astral planes in (string,int) overload
If the string element at index starts a surrogate pair, we decode the
full codepoint and "query" the higher planes of the database.
This fixes bug 480178.
CAUTION: This commit depends on the following runtime change:
System.Char icall: New Unicode category tables compatible with
MS .NET.
Without it, Mono will suffer a low-level (internal call) crash when
initializing System.Char.
COMPATIBILITY
The updated Mono runtime as been verified to produce the same results
as Microsoft's; here are the MD5 sums of their Unicode category
database dumps (generated via create-category-table --dump, compiled
and executed under the relevant runtime):
eba45e00acdc82f9a08873465110aef4 v2.0.50727.dump
eba45e00acdc82f9a08873465110aef4 v3.5.21022.dump
56fd5c828fbb9083693835680667fd2c v4.0.30319.dump
eba45e00acdc82f9a08873465110aef4 gmcs.dump
56fd5c828fbb9083693835680667fd2c dmcs.dump
These results will naturally differ if support for .NET v4 or astral
planes is disabled when building the runtime. Note, also, that these
are different from the results produced by Mono prior to the
introduction of these changes, even for the BMP.
Other versions of the internal database can be easily generated using
create-category-table(.cs), but this currently requires a rebuild.
PERFORMANCE
Direct array indexing is mandatory for code points in the
U+0000..U+FFFF range; as pointed out by Andreas Nahr, performing
bi-level lookups in the Char.Is*(char) predicates cause the JIT to
stop inlining them and results in an important performance drop.
The simple data access pattern used for higher planes, suggested by
Paolo Molaro, is fairly efficient but currenly only used by this
non-optimized method.
[PATCH v5 3/5] Globalization.IdnMappingTest: U+0221 is not unassigned anymore, use U+18B0
U+0221 is LATIN SMALL LETTER D WITH CURL in Unicode 4.0, and is recognized as a
valid letter in recent versions of the .NET framework. This causes
GetAsciiInvalid to (unexpectedly) succeed, and the test to fail.
Use U+18B0 instead, which is unassigned as of Unicode 5.2 and .NET 3.5 SP1.
[PATCH v5 4/5] CharCategoryTest: Update category dumps to match MS .NET
The category codes in newer versions of Microsoft's framework are not
compatible with the ones encoded in this class, causing tests to fail.
Fix that by using new sets of dumps generated under v3.5.21022 and
v4.0.30319.
Also, drop CharTest.IsWhiteSpace_Complete as the same checks are
performed by CharCategoryTest.IsWhiteSpace.
While we are at it, improve the generator so that:
- common code is shared between the generator and the test;
- a number of test cases can be generated at once via
command-line arguments;
- the dumps are word-wrapped, enhancing legibility and avoiding
line length limitations in some compilers.
[PATCH v5 5/5] CharTest: Add tests for astral plane characters/surrogate pairs
- Show that Char.GetUnicodeCategory(string,int) results in a
lookup into the higher planes of the Unicode database when invoked
on a valid surrogate pair, and returns the "normal" category code
otherwise.
- Perform a few sanity checks on the results of ConvertFromUtf32
and GetUnicodeCategory when invoked on codepoints from the
higher planes.
(cherry picked from commit 70dff8b4272dba83bed23e6502499646526efd59)
* CultureInfo.cs : it seems that zh-Hant and zh-Hans are introduced
as almost alias of zh-CHS/zh-CHT in vista and some new windows.
Add hacky support for them.
* CultureInfoTest.cs : test zh-Hant.
svn path=/trunk/mcs/; revision=134499
* CompareInfo.cs : Fixed bug #462957. With empty target string it
should always return 0.
* CompareInfoTest.cs : added test for #462957.
svn path=/trunk/mcs/; revision=122602
* HijriCalendar.cs ThaiBuddhistCalendar.cs HebrewCalendar.cs
GregorianCalendar.cs KoreanCalendar.cs JulianCalendar.cs :
implemented ToFourDigitYear(), actually base implementation mostly
works as is.
* CalendarTest.cs : enabled ToFourDigitYear() test from 5 years ago.
and added some more ToFourDigitYear() tests.
svn path=/trunk/mcs/; revision=81688
TextInfo if current culture is invariant. Remoted private ctor.
In Default, always construct new instance since the current culture
may change.
* CaseInsensitiveHashCodeProviderTest.cs: Improved test for
Default property to show the a new intance is constructed. Added test
for DefaultInvariant. Improved serialization tests. Added ctor test
for null CultureInfo. Code formatting.
* CultureInfoTest.cs: Fixed UseUserOverride tests to pass on systems
where invariant culture is default locale.
svn path=/trunk/mcs/; revision=81408
* FileSystemInfoTest.cs: Restore original CurrentCulture on teardown.
Removed stray tabs. Code formatting.
* FileTest.cs: Restore original CurrentCulture on teardown. Removed
stray tabs. Changes spaces to tabs.
* ThreadTest.cs: Added tests for bug #81930.
* CultureInfoTest.cs: Added exception tests when invoking NumberFormat
or DateTimeFormat on a neutral culture. Added tests for bug #81930.
No longer derive from deprecated Assertion class. Added tests for
GetCultureInfo (2.0).
* ArrayTest.cs: Added test for bug #81941.
* ByteTest.cs: Restore original CurrentCulture in teardown.
* DecimalFormatterTest.cs: Restore original CurrentCulture in teardown.
Code formatting.
* DecimalTest.cs: Code formatting.
* DoubleFormatterTest.cs: Test relies on specific culture, so set it
up here. This worked before since the CurrentCulture was set in another
TestFixture, and the original one was not restored.
* FloatingPointFormatterTest.cs: Same.
* NumberFormatterTest.cs: Same. Removed stray tabs and fixes code
formatting.
* SingleFormatterTest.cs: Restore original culture on teardown. Code
formatting.
* SingleTest.cs: Same.
* StringComparerTest.cs: Restore original culture on teardown.
svn path=/trunk/mcs/; revision=80626
2007-05-31 Rolf Bjarne Kvinge <[email protected]>
* corlib_test.dll.sources : added DateTimeFormatInfoTest.cs.
In Test/System.Globalization:
2007-05-31 Rolf Bjarne Kvinge <[email protected]>
* DateTimeFormatInfo.cs: Added EraName.
In System.Globalization:
2007-05-31 Rolf Bjarne Kvinge <[email protected]>
* GregorianCalendar.cs: Change era names from C.E./Common Era
to AD/A.D.
* DateTimeFormatInfo.cs: GetEra: search the abbreviated era names
as well for the era, and return the era, not the index of the era
in the array.
svn path=/trunk/mcs/; revision=78435
* Calendar.cs DateTimeFormatInfo.cs : 2.0 API fixes.
* CalendricalCalculations.cs : fixed AddMonth() and AddYears() to
consider that the max days in the target month (for AddYears()
leap year month could matter). Fixed bug #81783.
* CalendarTest.cs : test for bug #81783 and AddYears() for leap year
(similar issue).
svn path=/trunk/mcs/; revision=78429
* RegionInfo.cs, CultureInfo.cs : in 2.0 RegionInfo could be created
from locale name. To support that, retrieve CultureInfo from name
and use its Territory name if exists.
RegionInfo.Equals() still uses lcid, but hash code is based on
region ID. Added GeoId, which represents unique region ID (which
is returned in 1.x GetHashCode()).
* RegionInfoTest.cs : new tests.
* corlib_test.dll.sources : added RegionInfoTest.cs.
svn path=/trunk/mcs/; revision=77408
* SimpleCollator.cs : disable QuickCheckPossible(), which is
inaccurate and inefficient. Fixed bug #79714.
* CompareInfoTest.cs : added test for bug #79714.
svn path=/trunk/mcs/; revision=73785
* SimpleCollator.cs : character filtering is needed for
OrdinalIgnoreCase in 2.0 profile. Fixed bug #80865.
* CompareInfoTest.cs : added test for bug #80865.
svn path=/trunk/mcs/; revision=72902
* SimpleCollator.cs : GetTailContraction() was broken to pick correct
contraction/special sortkey out and thus LastIndexOf() failed when
it is involved. Fixed bug #80612.
* CompareInfoTest.cs : added test for bug #80612.
svn path=/trunk/mcs/; revision=71632
* SimpleCollator.cs : for non-StringSort comparison, level5 (- and ')
should be still skipped after initial level5 check is done (while
they were simply treated as a normal character). Fixed bug #78748.
* SortKeyBuffer.cs : Fixed NRE in french sort.
* CompareInfoTest.cs : Added Compare() tests for hyphens.
Removed NotWorking from previous new test (its existence wasn't
even recognized until today).
svn path=/trunk/mcs/; revision=71436
* SimpleCollator.cs : added IndexOf() implementation for Ordinal
and OrdinalIgnoreCase, though Ordinal version is not used (since
it is slower than icall).
* CompareInfo.cs : for OrdinalIgnoreCase, IndexOf() and LastIndexOf()
will use managed version (icalls for them are case sensitive).
Removed MonoTODOs.
* CompareInfoTest.cs : added tests for OrdinalIgnoreCase versions
of IndexOf() and LastIndexOf(). Added Ordinal LastIndexOf() tests.
svn path=/trunk/mcs/; revision=70037
* CompareInfoTest.cs : seems like Microsoft had introduced several
bugs in CompareInfo, so marked many of them as NotDotNet.
svn path=/trunk/mcs/; revision=59839
* CultureInfo.cs : Clone() attempted to clone number and date time
formats, thus failed. Fixed bug #77347.
* CultureInfoTest.cs : added test for bug #77347.
svn path=/trunk/mcs/; revision=56032
* CompareInfoTest.cs : split the last case in NullCharacter() out
and marked it as NotDotNet (.NET is inconsistent between Compare
and IndexOf).
svn path=/trunk/mcs/; revision=53920
* CompareInfo.cs : added support for OrdinalIgnoreCase.
* CompareInfoTest.cs : added test for Compare() w/ OrdinalIgnoreCase.
svn path=/trunk/mcs/; revision=53004
* SimpleCollator.cs :
- Added CompareOrdinalIgnoreCase() for NET_2_0 RTM.
- Reduced extra parameter from LastIndexOfSortKey().
- LastIndexOf() should use GetTailContraction for the source string.
And then, target could match in the middle of the possible
"replacement contraction" of the source string, so use
LastIndexOfSortKey() to catch them.
- Fixed GetTailContraction() that caused index out of range.
* CompareInfoTest.cs : added more tests for some specific cultures
(say, Japanese) that have replacement contraction in ASCII range
and caused incorrect results for LastIndexOf() and all dependent
members.
svn path=/trunk/mcs/; revision=53003
2005-10-14 Ben Maurer <[email protected]>
* DateTime.cs: Speed up when parsing date time objects by not
duplicating cultureinfo arrays.
In Test/System.Globalization:
2005-10-14 Ben Maurer <[email protected]>
* DateTimeFormatInfoTest.cs: New test to make sure we do not
return the same array from GetAllDateTimePatterns.
In System.Globalization:
2005-10-14 Ben Maurer <[email protected]>
* DateTimeFormatInfo.cs: Speed up GetAllDateTimePatterns by
caching the result and then cloning it.
svn path=/trunk/mcs/; revision=51696
* SimpleCollator.cs : in CompareInternal(), there was a possibility of
infinite loop. Fixed bug #76243.
* CompareInfoTest.cs : added testcase for bug #76243.
svn path=/trunk/mcs/; revision=50837
* SimpleCollator.cs : IsSuffix() optimization logic was buggy, so just
use pretty simple way with LastIndexOf() (no significant perf.
problem).
* CompareInfoTest.cs : added another test for IsSuffix() that I found.
svn path=/trunk/mcs/; revision=49793