The previous implementation of GetMaxByteCount has two bugs: it incorrectly took into account the byte order mark and it was taking the max of a byte count (3) and a character count (EncoderFallback.MaxCharCount). This could result in returning the max number of bytes required to convert a string that was less than the number bytes used when converting the string.
GetMaxCharCount did not previously take into account DecoderFallback.MaxCharCount or the fact that there may still be part of a surrogate pair left on the buffer from a previous call to convert.
I removed the #if NET_2_0 conditionals and NotWorking category since these methods now return results consistent with Microsoft's implementation. I also added two simple tests for custom EncoderFallback and DecoderFallback cases.
This commit is licensed under MIT/X11.
Similar to previously reported bug #6404, according to the documentation at http://msdn.microsoft.com/en-us/library/h6w985hz.aspx, System.Text.Encoder throws ArgumentOutOfRangeExceptions when it should not. From the documentation, an ArgumentOutOfRangeException should be thrown when:
charIndex, charCount, byteIndex, or byteCount is less than zero.
-or-
The length of chars - charIndex is less than charCount.
-or-
The length of bytes - byteIndex is less than byteCount.
To reproduce, run the following code:
int charsUsed, bytesUsed;
bool completed;
byte[] bytes = new byte[4096];
char[] chars = new char[4096];
System.Text.Encoding.UTF8.GetDecoder().Convert(bytes, 0, 0, chars, 4096, 0, false, out bytesUsed, out charsUsed, out completed);
Expected:
The code runs, charsUsed equals zero, bytesUsed equals zero and completed is true.
Actual:
Mono throws an ArgumentOutOfRangeException for charIndex.
There are also other behavioral differences between Mono and Microsoft's implementation for the GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) method, such as:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 0, chars, 10, false);
On Mono, this results in an ArgumentOutOfRangeException for charIndex, where for Microsoft charactersWritten = 0.
Similarly:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 4096, chars, 10, false);
On Mono, this results in the same ArgumentOutOfRangeException for charIndex, where Microsoft throws and ArgumentException. since "chars does not have enough capacity from charIndex to the end of the array to accommodate the resulting characters."
UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) eventually calls into UTF8Encoding.InternalGetChars, which does not perform any bounds checking on the passed in char* until after the first byte > 0x80 is encountered. For instance:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes,
0, 4096, chars, 9, false);
will result in no errors and charactersWritten = 4096.
* UTF8Encoding.cs: Use Equals to compare the objects as the
fallback objects do not overload operator ==. The comparison
was previously failing.
svn path=/trunk/mcs/; revision=138786
* *.cs: Convert all tests to new-style nunit classes/methods. Add
[Test] attributes to some tests which were missing them.
svn path=/trunk/mcs/; revision=136944
* UTF8Encoding.cs : surrogate characters are handled in
InternalGetChars() but not in InternalGetCharCount().
Fixed bug #415628.
* UTF8EncodingTest.cs : added test for bug #415628.
* 415628.bin : new test resource.
* Makefile: new resource, new EXTRA_DIST mess.
svn path=/trunk/mcs/; revision=125075
* UTF8EncodingTest.cs : looks like the issue is not fixed in .NET 1.x
and hence it looked like a regression. (It wasn't.)
svn path=/trunk/mcs/; revision=88248
* DecoderReplacementFallbackBuffer.cs : Reset() should also reset the
input buffer. When fallback is not assigned, just return '\0'.
* UnicodeEncoding.cs : handle throwOnInvalid .ctor argument.
Default replacement fallback buffer is now "\uFFFD".
* UTF8Encoding.cs : couple of replacement buffer size fixes.
Default replacement fallback buffer is now "\uFFFD".
* UTF32Encoding.cs : Default replacement is "\uFFFD" too here.
See http://support.microsoft.com/kb/940521/ for this change.
* UnicodeEncodingTest.cs,
UTF8EncodingTest.cs,
DecoderReplacementFallbackBufferTest.cs,
DecoderReplacementFallbackTest.cs : default replacement buffer fix.
Added test for Reset() for replacement buffer.
svn path=/trunk/mcs/; revision=88150
#332515.
* UTF32Encoding.cs: IsBrowserSave returns false for both Big-Endian and
Little-Endian.
* TestEncoding.cs: Also make class available on 1.0 profile.
* EncodingTest.cs: Added tests for Is* properties.
* UTF7EncodingTest.cs: Added tests for Is* properties.
* UnicodeEncodingTest.cs: Added tests for Is* properties. Fixed
line endings.
* UTF8EncodingTest.cs: Added tests for Is* properties. Spaces to
tabs. Numbered tests.
* ASCIIEncodingTest.cs: Added tests for Is* properties. No longer
derive from TestCase class. Spaces to tabs.
* UTF32EncodingTest.cs: Added tests for Is* properties. No longer
derive from deprecated Assertion class.
svn path=/trunk/mcs/; revision=87593
* UTF8Encoding.cs : In zero-charbuffer case, byte buffer count should
be checked. Also it should not ignore leftOver characters even if
byte buffer length is 0.
* UTF8EncodingTest.cs : added test for insufficient bytes for
flush=true and leftOver!='\0' case.
svn path=/trunk/mcs/; revision=56533
* UTF7Encoding.cs UTF8Encoding.cs :
Fixed bug #77315 (Patch by [email protected]).
Make strict check for invalid surrogate.
* UTF7EncodingTest.cs : added test for bug #77315.
* UTF8EncodingTest.cs : added test for bug #77315. Also, now a
bunch of tests are working (while they are marked as NotDotNet
under 1.1 profile).
svn path=/trunk/mcs/; revision=55971
* UTF7EncodingTest.cs UTF8EncodingTest.cs UnicodeEncodingTest.cs :
due to the introduction of fallback, some of their internals have
changed. Thus marked some tests as NotWorking.
* DecoderReplacementFallbackBufferTest.cs,
EncoderReplacementFallbackBufferTest.cs :
some behaviors have changed since beta2, so marked old tests as
NotWorking.
svn path=/trunk/mcs/; revision=55794
* Decoder.cs : added FallbackBuffer property.
* UTF8Encoding.cs : In NET_2_0, use DecoderFallbackBuffer instead of
"throwOnInvalid".
* UTF8EncodingTest.cs : Simply replaced all ArgumentException with
switched alias (it is DecodefFallbackException in NET_2_0).
svn path=/trunk/mcs/; revision=53137
in mcs:
* driver.cs : Use default encoding on the environment.
Removed (now that) extra parameter for SeekableStreamReader.
* support.cs : (SeekableStreamReader) third .ctor() argument for
StreamReader is not required (always true). preamble size could
be acquired in simpler and safe way.
in tests:
* test-444.cs :
UTF-8 BOM autodetect test (for encoding detection patch).
in class/corlib/System.Text:
* UTF8Encoding.cs : (InternalGetChars/InternalGetCharCount):
Don't exclude FEFF in the resulting text.
in class/corlib/Test/System.Text:
* UTF8EncodingTest.cs : added GetCharsFEFF().
in class/Managed.Windows.Forms:
* Makefile : added /codepage:65001 (otherwise the build gets broken
on KeyboardLayout.cs on some non-iso-8859-1 environment).
svn path=/trunk/mcs/; revision=48826
* UTF8EncodingTest.cs: Ajusted test 3.3.6 and 3.3.7 because MS accept
invalid UTF8 (when it shouldn't) and Mono doesn't. Both case are now
accepted by the unit tests.
svn path=/trunk/mcs/; revision=25486
* UTF8EncodingTest.cs: Added 70 new unit tests from UTF-8-test.txt
(available from http://www.cl.cam.ac.uk/~mgk25/). Tests are adapted
so MS implementation pass them all.
svn path=/trunk/mcs/; revision=25476