Removed unnecessary parameters ref byte [] bufferArg.
This changes are required by UTF8EncodingTest.EncodingFallback that in some cases is expecting multibyte values..
The previous implementation of GetMaxByteCount has two bugs: it incorrectly took into account the byte order mark and it was taking the max of a byte count (3) and a character count (EncoderFallback.MaxCharCount). This could result in returning the max number of bytes required to convert a string that was less than the number bytes used when converting the string.
GetMaxCharCount did not previously take into account DecoderFallback.MaxCharCount or the fact that there may still be part of a surrogate pair left on the buffer from a previous call to convert.
I removed the #if NET_2_0 conditionals and NotWorking category since these methods now return results consistent with Microsoft's implementation. I also added two simple tests for custom EncoderFallback and DecoderFallback cases.
This commit is licensed under MIT/X11.
The thing is, unlike the standards describe in codepoints, .NET String
characters beyond BMP are represented as surrogate pairs and they can be
converted all in 3 bytes.
The implementation didn't take care of encoder fallbacks, or byte order
marks either, so take them into considertion too.
Added methods InternalGetCharsCount and InternalGetCharsEncode for improving readability.
Accept zero-length input buffer in 'InternalGetBytesDecode/Count' for processing void char buffers.
Removed a redundant 'unsafe' keyword.
Similar to previously reported bug #6404, according to the documentation at http://msdn.microsoft.com/en-us/library/h6w985hz.aspx, System.Text.Encoder throws ArgumentOutOfRangeExceptions when it should not. From the documentation, an ArgumentOutOfRangeException should be thrown when:
charIndex, charCount, byteIndex, or byteCount is less than zero.
-or-
The length of chars - charIndex is less than charCount.
-or-
The length of bytes - byteIndex is less than byteCount.
To reproduce, run the following code:
int charsUsed, bytesUsed;
bool completed;
byte[] bytes = new byte[4096];
char[] chars = new char[4096];
System.Text.Encoding.UTF8.GetDecoder().Convert(bytes, 0, 0, chars, 4096, 0, false, out bytesUsed, out charsUsed, out completed);
Expected:
The code runs, charsUsed equals zero, bytesUsed equals zero and completed is true.
Actual:
Mono throws an ArgumentOutOfRangeException for charIndex.
There are also other behavioral differences between Mono and Microsoft's implementation for the GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) method, such as:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 0, chars, 10, false);
On Mono, this results in an ArgumentOutOfRangeException for charIndex, where for Microsoft charactersWritten = 0.
Similarly:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 4096, chars, 10, false);
On Mono, this results in the same ArgumentOutOfRangeException for charIndex, where Microsoft throws and ArgumentException. since "chars does not have enough capacity from charIndex to the end of the array to accommodate the resulting characters."
UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) eventually calls into UTF8Encoding.InternalGetChars, which does not perform any bounds checking on the passed in char* until after the first byte > 0x80 is encountered. For instance:
byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes,
0, 4096, chars, 9, false);
will result in no errors and charactersWritten = 4096.