Commit Graph
300 Commits
Author SHA1 Message Date
Rodrigo Kumpera 0ac61b03fc [corlib] Fix UnicodeEncoding GetMaxByteCount and GetMaxCharCount to return the same values as .net. 2014-10-01 15:11:20 -04:00
Zoltan Varga b533e2abac Fix a warning. 2014-09-28 19:57:52 -04:00
Marcos Henrich f79c64942e Changed overlong byte to be processed again.
The byte is processed again because it can be the begin of a new multi-byte encoded character.
2014-09-02 23:14:09 +08:00
Marcos Henrich 91fa140dd0 Changed UTF8Encoding to call fallback with whole byte sequences.
Removed unnecessary parameters ref byte [] bufferArg.
This changes are required by UTF8EncodingTest.EncodingFallback that in some cases is expecting multibyte values..
2014-09-02 22:57:26 +08:00
Marcos Henrich 09bb2dc4bd Changed InternalGetCharCount to iterate the bytes as InternalGetChars. 2014-09-02 18:38:20 +08:00
Marek Safar 11a3daa71d Merge pull request #1190 from esdrubal/encoder-convert
System.Text.Encoder convert
2014-08-11 10:50:03 +02:00
Marcos Henrich 631f3685bb Added classes EncodingEncoder and EncodingDecoder.
EncodingEncoder and EncodingDecoder override Encoder.Convert.
The Convert overrides allow retrieval of chars/bytes buffers with the same size as .NET.
2014-08-07 09:28:44 +01:00
Marek Safar 8d282b527e [corlib] Fix typo 2014-06-26 17:20:29 +02:00
Martin Potter 1b318701b9 Fixing bugs in UTF8Encoding's GetMaxByteCount and GetMaxCharCount.
The previous implementation of GetMaxByteCount has two bugs: it incorrectly took into account the byte order mark and it was taking the max of a byte count (3) and a character count (EncoderFallback.MaxCharCount). This could result in returning the max number of bytes required to convert a string that was less than the number bytes used when converting the string.

GetMaxCharCount did not previously take into account DecoderFallback.MaxCharCount or the fact that there may still be part of a surrogate pair left on the buffer from a previous call to convert.

I removed the #if NET_2_0 conditionals and NotWorking category since these methods now return results consistent with Microsoft's implementation. I also added two simple tests for custom EncoderFallback and DecoderFallback cases.

This commit is licensed under MIT/X11.
2013-10-13 14:37:45 -07:00
Atsushi Eno f0478f1f51 UTF8Encoding.GetMaxByteCount() was returning too big number.
The thing is, unlike the standards describe in codepoints, .NET String
characters beyond BMP are represented as surrogate pairs and they can be
converted all in 3 bytes.

The implementation didn't take care of encoder fallbacks, or byte order
marks either, so take them into considertion too.
2013-09-14 00:56:48 +09:00
Jeffrey Stedfast 51b648a118 Fixed build breakage from my previous commit 2013-08-12 12:12:47 -04:00
Jeffrey Stedfast b1d7375726 Fixed System.Text.Decoder.GetChars(byte*,int,char*,int,bool) to copy output into the char* buffer. 2013-08-12 08:22:49 -04:00
Atsushi Eno 132a16ca07 Revert "Merge pull request #622 from killabytenow/master"
This reverts commit 392bae3c45, reversing
changes made to 61cd9ee8b6.
2013-07-13 23:56:59 +09:00
Gerardo García Peña 22892bc401 Better and more homogeneous error handling. 2013-07-11 23:49:47 +02:00
Gerardo García Peña 54f3f6ae21 If 'flush' flag is enabled when calling decoding/encoding function, flush buffer when input run out (satisfies test 'SufficientByteArray'). 2013-07-11 18:54:11 +02:00
Gerardo García Peña 72ed65fdd8 Added counter mode in InternalGetChars.
Added methods InternalGetCharsCount and InternalGetCharsEncode for improving readability.
Accept zero-length input buffer in 'InternalGetBytesDecode/Count' for processing void char buffers.
Removed a redundant 'unsafe' keyword.
2013-05-23 17:12:02 +02:00
Gerardo García Peña bf9b74f05a bugfix: accept zero-length input buffer in 'InternalGetChars' for building void strings. 2013-05-23 15:57:38 +02:00
Gerardo García Peña c22ce0e053 Reverted all changes in 'System.Text/Encoding.cs' and disabled the emitIdentifier feature in the new 'System.Text/UTF8Encoding.cs'. 2013-05-23 15:43:27 +02:00
Gerardo García Peña 34aa103a3c Added counter mode in InternalGetBytes.
Added methods InternalGetBytesCount and InternalGetBytesEncode for improving readability.
2013-05-23 15:39:12 +02:00
Gerardo García Peña ade6a99d9e By default return an 'UTF8Unmarked' object. Removed unused 'utf8EncodingWithMarkers' attribute. 2013-05-23 12:53:15 +02:00
Gerardo García Peña ce246f6a5b - improved flush handling in encoder/decoder for fixing regression bug #565129
- fixed some checks to satisfy test 'ConvertZeroCharacters'.
2013-05-22 18:53:13 +02:00
Gerardo García Peña 062dfd03eb Accept a null input/output buffer in InternalGetBytes()/InternalGetChars() if its length is zero.
Throw ArgumentException in GetBytes()/GetChars() if no enough output buffer space.
2013-05-21 15:51:18 +02:00
Gerardo García Peña b87ca2196c bugfixed returned charsUsed counter 2013-05-21 13:41:13 +02:00
Gerardo García Peña 4c7bcaaca3 some minor code style changes 2013-04-29 08:30:55 +02:00
Gerardo García Peña c9e3b37bb8 Disable BOM preamble in UTF8 by default (following best practices). 2013-04-29 08:09:23 +02:00
Gerardo García Peña a93fa79df7 Rewrite of core coder & decoder functions to fix several bugs and limitations, and improve error handling. 2013-04-25 07:58:49 +02:00
Gerardo García Peña 1eb5cd8654 removed index check because (1) it cannot be checked in this context and (2) it
only explains when a bad sequence started in the current or past buffer (in
this case it would take a negative value).
2013-04-12 00:12:29 +02:00
Gerardo García Peña 026a80eea2 removed index check because (1) it cannot be checked in this context and (2) it
only explains when a bad sequence started in the current or past buffer (in
this case it would take a negative value).
2013-04-12 00:06:16 +02:00
Gerardo García Peña e62924259a Implemented method 'IsUnknownSurrogate()'. 2013-04-03 17:26:14 +02:00
Gerardo García Peña 8f6a053b80 Refactorized Convert(char[], int, ...) to use the pointer version. 2013-04-03 17:24:35 +02:00
Gerardo García Peña a0a12c128d Refactorized convert(byte[], int, ...), now uses the pointer version. 2013-04-03 17:21:58 +02:00
Rodrigo Kumpera abe9dc6206 Revert "Fix object::GetType when remoting is enabled."
This reverts commit 93e2d1dc48.
2013-03-01 10:54:09 -05:00
Rodrigo Kumpera 93e2d1dc48 Fix object::GetType when remoting is enabled. 2013-03-01 10:52:03 -05:00
Martin Potter 643d8b8b3a System.Text.Decoder throws ArgumentOutOfRangeException when it should not. Fixes Bug #10789
Similar to previously reported bug #6404, according to the documentation at http://msdn.microsoft.com/en-us/library/h6w985hz.aspx, System.Text.Encoder throws ArgumentOutOfRangeExceptions when it should not. From the documentation, an ArgumentOutOfRangeException should be thrown when:

charIndex, charCount, byteIndex, or byteCount is less than zero.
-or-
The length of chars - charIndex is less than charCount.
-or-
The length of bytes - byteIndex is less than byteCount.

To reproduce, run the following code:

int charsUsed, bytesUsed;
bool completed;
byte[] bytes = new byte[4096];
char[] chars = new char[4096];
System.Text.Encoding.UTF8.GetDecoder().Convert(bytes, 0, 0, chars, 4096, 0, false, out bytesUsed, out charsUsed, out completed);

Expected:

The code runs, charsUsed equals zero, bytesUsed equals zero and completed is true.

Actual:

Mono throws an ArgumentOutOfRangeException for charIndex.

There are also other behavioral differences between Mono and Microsoft's implementation for the GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) method, such as:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 0, chars, 10, false);

On Mono, this results in an ArgumentOutOfRangeException for charIndex, where for Microsoft charactersWritten = 0.

Similarly:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 4096, chars, 10, false);

On Mono, this results in the same ArgumentOutOfRangeException for charIndex, where Microsoft throws and ArgumentException. since "chars does not have enough capacity from charIndex to the end of the array to accommodate the resulting characters."
2013-03-01 05:05:34 -08:00
Martin Potter 7ca2aa01d7 UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) can write past end of char[]. Fixes Bug#10788
UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) eventually calls into UTF8Encoding.InternalGetChars, which does not perform any bounds checking on the passed in char* until after the first byte > 0x80 is encountered. For instance:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes,
0, 4096, chars, 9, false);

will result in no errors and charactersWritten = 4096.
2013-02-28 16:26:02 -08:00
Rodrigo Kumpera 1b92a4148c Since MOBILE implies NET_4_* now, take this into account and simplify our ifdefs. 2013-02-11 18:42:18 -05:00
Rodrigo Kumpera b96d05dac3 Kill the MOONLIGHT define in corlib. 2013-02-11 16:03:28 -05:00
Atsushi Eno c1fe3fb643 fix mobile build. 2013-01-29 18:25:09 +09:00
Miguel de Icaza 505a1e2bae [mscorlib/mobile] Encoding.Default is always UTF8 in mobile, strip out all the code that brings the encoding system 2013-01-28 20:48:59 -05:00
Martin Baulig cd8ca2f658 [corlib] Fix ASCIIEncoding with custom DecoderFallback.
Only pass one single byte to DecoderFallback.Fallback(),
throw if there's not enough space in the char array.
2012-12-22 05:14:25 +01:00
Marek Safar 7883a2dafd Clean up some static initializers 2012-10-03 12:19:55 +01:00
Martin Potter e3d59a330e Updating System.Text.Encoder's Convert method bounds checking to be compliant with Microsoft's documentation. (http://msdn.microsoft.com/en-us/library/dh0797c4.aspx) 2012-08-05 21:18:33 -07:00
Marek Safar c45375b4c8 Fix types signature 2012-07-27 12:25:43 +01:00
Marek Safar f6edbba2d1 Cache empty byte arrays 2012-06-22 18:55:51 +01:00
Marek Safar 5b82f0e08c Clean up some NET_2_0 2012-06-14 15:44:35 +01:00
Rodrigo Kumpera 83dccf8fe7 Add all missing sequential layout directives to corlib and system. 2011-12-13 11:57:23 -02:00
Miguel de Icaza 083d1cfe2c Updated with Xamarin copyrights 2011-11-23 22:43:03 -05:00
Paolo Molaro 8397d2885e Fixed StringBuilder.Clear() to properly clear the cached string. 2011-11-22 12:04:05 +01:00
Miguel de Icaza 2852ed0b59 Update copyrights 2011-09-13 18:26:13 -04:00
Atsushi Eno 5b18ed7d41 EncoderReplacementFallbackBuffer.Reset() did not really reset its internals. 2011-08-31 13:59:56 -07:00