Commit Graph
87 Commits
Author SHA1 Message Date
Zoltan Varga b533e2abac Fix a warning. 2014-09-28 19:57:52 -04:00
Marcos Henrich f79c64942e Changed overlong byte to be processed again.
The byte is processed again because it can be the begin of a new multi-byte encoded character.
2014-09-02 23:14:09 +08:00
Marcos Henrich 91fa140dd0 Changed UTF8Encoding to call fallback with whole byte sequences.
Removed unnecessary parameters ref byte [] bufferArg.
This changes are required by UTF8EncodingTest.EncodingFallback that in some cases is expecting multibyte values..
2014-09-02 22:57:26 +08:00
Marcos Henrich 09bb2dc4bd Changed InternalGetCharCount to iterate the bytes as InternalGetChars. 2014-09-02 18:38:20 +08:00
Marcos Henrich 631f3685bb Added classes EncodingEncoder and EncodingDecoder.
EncodingEncoder and EncodingDecoder override Encoder.Convert.
The Convert overrides allow retrieval of chars/bytes buffers with the same size as .NET.
2014-08-07 09:28:44 +01:00
Martin Potter 1b318701b9 Fixing bugs in UTF8Encoding's GetMaxByteCount and GetMaxCharCount.
The previous implementation of GetMaxByteCount has two bugs: it incorrectly took into account the byte order mark and it was taking the max of a byte count (3) and a character count (EncoderFallback.MaxCharCount). This could result in returning the max number of bytes required to convert a string that was less than the number bytes used when converting the string.

GetMaxCharCount did not previously take into account DecoderFallback.MaxCharCount or the fact that there may still be part of a surrogate pair left on the buffer from a previous call to convert.

I removed the #if NET_2_0 conditionals and NotWorking category since these methods now return results consistent with Microsoft's implementation. I also added two simple tests for custom EncoderFallback and DecoderFallback cases.

This commit is licensed under MIT/X11.
2013-10-13 14:37:45 -07:00
Atsushi Eno f0478f1f51 UTF8Encoding.GetMaxByteCount() was returning too big number.
The thing is, unlike the standards describe in codepoints, .NET String
characters beyond BMP are represented as surrogate pairs and they can be
converted all in 3 bytes.

The implementation didn't take care of encoder fallbacks, or byte order
marks either, so take them into considertion too.
2013-09-14 00:56:48 +09:00
Atsushi Eno 132a16ca07 Revert "Merge pull request #622 from killabytenow/master"
This reverts commit 392bae3c45, reversing
changes made to 61cd9ee8b6.
2013-07-13 23:56:59 +09:00
Gerardo García Peña 22892bc401 Better and more homogeneous error handling. 2013-07-11 23:49:47 +02:00
Gerardo García Peña 54f3f6ae21 If 'flush' flag is enabled when calling decoding/encoding function, flush buffer when input run out (satisfies test 'SufficientByteArray'). 2013-07-11 18:54:11 +02:00
Gerardo García Peña 72ed65fdd8 Added counter mode in InternalGetChars.
Added methods InternalGetCharsCount and InternalGetCharsEncode for improving readability.
Accept zero-length input buffer in 'InternalGetBytesDecode/Count' for processing void char buffers.
Removed a redundant 'unsafe' keyword.
2013-05-23 17:12:02 +02:00
Gerardo García Peña bf9b74f05a bugfix: accept zero-length input buffer in 'InternalGetChars' for building void strings. 2013-05-23 15:57:38 +02:00
Gerardo García Peña c22ce0e053 Reverted all changes in 'System.Text/Encoding.cs' and disabled the emitIdentifier feature in the new 'System.Text/UTF8Encoding.cs'. 2013-05-23 15:43:27 +02:00
Gerardo García Peña 34aa103a3c Added counter mode in InternalGetBytes.
Added methods InternalGetBytesCount and InternalGetBytesEncode for improving readability.
2013-05-23 15:39:12 +02:00
Gerardo García Peña ce246f6a5b - improved flush handling in encoder/decoder for fixing regression bug #565129
- fixed some checks to satisfy test 'ConvertZeroCharacters'.
2013-05-22 18:53:13 +02:00
Gerardo García Peña 062dfd03eb Accept a null input/output buffer in InternalGetBytes()/InternalGetChars() if its length is zero.
Throw ArgumentException in GetBytes()/GetChars() if no enough output buffer space.
2013-05-21 15:51:18 +02:00
Gerardo García Peña b87ca2196c bugfixed returned charsUsed counter 2013-05-21 13:41:13 +02:00
Gerardo García Peña 4c7bcaaca3 some minor code style changes 2013-04-29 08:30:55 +02:00
Gerardo García Peña a93fa79df7 Rewrite of core coder & decoder functions to fix several bugs and limitations, and improve error handling. 2013-04-25 07:58:49 +02:00
Rodrigo Kumpera abe9dc6206 Revert "Fix object::GetType when remoting is enabled."
This reverts commit 93e2d1dc48.
2013-03-01 10:54:09 -05:00
Rodrigo Kumpera 93e2d1dc48 Fix object::GetType when remoting is enabled. 2013-03-01 10:52:03 -05:00
Martin Potter 643d8b8b3a System.Text.Decoder throws ArgumentOutOfRangeException when it should not. Fixes Bug #10789
Similar to previously reported bug #6404, according to the documentation at http://msdn.microsoft.com/en-us/library/h6w985hz.aspx, System.Text.Encoder throws ArgumentOutOfRangeExceptions when it should not. From the documentation, an ArgumentOutOfRangeException should be thrown when:

charIndex, charCount, byteIndex, or byteCount is less than zero.
-or-
The length of chars - charIndex is less than charCount.
-or-
The length of bytes - byteIndex is less than byteCount.

To reproduce, run the following code:

int charsUsed, bytesUsed;
bool completed;
byte[] bytes = new byte[4096];
char[] chars = new char[4096];
System.Text.Encoding.UTF8.GetDecoder().Convert(bytes, 0, 0, chars, 4096, 0, false, out bytesUsed, out charsUsed, out completed);

Expected:

The code runs, charsUsed equals zero, bytesUsed equals zero and completed is true.

Actual:

Mono throws an ArgumentOutOfRangeException for charIndex.

There are also other behavioral differences between Mono and Microsoft's implementation for the GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) method, such as:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 0, chars, 10, false);

On Mono, this results in an ArgumentOutOfRangeException for charIndex, where for Microsoft charactersWritten = 0.

Similarly:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes, 0, 4096, chars, 10, false);

On Mono, this results in the same ArgumentOutOfRangeException for charIndex, where Microsoft throws and ArgumentException. since "chars does not have enough capacity from charIndex to the end of the array to accommodate the resulting characters."
2013-03-01 05:05:34 -08:00
Martin Potter 7ca2aa01d7 UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) can write past end of char[]. Fixes Bug#10788
UTF8Decoder.GetChars(Byte[], Int32, Int32, Char[], Int32, Boolean) eventually calls into UTF8Encoding.InternalGetChars, which does not perform any bounds checking on the passed in char* until after the first byte > 0x80 is encountered. For instance:

byte[] bytes = new byte[4096];
char[] chars = new char[10];
int charactersWritten = System.Text.Encoding.UTF8.GetDecoder().GetChars(bytes,
0, 4096, chars, 9, false);

will result in no errors and charactersWritten = 4096.
2013-02-28 16:26:02 -08:00
Marek Safar 7883a2dafd Clean up some static initializers 2012-10-03 12:19:55 +01:00
Marek Safar f6edbba2d1 Cache empty byte arrays 2012-06-22 18:55:51 +01:00
Marek Safar 5b82f0e08c Clean up some NET_2_0 2012-06-14 15:44:35 +01:00
Atsushi Eno f4aff310cf EncoderFallback support in UTF8Encoding. Fixed bug #565129 and regression in d961246.
Patch mostly by JB Evain.
2010-08-10 20:22:52 +09:00
Marek Habersack 7e0a0a69f0 Revert "EncoderFallback support in UTF8Encoding. Fixed bug #565129."
This reverts commit d961246014 because it breaks 2 tests in
System.Web on 2.0 and 4 on 4.0 profile.
2010-08-05 15:32:58 +02:00
atsushieno d961246014 EncoderFallback support in UTF8Encoding. Fixed bug #565129.
Patch mostly by JB Evain.
2010-08-04 15:32:54 +09:00
Marek Safar 5d6e605db6 2009-11-24 Marek Safar <[email protected]>
* UTF8Encoding.cs (GetPreamble): Let compiler optimize it.


svn path=/trunk/mcs/; revision=146838
2009-11-24 16:08:59 +00:00
Miguel de Icaza 53d746c7a1 2009-11-02 Miguel de Icaza <[email protected]>
* Jumbo patch to drop support for pre-NET_2_0 code:

        Remove NET_2_0 defines assuming the value is true.


svn path=/trunk/mcs/; revision=145232
2009-11-03 00:27:28 +00:00
Miguel de Icaza e13c611802 2009-07-28 Miguel de Icaza <[email protected]>
* UTF8Encoding.cs: Small optimization, reuse the static
	EncoderFallback and DecoderFallback instead of creating new ones. 

svn path=/trunk/mcs/; revision=138787
2009-07-28 05:06:31 +00:00
Miguel de Icaza 9afbbceb72 2009-07-28 Miguel de Icaza <[email protected]>
* UTF8Encoding.cs: Use Equals to compare the objects as the
	fallback objects do not overload operator ==.    The comparison
	was previously failing.

svn path=/trunk/mcs/; revision=138786
2009-07-28 05:04:08 +00:00
Atsushi Eno 3ef06e62ed 2009-01-30 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : surrogate characters are handled in 
	  InternalGetChars() but not in InternalGetCharCount().
	  Fixed bug #415628.

	* UTF8EncodingTest.cs : added test for bug #415628.

	* 415628.bin : new test resource.

	* Makefile: new resource, new EXTRA_DIST mess.


svn path=/trunk/mcs/; revision=125075
2009-01-30 12:13:56 +00:00
Paolo Molaro 3555435166 Mon Oct 6 09:46:09 CEST 2008 Paolo Molaro <[email protected]>
* UTF8Encoding.cs: rewritten InternalGetByteCount () and
	InternalGetBytes (): among other things this versions are
	10% to 60% faster, depending on input type and size.


svn path=/trunk/mcs/; revision=114925
2008-10-06 07:50:20 +00:00
Marek Safar cc9c53a109 Warnings cleanup
svn path=/trunk/mcs/; revision=111713
2008-08-27 15:40:13 +00:00
Andreas N da7510bf08 2008-07-03 Andreas Nahr <[email protected]>
* UTF8Encoding.cs: Fix parameter names, Remove unfounded TODO

svn path=/trunk/mcs/; revision=107134
2008-07-03 08:16:05 +00:00
Andreas N 5122e976a5 2008-07-03 Andreas Nahr <[email protected]>
* UTF8Encoding.cs:
	* UTF7Encoding.cs:
	* UTF32Encoding.cs:
	* EncodingInfo.cs:
	* Encoding.cs:
	* Encoder.cs:
	* ASCIIEncoding.cs: Fix parameter names

svn path=/trunk/mcs/; revision=107125
2008-07-02 23:55:44 +00:00
Atsushi Eno f4071fef2b 2007-10-25 Atsushi Enomoto <[email protected]>
* DecoderReplacementFallbackBuffer.cs : Reset() should also reset the
	  input buffer. When fallback is not assigned, just return '\0'.
	* UnicodeEncoding.cs : handle throwOnInvalid .ctor argument.
	  Default replacement fallback buffer is now "\uFFFD".
	* UTF8Encoding.cs : couple of replacement buffer size fixes.
	  Default replacement fallback buffer is now "\uFFFD".
	* UTF32Encoding.cs : Default replacement is "\uFFFD" too here.
	  See http://support.microsoft.com/kb/940521/ for this change.

	* UnicodeEncodingTest.cs,
	  UTF8EncodingTest.cs,
	  DecoderReplacementFallbackBufferTest.cs,
	  DecoderReplacementFallbackTest.cs : default replacement buffer fix.
	  Added test for Reset() for replacement buffer.


svn path=/trunk/mcs/; revision=88150
2007-10-25 09:42:03 +00:00
Gert Driesen 797ad860d5 * UTF8Encoding.cs: IsMailNewsSave should return true. Fixes bug
#332515.
* UTF32Encoding.cs: IsBrowserSave returns false for both Big-Endian and
Little-Endian.
* TestEncoding.cs: Also make class available on 1.0 profile.
* EncodingTest.cs: Added tests for Is* properties.
* UTF7EncodingTest.cs: Added tests for Is* properties.
* UnicodeEncodingTest.cs: Added tests for Is* properties. Fixed 
line endings.
* UTF8EncodingTest.cs: Added tests for Is* properties. Spaces to
tabs. Numbered tests.
* ASCIIEncodingTest.cs: Added tests for Is* properties. No longer
derive from TestCase class. Spaces to tabs.    
* UTF32EncodingTest.cs: Added tests for Is* properties. No longer
derive from deprecated Assertion class.

svn path=/trunk/mcs/; revision=87593
2007-10-16 14:48:47 +00:00
Miguel de Icaza 8a1af5fd38 2006-11-16 Miguel de Icaza <[email protected]>
* DirectoryInfo.cs (GetFiles): Implement option with SearchOptions
	== AllDirectories

And a lot of MonoTODOs updated as well


svn path=/trunk/mcs/; revision=67998
2006-11-16 14:14:01 +00:00
Gert Driesen b136e2077e * ASCIIEncoding.cs: Marked class ComVisible (true) on 2.0 profile.
Marked 2.0-only members as ComVisible (false).
* Decoder.cs: Marked class ComVisible (true) on 2.0 profile. Marked
2.0-only members as ComVisible (false).
* Encoder.cs: Marked class ComVisible (true) on 2.0 profile. Marked
2.0-only members as ComVisible (false).
* Encoding.cs: Marked class ComVisible (true) on 2.0 profile. Marked
2.0-only members as ComVisible (false).
* NormalizationForm.cs: Marked ComVisible (true).
* StringBuilder.cs: Marked class ComVisible (true) on 2.0 profile.
Marked 2.0-only methods as ComVisible (false).
* UnicodeEncoding.cs: GetString (byte[], int, int) is only available
in 2.0 profile. Marked 2.0-only methods ComVisible (false). Added
missing ctor in 2.0 profile, and marked in MonoTODO.
* UTF7Encoding.cs: Marked class ComVisible (true) on 2.0 profile.
* UTF8Encoding.cs: Marked 2.0-only methods as ComVisible (false).

svn path=/trunk/mcs/; revision=61250
2006-05-30 09:02:37 +00:00
Atsushi Eno abcea4f6f0 2006-03-24 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : made internal implementation as pointer-based,
	  and added pointer-based converter method overloads.


svn path=/trunk/mcs/; revision=58408
2006-03-24 11:20:24 +00:00
Atsushi Eno ce4d8a502d 2006-02-14 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : Fallback was indicating incorrect index.
	  Fixed bug #77550.

	* UTF8EncodingTest.cs : test for bug #77550.


svn path=/trunk/mcs/; revision=56848
2006-02-14 10:31:51 +00:00
Atsushi Eno d6d80fd7c1 2006-02-03 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : avoid possible overflow.


svn path=/trunk/mcs/; revision=56535
2006-02-03 17:05:10 +00:00
Atsushi Eno e85a1fe77c 2006-02-03 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : In zero-charbuffer case, byte buffer count should
	  be checked. Also it should not ignore leftOver characters even if
	  byte buffer length is 0.

	* UTF8EncodingTest.cs : added test for insufficient bytes for
	  flush=true and leftOver!='\0' case.


svn path=/trunk/mcs/; revision=56533
2006-02-03 16:54:56 +00:00
Atsushi Eno 1d4633d57d 2006-02-03 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : Fast path optimization for InternalGetByteCount()
	  and InternalGetBytes().


svn path=/trunk/mcs/; revision=56518
2006-02-03 11:59:07 +00:00
Atsushi Eno 90f44f360a 2006-02-02 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs :
	  actually leftover bits needs more careful handleding. So it's
	  better to be rather close to the original GetBytes().
	  Also changed leftOver handling so that it will be able to support
	  fallback.


svn path=/trunk/mcs/; revision=56516
2006-02-03 10:37:42 +00:00
Atsushi Eno c9bb7c1bc1 2006-02-02 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs :
	  GetBytes(string, ...) could reuse InternalGetBytes() using fixed
	  pointers. However, InternalGetBytes() was slower than
	  GetBytes(string, ...), so first replaced existing InternalGetBytes()
	  with GetBytes() internals, and GetBytes(string, ...) switched to 
	  InternalGetBytes(). Now GetBytes() implementation code is reduced
	  to one method.


svn path=/trunk/mcs/; revision=56509
2006-02-03 07:55:02 +00:00
Atsushi Eno da8fc189ae 2006-02-02 Atsushi Enomoto <[email protected]>
* UTF8Encoding.cs : fast path optimization was pretty insufficient.
	  Now it handles the entire bytes, not just half of them.


svn path=/trunk/mcs/; revision=56449
2006-02-02 11:16:33 +00:00