mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
esengine
f28d4f3c94
edit_file on a UTF-8 file carrying CJK text rewrote bytes it never touched:
continuation bytes of characters far from the edit came back as 84 31 A4 37.
Detect took a nil error from the GB18030 decoder as proof the file was
GB18030. The decoder never fails: it maps every invalid sequence to U+FFFD.
So any file that was not valid UTF-8 -- one stray byte is enough -- was read
as GB18030, and the read-modify-write tools (edit_file, multi_edit,
delete_range, write_file over an existing file), checkpoint restore and the
config migrations then re-encoded every unreadable sequence as the GB18030
form of U+FFFD.
Detect now names a charset only when decoding and re-encoding restores the
input byte for byte. GB18030 is tried first, then GBK, which restores what
GB18030 cannot such as CP936's single-byte euro 0x80, so such a file stays
editable in its own encoding; anything else is LossyUTF8, whose bytes pass
through untouched. The new Kind is appended because checkpoints persist it
by value.
GBK cannot hold every character, so Encode now returns an error instead of
falling back to UTF-8, which rewrote a whole CP936 file as UTF-8 when an edit
added a character such as an emoji. A legacy charset that cannot represent a
character answers an *UnencodableError naming it, its byte offset and the
charset, with ErrUnencodable as its identity; the file tools, checkpoint
restore and the config migration refuse the write and leave the file as it
was. Fixtures encode through encodingtest.MustEncode.
Readers that hold only part of a stream say so rather than inherit the
whole-file answer:
- DetectFragment runs the round trip on the prefix the decoder consumes when
not at EOF, so a character the cut split is dropped structurally. grep's
8 KiB peek and read_file's 256 KiB sample use it.
- DecodeOutput is the one entry point for bytes that are only displayed.
The producer says which ends its bound cut (Cut{Head, Tail}); UTF-8 is read
without a character split at a cut end, and an uncut end keeps its bytes,
since a GBK pair there is output rather than half a rune. What no charset
restores is still read as GB18030. Hook output (tail cut when truncated),
the shell tail (head cut once it drops bytes) and the shell's combined
output (uncut) call only it.
- DetectAndDecode decodes a legacy charset once; the edit tools and
DecodeToUTF8 use it instead of detecting and decoding separately.
Tests that fail before this change (on the base or an earlier revision of it)
and pass here:
- TestE2EEditKeepsBytesOfUTF8FileWithStrayByte: e4 b8 ad became
e4 b8 84 31 a4 37 outside the edit (fails on origin/studio).
- TestDetectUTF8WithStrayByteIsNotGB18030 (fails on origin/studio).
- TestWritesRefuseCharacterCP936CannotHold (edit_file, multi_edit,
write_file): nil error and the whole CP936 file rewritten as UTF-8.
- TestDecodeOutputTrimsCutUTF8Edges: UTF-8 cut at the front read as mojibake.
- TestDecodeOutputKeepsUncutCodePageEdges, TestDecodeHookOutputKeepsUncutCodePageEdges:
GBK pairs at an uncut edge (bc db, b0 a1 0d 0a, 6f 6b 20 e4 a1,
b0 a1 20 6f 6b) were dropped as partial UTF-8 runes.
- TestEditKeepsCP936FileWithEuroByte: an edit adding Chinese wrote UTF-8
into a CP936 file.
- TestGrepGB18030PeekWithoutNewline, TestGrepGB18030PeekCutMidCharacter:
no matches past a peek cut mid-character.
- TestReadFileGB18030SampleWithoutNewline: garbled text past the sample.
- TestDecodeHookOutputReadsTruncatedCodePageText,
TestDecodeHookOutputReadsCP936Euro, TestCodePageOutputCutMidCharacterIsReadable:
mojibake for cut or euro-carrying cmd.exe output.
- TestEncodeRefusesCharacterTheCharsetCannotHold, TestDetectGBKWithEuroByte,
TestDetectFragmentDropsTheSplitCharacter, TestDecodeOutputReadsCutCodePageText:
the API's contract.