anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 10:15:31 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

CJK characters break some features like @ mentions #2470

New Issue
Open
opened 2026-02-16 17:35:48 -05:00 by yindo · 9 comments
yindo commented 2026-02-16 17:35:48 -05:00
Owner
Copy Link
Copy Source

Originally created by @yukukotani on GitHub (Nov 2, 2025).

Originally assigned to: @kommander on GitHub.

Description

Expects that inputting 世界 @ shows auto-completion of @ mentions, but actually shows nothing.

The problem is here.

https://github.com/sst/opencode/blob/89492b30026e9b635b4b7e57877cbe7c8821b8b7/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L374-L376

props.input().visualCursor.offset is based on Unicode codepoints and counts 世界 @ as 5, while props.value.at counts it as 3. Also props.input().logicalCursor.offset returns 5.

Looking more closely at input, it seems expected behaviour in the test.

https://github.com/sst/opentui/blob/54365ebca04b8e556338d417d33a8d042dcc3169/packages/core/src/editor-view.test.ts#L326-L355

I'm not sure what logical means here, but maybe it should returns 3? Or to align JavaScript side to zig implementation, Bun.stringWidth looks similar but has some difference like \n proceed the cursor by 1 but Bun.stringWidth returns 0.

OpenCode version

v1.0.10

Steps to reproduce

input @ after CJK characters, like 世界 @

Screenshot and/or share link

No response

Operating System

No response

Terminal

No response

Originally created by @yukukotani on GitHub (Nov 2, 2025). Originally assigned to: @kommander on GitHub. ### Description Expects that inputting `世界 @` shows auto-completion of `@` mentions, but actually shows nothing. The problem is here. https://github.com/sst/opencode/blob/89492b30026e9b635b4b7e57877cbe7c8821b8b7/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L374-L376 `props.input().visualCursor.offset` is based on Unicode codepoints and counts `世界 @` as 5, while `props.value.at` counts it as 3. Also `props.input().logicalCursor.offset` returns 5. Looking more closely at input, it seems expected behaviour in the test. https://github.com/sst/opentui/blob/54365ebca04b8e556338d417d33a8d042dcc3169/packages/core/src/editor-view.test.ts#L326-L355 I'm not sure what *logical* means here, but maybe it should returns 3? Or to align JavaScript side to zig implementation, [Bun.stringWidth](https://bun.com/docs/runtime/utils#bun-stringwidth) looks similar but has some difference like `\n` proceed the cursor by 1 but Bun.stringWidth returns 0. ### OpenCode version v1.0.10 ### Steps to reproduce input `@` after CJK characters, like `世界 @` ### Screenshot and/or share link _No response_ ### Operating System _No response_ ### Terminal _No response_
yindo added the opentuibug labels 2026-02-16 17:35:48 -05:00
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Nov 2, 2025):

This issue might be a duplicate of existing issues. Please check:

  • #2920: TUI editor hides some double-byte characters in the input field (very similar CJK display issue in TUI)
  • #3493: Korean (Hangul) characters display as broken/garbled in input field (similar CJK character rendering problems)
  • #2904: Icons/Unicode characters cause file editing failures (broader Unicode handling issues throughout the system)

All of these issues seem to stem from similar underlying problems with Unicode/CJK character handling in the TUI components. The root cause appears to be inconsistent character width calculation between Unicode codepoints and display width, which is exactly what you've identified in the autocomplete.tsx file.

Feel free to ignore if none of these address your specific case with @ mention autocomplete functionality.

@github-actions[bot] commented on GitHub (Nov 2, 2025): This issue might be a duplicate of existing issues. Please check: - #2920: TUI editor hides some double-byte characters in the input field (very similar CJK display issue in TUI) - #3493: Korean (Hangul) characters display as broken/garbled in input field (similar CJK character rendering problems) - #2904: Icons/Unicode characters cause file editing failures (broader Unicode handling issues throughout the system) All of these issues seem to stem from similar underlying problems with Unicode/CJK character handling in the TUI components. The root cause appears to be inconsistent character width calculation between Unicode codepoints and display width, which is exactly what you've identified in the autocomplete.tsx file. Feel free to ignore if none of these address your specific case with @ mention autocomplete functionality.
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@yukukotani commented on GitHub (Nov 2, 2025):

@kommander FYI since I assume you are working on this

@yukukotani commented on GitHub (Nov 2, 2025): @kommander FYI since I assume you are working on this
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@yukukotani commented on GitHub (Nov 2, 2025):

I can take this on if I get a bit of guidance on the right approach.

@yukukotani commented on GitHub (Nov 2, 2025): I can take this on if I get a bit of guidance on the right approach.
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@kommander commented on GitHub (Nov 2, 2025):

That is a tricky one. The 3 is in "number of utf16 characters", while the 5 is in "in number of terminal cells", so visualCursor.offset is actually number of cells. Currently the underlying EditBuffer in zig uses "number of cells occupied" as main coordinate system for the cursor. Which is bad.

However, there are methods to convert to a byte offset. So I think opentui needs to expose some methods to work with the Editbuffer properly.

String.at is not grapheme aware, it coincidentally works in this specific case, but for combining characters like acutes or devanagari viramas it would treat the combining chars as separate chars.

For example 世 is U+4E16 in three utf8 bytes.

Edit: I'll follow up on this.

@kommander commented on GitHub (Nov 2, 2025): That is a tricky one. The 3 is in "number of utf16 characters", while the 5 is in "in number of terminal cells", so `visualCursor.offset` is actually number of cells. Currently the underlying `EditBuffer` in zig uses "number of cells occupied" as main coordinate system for the cursor. Which is bad. However, there are methods to convert to a byte offset. So I think opentui needs to expose some methods to work with the Editbuffer properly. `String.at` is not grapheme aware, it coincidentally works in this specific case, but for combining characters like acutes or devanagari viramas it would treat the combining chars as separate chars. For example 世 is U+4E16 in three utf8 bytes. Edit: I'll follow up on this.
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@kommander commented on GitHub (Nov 2, 2025):

I think the simplest solution to make this work properly, without going into the huge refactor that is needed for the text handling in opentui, is exposing a getTextRange method for the TextBuffer in opentui, so opencode can use that instead of String.at. Then charBeforeCursor would be something like input.getTextRange(cursor.offset - 1, cursor.offset). I'll check how hard it would be to implement that. Would at least make it reliable for now, no matter how opentui treats text internally.

@kommander commented on GitHub (Nov 2, 2025): I think the simplest solution to make this work properly, without going into the huge refactor that is needed for the text handling in opentui, is exposing a `getTextRange` method for the `TextBuffer` in opentui, so opencode can use that instead of `String.at`. Then `charBeforeCursor` would be something like `input.getTextRange(cursor.offset - 1, cursor.offset)`. I'll check how hard it would be to implement that. Would at least make it reliable for now, no matter how opentui treats text internally.
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@kommander commented on GitHub (Nov 3, 2025):

This should be mitigated in 1.0.16

@kommander commented on GitHub (Nov 3, 2025): This should be mitigated in `1.0.16`
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@yukukotani commented on GitHub (Nov 3, 2025):

@kommander Thanks, the problem of auto-completion is solved!

Syntax highlighting is still broken. It seems that extmarks from opentui is based on terminal cells while PromptInfo.parts is based on UTF16 characters.

https://github.com/sst/opencode/blob/d06afd87e5a0325730b11478eb49af4ced12608a/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L87-L108

Image
@yukukotani commented on GitHub (Nov 3, 2025): @kommander Thanks, the problem of auto-completion is solved! Syntax highlighting is still broken. It seems that `extmarks` from opentui is based on terminal cells while `PromptInfo.parts` is based on UTF16 characters. https://github.com/sst/opencode/blob/d06afd87e5a0325730b11478eb49af4ced12608a/packages/opencode/src/cli/cmd/tui/component/prompt/autocomplete.tsx#L87-L108 <img width="1486" height="900" alt="Image" src="https://github.com/user-attachments/assets/ad539179-9136-4ea7-903b-7f36702c56e0" />
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@kommander commented on GitHub (Nov 12, 2025):

Should be better in the latest version?

@kommander commented on GitHub (Nov 12, 2025): Should be better in the latest version?
yindo commented 2026-02-16 17:35:48 -05:00
Author
Owner
Copy Link
Copy Source

@zenyr commented on GitHub (Nov 26, 2025):

Update as of v1.0.117

  1. The original issue seems mostly fixed ~2 weeks ago
    👇 Working fine now 👍
    Image

  2. However [Pasted] is a bit broken AFTER submitting
    👇 CJK/Emojis breaking [Pasted] marker 🥲 (Before & after)
    Image

    안녕 {cmd+v} 하세요

@zenyr commented on GitHub (Nov 26, 2025): ### Update as of v1.0.117 1. The original issue seems mostly fixed ~2 weeks ago 👇 Working fine now 👍 <img width="159" height="47" alt="Image" src="https://github.com/user-attachments/assets/510a11b0-98f4-4f36-b494-376ffe64db55" /> 2. However `[Pasted]` is a bit broken AFTER submitting 👇 CJK/Emojis breaking `[Pasted]` marker 🥲 (Before & after) <img width="787" height="142" alt="Image" src="https://github.com/user-attachments/assets/13301ac5-385b-4ca5-b256-1153972fa57e" /> > 안녕 {cmd+v} 하세요
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label bug opentui
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#2470
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?