anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

GPT-5 seems unmistakably "dumber" than Claude Opus 4.1 in Plan mode? #1572

New Issue
Closed
opened 2026-02-16 17:31:36 -05:00 by yindo · 14 comments
yindo commented 2026-02-16 17:31:36 -05:00
Owner
Copy Link
Copy Source

Originally created by @science on GitHub (Sep 2, 2025).

Originally assigned to: @thdxr on GitHub.

I've been comparing GPT-5 and Claude Opus 4.1 using Opencode. And while both seem to implement code well in Build mode, it seems that GPT-5 really doesn't perform well in Plan mode. Here are some things I'm noting:

  1. Opus 4.1 will use more of the built-in tools to read files, run bash commands and generally "look around" more
  2. Opus 4.1 is able to develop root cause analyses better
  3. And, it seems to develop better plans for coding implementation (probably as a result of the first two)

My question: it seems unlikely to me that this is a fundamental fact of the two models being this dissimilar in capability? If that's right, is this more about how opencode leverages the two models? Is there a roadmap for enabling GPT-5 to use Opencode's tools effectively, in the way that Opus 4.1 does now? Or is the problem really that GPT-5 isn't good at operating Opencode-style infrastructure?

Feel free to refute any of my assumptions in the question too! GPT-5 is a lot cheaper and if it would function better in plan mode, it would be pretty easy to switch over. But as of right now, I really need to use Claude for harder implement/debug issues, and I'd prefer to use GPT-5 simply because it is a lot cheaper when using pay-as-you-go API keys.

Originally created by @science on GitHub (Sep 2, 2025). Originally assigned to: @thdxr on GitHub. I've been comparing GPT-5 and Claude Opus 4.1 using Opencode. And while both seem to implement code well in Build mode, it seems that GPT-5 really doesn't perform well in Plan mode. Here are some things I'm noting: 1. Opus 4.1 will use more of the built-in tools to read files, run bash commands and generally "look around" more 2. Opus 4.1 is able to develop root cause analyses better 3. And, it seems to develop better plans for coding implementation (probably as a result of the first two) My question: it seems unlikely to me that this is a fundamental fact of the two models being this dissimilar in capability? If that's right, is this more about how opencode leverages the two models? Is there a roadmap for enabling GPT-5 to use Opencode's tools effectively, in the way that Opus 4.1 does now? Or is the problem really that GPT-5 isn't good at operating Opencode-style infrastructure? Feel free to refute any of my assumptions in the question too! GPT-5 is a lot cheaper and if it would function better in plan mode, it would be pretty easy to switch over. But as of right now, I really need to use Claude for harder implement/debug issues, and I'd prefer to use GPT-5 simply because it is a lot cheaper when using pay-as-you-go API keys.
yindo added the model-problem label 2026-02-16 17:31:36 -05:00
yindo closed this issue 2026-02-16 17:31:36 -05:00
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Sep 2, 2025):

Mind sharing a session to show what you mean?

@rekram1-node commented on GitHub (Sep 2, 2025): Mind sharing a session to show what you mean?
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@science commented on GitHub (Sep 2, 2025):

Thanks @rekram1-node for noticing.

Claude Opus 4.1 debugging a test issue (and succeeding): https://opencode.ai/s/fuzdgGRi
GPT 5 debugging the same issue (and failing): https://opencode.ai/s/KXqF1rDw

The Opus 4.1 solution fixed the problem. The GPT 5 solution seemed pretty wacky/unlikely to me, and when I asked it to implement, the changes it made caused the system to break in a worse way. In this case I ran the two tests 100% side-by-side, even reverting the repo after Opus made changes, so GPT 5 had the exact same codebase. (Normally I'm not head-to-heading the AI so precisely b/c I'm just trying to make forward progress on my hobby codebase.)

Of course, this is just one example. But I have had 10+ experiences with this type of problem where GPT 5 feels kind of lazy, and its debugging thinking is weak. It often wants to jump to "race conditions" or "network flakiness" as explanations, when those are obviously not the root cause problems (and don't exist at all usually). Maybe GPT 5 is just dumber, but given all the online coding benchmarks I'm seeing, the two models appear quite competitive, so I'm wondering why I'm getting such weak performance in Opencode from GPT 5 vs Opus 4.1?

@science commented on GitHub (Sep 2, 2025): Thanks @rekram1-node for noticing. Claude Opus 4.1 debugging a test issue (and succeeding): https://opencode.ai/s/fuzdgGRi GPT 5 debugging the same issue (and failing): https://opencode.ai/s/KXqF1rDw The Opus 4.1 solution fixed the problem. The GPT 5 solution seemed pretty wacky/unlikely to me, and when I asked it to implement, the changes it made caused the system to break in a worse way. In this case I ran the two tests 100% side-by-side, even reverting the repo after Opus made changes, so GPT 5 had the exact same codebase. (Normally I'm not head-to-heading the AI so precisely b/c I'm just trying to make forward progress on my hobby codebase.) Of course, this is just one example. But I have had 10+ experiences with this type of problem where GPT 5 feels kind of lazy, and its debugging thinking is weak. It often wants to jump to "race conditions" or "network flakiness" as explanations, when those are obviously not the root cause problems (and don't exist at all usually). Maybe GPT 5 is just dumber, but given all the online coding benchmarks I'm seeing, the two models appear quite competitive, so I'm wondering why I'm getting such weak performance in Opencode from GPT 5 vs Opus 4.1?
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Dragonator-EU commented on GitHub (Sep 2, 2025):

I tried GPT5 in the Copilot extension too and it is noticeably "dumber" than even Claude Sonnet 4. Or at the very least it thinks very differently and I find Sonnet much more reasonable.

@Dragonator-EU commented on GitHub (Sep 2, 2025): I tried GPT5 in the Copilot extension too and it is noticeably "dumber" than even Claude Sonnet 4. Or at the very least it thinks very differently and I find Sonnet much more reasonable.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Ramblurr commented on GitHub (Sep 4, 2025):

In my experience the past weeks:

  • It is difficult to get gpt5 in plan mode to use tools, it tries to just one shot answers without gathering information. I have to be very explicit telling it to use the read tools to pick up context, and after a few turns it stops
  • In build mode gpt5 has the opposite problem, it will decide it needs some information then get stuck in a forever loop of reading files, looking at the code of dependencies, etc and never return to building. Even if I stop it and tell it to start developing again it'll say "But I need some more information let me gather that, then I can fix the bug" and return to the read loop

Unfortunately I don't have any sessions to share.

@Ramblurr commented on GitHub (Sep 4, 2025): In my experience the past weeks: - It is difficult to get gpt5 in plan mode to use tools, it tries to just one shot answers without gathering information. I have to be very explicit telling it to use the read tools to pick up context, and after a few turns it stops - In build mode gpt5 has the opposite problem, it will decide it needs some information then get stuck in a forever loop of reading files, looking at the code of dependencies, etc and never return to building. Even if I stop it and tell it to start developing again it'll say "But I need some more information let me gather that, then I can fix the bug" and return to the read loop Unfortunately I don't have any sessions to share.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sewer56 commented on GitHub (Sep 5, 2025):

Image

There's also the fact max context is misrepresented when using GPT-5 via CoPilot (in 0.6.4). It thinks it's a 128K model

Image

Should be an easy fix I imagine, but also worth mentioning.

@Sewer56 commented on GitHub (Sep 5, 2025): <img width="180" height="48" alt="Image" src="https://github.com/user-attachments/assets/42a96dfb-e6af-45a1-98e7-568555d54540" /> There's also the fact max context is misrepresented when using GPT-5 via CoPilot (in 0.6.4). It thinks it's a 128K model <img width="290" height="65" alt="Image" src="https://github.com/user-attachments/assets/c61bcbe3-6f47-4a0a-ad62-df553d8839e7" /> Should be an easy fix I imagine, but also worth mentioning.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Sep 6, 2025):

There's also the fact max context is misrepresented when using GPT-5 via CoPilot (in 0.6.4). It thinks it's a 128K model

copilot limits the context on their end this is NOT a bug

Last I checked the 128k limit is accurate and remains so afaik, this limit is also present for most other models they host

@rekram1-node commented on GitHub (Sep 6, 2025): > There's also the fact max context is misrepresented when using GPT-5 via CoPilot (in 0.6.4). It thinks it's a 128K model copilot limits the context on their end this is NOT a bug Last I checked the 128k limit is accurate and remains so afaik, this limit is also present for most other models they host
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sewer56 commented on GitHub (Sep 6, 2025):

@rekram1-node Is it normal for opencode to report over 300k context when using copilot in this case? I've had it climb to that.

@Sewer56 commented on GitHub (Sep 6, 2025): @rekram1-node Is it normal for opencode to report over 300k context when using copilot in this case? I've had it climb to that.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Sep 6, 2025):

I'm pretty sure that context you are seeing is the total context for session, opencode will auto compact as it goes to allow you to continue the convo

@rekram1-node commented on GitHub (Sep 6, 2025): I'm pretty sure that context you are seeing is the total context for session, opencode will auto compact as it goes to allow you to continue the convo
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sewer56 commented on GitHub (Sep 6, 2025):

Isn't that questionable UX then?

Running /compact does reset the token counter at the top of the window, but you're saying here that auto compact does not, and continues over 100%.

@Sewer56 commented on GitHub (Sep 6, 2025): Isn't that questionable UX then? Running `/compact` does reset the token counter at the top of the window, but you're saying here that `auto compact` does not, and continues over 100%.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Sep 6, 2025):

@Sewer56 ah I misunderstood you.

I guess it depends, what model did you see 300K + for? I think some of their models allow it just 128k is default

@rekram1-node commented on GitHub (Sep 6, 2025): @Sewer56 ah I misunderstood you. I guess it depends, what model did you see 300K + for? I think some of their models allow it just 128k is default
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@Sewer56 commented on GitHub (Sep 6, 2025):

@rekram1-node GPT-5 on CoPilot, per screenshot snippet.
Fairly consistently in the mid 200%, sometimes 300%+

@Sewer56 commented on GitHub (Sep 6, 2025): @rekram1-node GPT-5 on CoPilot, per screenshot snippet. Fairly consistently in the mid 200%, sometimes 300%+
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@ferenci84 commented on GitHub (Sep 14, 2025):

FYI, I recently started a session with GPT-5 with empty system prompt (developer message) by mistake, and it worked really great, then I corrected my mistake, and wasn't able to finish the job. I then tested a problem using several models, and GPT-5 and GPT-5 mini was successful only with empty system prompt.

@ferenci84 commented on GitHub (Sep 14, 2025): FYI, I recently started a session with GPT-5 with empty system prompt (developer message) by mistake, and it worked really great, then I corrected my mistake, and wasn't able to finish the job. I then tested a problem using several models, and GPT-5 and GPT-5 mini was successful only with empty system prompt.
yindo commented 2026-02-16 17:31:36 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Sep 14, 2025):

@Sewer56 btw a token counting bug in tui was fixed yesterday I think that may have been culprit, it would tell u a much higher context than it was sometimes

@rekram1-node commented on GitHub (Sep 14, 2025): @Sewer56 btw a token counting bug in tui was fixed yesterday I think that may have been culprit, it would tell u a much higher context than it was sometimes
yindo commented 2026-02-16 17:31:37 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Dec 27, 2025):

[automated] Closing due to 90+ days of inactivity. Feel free to reopen if you still need this!

@rekram1-node commented on GitHub (Dec 27, 2025): [automated] Closing due to 90+ days of inactivity. Feel free to reopen if you still need this!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label model-problem
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#1572
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?