anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 18:25:32 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

max_tokens defaults to 32000 when using a custom provider #1186

New Issue
Open
opened 2026-02-16 17:29:55 -05:00 by yindo · 10 comments
yindo commented 2026-02-16 17:29:55 -05:00
Owner
Copy Link
Copy Source

Originally created by @nmartorell on GitHub (Aug 8, 2025).

Originally assigned to: @thdxr on GitHub.

Hi,

I'm using LLM (Anthropic, OpenAI and Bedrock) models through an OpenAI-compliant LLM API Gateway. I configured a custom provider in opencode to use this LLM Gateway using the opencode.json file, e.g.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "myprovider": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Custom LLM Gateway",
      "options": {
        "baseURL": "<GATEWAY_URL",
        "apiKey": "<API_KEY>",
      },
      "models": {
        "openai:gpt-4o-mini": {
          "name": "gpt-4o-mini"
        }
        "anthropic:claude-3-5-haiku-20241022": {
          "name": "claude-3-5-haiku-20241022"
        }
      }
    }
  }
}

When I try to use either of these models through opencode, I'm receiving error messages that suggest that something (maybe opencode, maybe the Vercel AI SDK) is defaulting the max_tokens OpenAI Completions field to 32000, which unfortunately is far too large for these models.

As an example, here is the error I see in the LLM Gateway logs when attempting to use the gpt-4o model (I see a similar error with Anthropic models):

  "error": {
    "message": "max_tokens is too large: 32000. This model supports at most 16384 completion tokens, whereas you provided 32000.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "invalid_value"
  }
}

I've tried searching the opencode and Vercel AI SDK codebases to try to find where this max_tokens value is being set, but I unfortunately can't find it. Assuming it is being set by opencode, it would be ideal if this value were left unset when using custom providers, as in the case of my LLM Gateway, it already keeps track of the max_token parameter for each provider / model.

A couple other notes:

  • The reason I know the max_tokens parameter is being set by opencode or Vercel SDK, is because when I write my own http request to my LLM Gateway without specifying max_tokens, the query goes through, e.g.
curl --header 'Authorization: Bearer <API TOKEN>' -H "Content-Type: application/json" -X POST --data '{"model":"openai:gpt-4o-mini","messages":[ {"role": "user", "content": "Hello!"}]}' <GATEWAY_URL>/v1/chat/completions
  • When I modify the opencode.json file to include context and max output tokens, opencode works as expected (I know this is a workaround, but really trying not to have to hardcode context and output tokens in both places):
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "myprovider": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Custom LLM Gateway",
      "options": {
        "baseURL": "<GATEWAY_URL",
        "apiKey": "<API_KEY>",
      },
      "models": {
        "openai:gpt-4o-mini": {
          "name": "gpt-4o-mini",
          "limit": {
            "context": 10000,
            "output": 5000
          }
        }
        "anthropic:claude-3-5-haiku-20241022": {
          "name": "claude-3-5-haiku-20241022",
          "limit": {
            "context": 10000,
            "output": 5000
          }
        }
      }
    }
  }
}

Please let me know if my understanding is wrong, and this parameter is being set elsewhere. Also, please let me know if there is any additional information required to troubleshoot.

Thank you, and thanks for making such an awesome tool!

Originally created by @nmartorell on GitHub (Aug 8, 2025). Originally assigned to: @thdxr on GitHub. Hi, I'm using LLM (Anthropic, OpenAI and Bedrock) models through an OpenAI-compliant LLM API Gateway. I configured a custom provider in opencode to use this LLM Gateway using the opencode.json file, e.g. ``` { "$schema": "https://opencode.ai/config.json", "provider": { "myprovider": { "npm": "@ai-sdk/openai-compatible", "name": "Custom LLM Gateway", "options": { "baseURL": "<GATEWAY_URL", "apiKey": "<API_KEY>", }, "models": { "openai:gpt-4o-mini": { "name": "gpt-4o-mini" } "anthropic:claude-3-5-haiku-20241022": { "name": "claude-3-5-haiku-20241022" } } } } } ``` When I try to use either of these models through opencode, I'm receiving error messages that suggest that something (maybe opencode, maybe the Vercel AI SDK) is defaulting the max_tokens OpenAI Completions field to 32000, which unfortunately is far too large for these models. As an example, here is the error I see in the LLM Gateway logs when attempting to use the gpt-4o model (I see a similar error with Anthropic models): ``` "error": { "message": "max_tokens is too large: 32000. This model supports at most 16384 completion tokens, whereas you provided 32000.", "type": "invalid_request_error", "param": "max_tokens", "code": "invalid_value" } } ``` I've tried searching the opencode and Vercel AI SDK codebases to try to find where this max_tokens value is being set, but I unfortunately can't find it. Assuming it is being set by opencode, it would be ideal if this value were left unset when using custom providers, as in the case of my LLM Gateway, it already keeps track of the max_token parameter for each provider / model. A couple other notes: - The reason I know the max_tokens parameter is being set by opencode or Vercel SDK, is because when I write my own http request to my LLM Gateway without specifying max_tokens, the query goes through, e.g. ``` curl --header 'Authorization: Bearer <API TOKEN>' -H "Content-Type: application/json" -X POST --data '{"model":"openai:gpt-4o-mini","messages":[ {"role": "user", "content": "Hello!"}]}' <GATEWAY_URL>/v1/chat/completions ``` - When I modify the opencode.json file to include context and max output tokens, opencode works as expected (I know this is a workaround, but really trying not to have to hardcode context and output tokens in both places): ``` { "$schema": "https://opencode.ai/config.json", "provider": { "myprovider": { "npm": "@ai-sdk/openai-compatible", "name": "Custom LLM Gateway", "options": { "baseURL": "<GATEWAY_URL", "apiKey": "<API_KEY>", }, "models": { "openai:gpt-4o-mini": { "name": "gpt-4o-mini", "limit": { "context": 10000, "output": 5000 } } "anthropic:claude-3-5-haiku-20241022": { "name": "claude-3-5-haiku-20241022", "limit": { "context": 10000, "output": 5000 } } } } } } ``` Please let me know if my understanding is wrong, and this parameter is being set elsewhere. Also, please let me know if there is any additional information required to troubleshoot. Thank you, and thanks for making such an awesome tool!
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@thdxr commented on GitHub (Aug 8, 2025):

opencode needs to know what the max output length is - when it doesn't it uses 32_000

i can't think of a way around this - you need to specify it somewhere

@thdxr commented on GitHub (Aug 8, 2025): opencode needs to know what the max output length is - when it doesn't it uses 32_000<br><br>i can't think of a way around this - you need to specify it somewhere
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@patrickwork28 commented on GitHub (Sep 7, 2025):

try this please. Worked for me with kimi k2 free.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "openrouter": {
      "models": {
        "moonshotai/kimi-k2:free": {
          "options": {
            "provider": {
              "order": ["baseten", "together", "openrouter"],
              "allow_fallbacks": true
            },
            "transform": "middle-out",
            "max_tokens": 6000
          }
        }
      }
    }
  }
}
@patrickwork28 commented on GitHub (Sep 7, 2025): try this please. Worked for me with kimi k2 free. ``` { "$schema": "https://opencode.ai/config.json", "provider": { "openrouter": { "models": { "moonshotai/kimi-k2:free": { "options": { "provider": { "order": ["baseten", "together", "openrouter"], "allow_fallbacks": true }, "transform": "middle-out", "max_tokens": 6000 } } } } } } ```
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@njbrake commented on GitHub (Dec 4, 2025):

I'm having issues with this as well :( I've tried throwing the max_token setting everywhere I could think, but it still shows up when I look at the logs in LM studio as

25-12-04 17:11:17 [DEBUG]
 Received request: POST to /v1/chat/completions with body  {
  "model": "qwen/qwen3-coder-30b",
  "max_tokens": 32000,
  "temperature": 0.55,

The config

{
    "$schema": "https://opencode.ai/config.json",
    "permission": {
      "edit": "allow",
      "bash": "allow",
      "webfetch": "ask"
    },
    "provider": {
      "lmstudio": {
        "npm": "@ai-sdk/openai-compatible",
        "name": "Mac Mini",
        "options": {
          "baseURL": "http://addres:8123/v1",
          "timeout": 100000000000,
	  "max_tokens": 200000,
          "max_completion_tokens": 200000
        },
        "models": {
          "qwen/qwen3-coder-30b": {
            "name": "Qwen Coder (local)",
            "max_tokens": 131072,
            "max_completion_tokens": 200000,
            "limit": {
              "context": 209600,
              "output": 50000
            }
          }
        }
      }
    }
}
@njbrake commented on GitHub (Dec 4, 2025): I'm having issues with this as well :( I've tried throwing the max_token setting everywhere I could think, but it still shows up when I look at the logs in LM studio as ``` 25-12-04 17:11:17 [DEBUG] Received request: POST to /v1/chat/completions with body { "model": "qwen/qwen3-coder-30b", "max_tokens": 32000, "temperature": 0.55, ``` The config ``` { "$schema": "https://opencode.ai/config.json", "permission": { "edit": "allow", "bash": "allow", "webfetch": "ask" }, "provider": { "lmstudio": { "npm": "@ai-sdk/openai-compatible", "name": "Mac Mini", "options": { "baseURL": "http://addres:8123/v1", "timeout": 100000000000, "max_tokens": 200000, "max_completion_tokens": 200000 }, "models": { "qwen/qwen3-coder-30b": { "name": "Qwen Coder (local)", "max_tokens": 131072, "max_completion_tokens": 200000, "limit": { "context": 209600, "output": 50000 } } } } } } ```
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@jimlaren1 commented on GitHub (Jan 14, 2026):

It can be modified by setting the OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX environment variable.

@jimlaren1 commented on GitHub (Jan 14, 2026): It can be modified by setting the OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX environment variable.
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@lee-b commented on GitHub (Jan 20, 2026):

opencode needs to know what the max output length is - when it doesn't it uses 32_000i can't think of a way around this - you need to specify it somewhere

If the server can produce an error saying that the max tokens are X, then opencode could reasonably be expected to detect that response and reduce its max tokens to X, activating a context compaction if necessary.

@lee-b commented on GitHub (Jan 20, 2026): > opencode needs to know what the max output length is - when it doesn't it uses 32_000i can't think of a way around this - you need to specify it somewhere If the server can produce an error saying that the max tokens are X, then opencode could reasonably be expected to detect that response and reduce its max tokens to X, activating a context compaction if necessary.
yindo commented 2026-02-16 17:29:55 -05:00
Author
Owner
Copy Link
Copy Source

@Onik97 commented on GitHub (Jan 27, 2026):

I ended up doing:

"limit": { "context": 0, "output": 8196 }

Perhaps some playing around and the error should subside

@Onik97 commented on GitHub (Jan 27, 2026): I ended up doing: ` "limit": { "context": 0, "output": 8196 } ` Perhaps some playing around and the error should subside
yindo commented 2026-02-16 17:29:56 -05:00
Author
Owner
Copy Link
Copy Source

@mercurial-moon commented on GitHub (Jan 28, 2026):

@njbrake did you manage to solve the issue, I'm facing a similar issue with KoboldCpp endpoint on Openai compatibility mode. It always truncates the response at 640 tokens.
I tried the limit (context and output) settings but they don't seem to take any effect.

@mercurial-moon commented on GitHub (Jan 28, 2026): @njbrake did you manage to solve the issue, I'm facing a similar issue with KoboldCpp endpoint on Openai compatibility mode. It always truncates the response at 640 tokens. I tried the limit (context and output) settings but they don't seem to take any effect.
yindo commented 2026-02-16 17:29:56 -05:00
Author
Owner
Copy Link
Copy Source

@toralux commented on GitHub (Feb 7, 2026):

"limit": { "context": 0, "output": 8196 }

Thank. Better to use 8192 though? Can confirm it is working. Prefer to put in opencode.json next to model than in global environment variable.

@toralux commented on GitHub (Feb 7, 2026): > "limit": { "context": 0, "output": 8196 } Thank. Better to use 8192 though? Can confirm it is working. Prefer to put in opencode.json next to model than in global environment variable.
yindo commented 2026-02-16 17:29:56 -05:00
Author
Owner
Copy Link
Copy Source

@lee-b commented on GitHub (Feb 7, 2026):

opencode needs to know what the max output length is - when it doesn't it uses 32_000i can't think of a way around this - you need to specify it somewhere

I think this information is returned by /v1/models

@lee-b commented on GitHub (Feb 7, 2026): > opencode needs to know what the max output length is - when it doesn't it uses 32_000i can't think of a way around this - you need to specify it somewhere I think this information is returned by /v1/models
yindo commented 2026-02-16 17:29:56 -05:00
Author
Owner
Copy Link
Copy Source

@Onik97 commented on GitHub (Feb 7, 2026):

"limit": { "context": 0, "output": 8196 }

Thank. Better to use 8192 though? Can confirm it is working. Prefer to put in opencode.json next to model than in global environment variable.

Yeah I change it to anything to be honest haha, just wanted it to work,

@Onik97 commented on GitHub (Feb 7, 2026): > > "limit": { "context": 0, "output": 8196 } > > Thank. Better to use 8192 though? Can confirm it is working. Prefer to put in opencode.json next to model than in global environment variable. Yeah I change it to anything to be honest haha, just wanted it to work,
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#1186
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?