[GH-ISSUE #79] Catch server errors and return alert to the user #45

New Issue

2026-02-15T18:17:09-05:00

yindo commented

2026-02-15 18:17:09 -05:00

Originally created by @rlancemartin on GitHub (Apr 30, 2023).
Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/79

Originally assigned to: @benisgold on GitHub.

Test w/ GPT-4 paper (example of invalid token):

ValueError: Encountered text corresponding to disallowed special token '<|endofprompt|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

Originally created by @rlancemartin on GitHub (Apr 30, 2023). Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/79 Originally assigned to: @benisgold on GitHub. Test w/ GPT-4 paper (example of invalid token): ``` ValueError: Encountered text corresponding to disallowed special token '<|endofprompt|>'. If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`. If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`. To disable this check for all special tokens, pass `disallowed_special=()`. ```

yindo added the good first issue label 2026-02-15 18:17:09 -05:00

yindo closed this issue

2026-02-15 18:17:09 -05:00

yindo commented

2026-02-15 18:17:10 -05:00

@rlancemartin commented on GitHub (Apr 30, 2023):

Confirm now we will see a server error:

Error
There was an error from the server.

@rlancemartin commented on GitHub (Apr 30, 2023): Confirm now we will see a server error: ``` Error There was an error from the server. ``` ![image](https://user-images.githubusercontent.com/122662504/235389749-36c12519-fea1-4a98-90ab-e030ca62ab32.png)

yindo commented

2026-02-15 18:17:10 -05:00

@rlancemartin commented on GitHub (May 1, 2023):

But, the railway deployed API actually will just return None.

This throws a server error:

If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`.
ValueError: Encountered text corresponding to disallowed special token '<|endofprompt|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`.
To disable this check for all special tokens, pass `disallowed_special=()`.

But the API returns None:

curl -X POST -F "files=@docs/transformers/2303.08774.pdf" -F "num_eval_questions=1" -F "chunk_chars=1000" -F "overlap=100" -F "split_method=RecursiveTextSplitter" -F "retriever_type=similarity-search" -F "embeddings=OpenAI" -F "model_version=gpt-3.5-turbo" -F "grade_prompt=Fast" -F "num_neighbors=3" https://auto-evaluator-production.up.railway.app/evaluator-stream

So, in prod we could catch None from the server and just alert an error.

@rlancemartin commented on GitHub (May 1, 2023): But, the railway deployed API actually will just return `None`. This throws a server error: ``` If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`. ValueError: Encountered text corresponding to disallowed special token '<|endofprompt|>'. If you want this text to be encoded as a special token, pass it to `allowed_special`, e.g. `allowed_special={'<|endofprompt|>', ...}`. If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`. To disable this check for all special tokens, pass `disallowed_special=()`. If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=(enc.special_tokens_set - {'<|endofprompt|>'})`. To disable this check for all special tokens, pass `disallowed_special=()`. ``` But the API returns `None`: ``` curl -X POST -F "files=@docs/transformers/2303.08774.pdf" -F "num_eval_questions=1" -F "chunk_chars=1000" -F "overlap=100" -F "split_method=RecursiveTextSplitter" -F "retriever_type=similarity-search" -F "embeddings=OpenAI" -F "model_version=gpt-3.5-turbo" -F "grade_prompt=Fast" -F "num_neighbors=3" https://auto-evaluator-production.up.railway.app/evaluator-stream ``` So, in prod we could catch `None` from the server and just alert an error.

yindo referenced this issue

2026-02-15 19:14:57 -05:00

[PR #45] [MERGED] @benisgold/prerelease #83

yindo changed title from ~~Catch server errors and return alert to the user~~ to [GH-ISSUE #79] Catch server errors and return alert to the user

2026-06-05 17:14:12 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/auto-evaluator#45