fix(channels): keep GitHub replies hit by a secondary rate limit (#16334)

* fix(channels): keep GitHub replies hit by a secondary rate limit

A final reply that GitHub rejected with a secondary rate limit was
classified as an ambiguous delivery failure: it was not retried, not kept
as a pending delivery, and no error comment was posted, so the answer was
lost while the notification was marked read. GitHub sends secondary
limits with quota left (and not always with retry-after), so only the
exhausted-budget shape counted as a definite no-write.

Treat a 403/429 that carries GitHub's rate-limit headers and names a
secondary limit as a definite no-write too. The reply is retried in
process and, if the limit persists, kept as a pending delivery for the
next channel start. Responses without GitHub's rate-limit headers (for
example a 429 from a proxy) and permission 403s stay single-shot.

* fix(channels): wait out GitHub secondary limits before retrying

A secondary rate limit without retry-after fell through to the generic
1 s / 2 s backoff, so the final reply's three attempts landed within about
three seconds while GitHub was still throttling the account; GitHub asks
for at least a minute in that case, longer on each repeat, and warns that
continued requests can get an integration banned. Wait 60 s and then 120 s
for such a limit; retry-after and the budget reset still take precedence.
Note in the classifier why a retry-after header alone is not proof of a
rejected write there.

* fix(channels): stop the pending GitHub retry pass on a rate limit

Now that a secondary limit counts as a definite no-write, the retry pass
on channel start spent three attempts on every pending reply while the
same account-wide limit answered each of them; stop the pass at the first
rate-limited record and keep the rest for the next start. Pin the retry
count and waits per secondary-limit shape, the older abuse-detection
wording, and that a 502 carrying an exhausted-budget header stays
single-shot. Reword the classifier comment: the header shows a rate
limiter answered, and GitHub's wording tells which one.

* fix(channels): end GitHub rate-limit waits on stop, keep cleaning pending replies

- disconnect() used to cut a rate-limit wait short and let the remaining
  attempts go out back to back into the same limit. A stopped wait now gives
  up with GitHub's last answer, so a final reply stays a definite no-write
  and is kept as a pending delivery.
- The pending pass stops sending at the first record GitHub still
  rate-limits (either limit) but keeps dropping later records the audit
  already shows as posted, which needs no request.
- The PR watch pass treats a 403 carrying GitHub's secondary-limit wording
  as shared, like the publication and notification paths.
- A retry-after that is not a positive number of seconds (an HTTP date,
  "0") no longer skips the secondary-limit minimum.

* fix(channels): treat retry-after on a GitHub-answered 403/429 as a rate limit

The final reply's definite no-write required an exhausted budget or GitHub's
secondary-limit wording, so a rate-limit rejection that carried GitHub's
x-ratelimit-remaining and retry-after but worded its body differently was
audited ambiguous and the reply was dropped. retry-after is GitHub's own
documented signal for that limit; with the x-ratelimit-remaining header still
required, a proxy's bare 429 stays single-shot. A 403/429 whose retry-after is
unusable now waits the secondary-limit minute as well.

Also pin the shape where GitHub rejects the reply once and accepts the retry.

* fix(channels): stop GitHub retries by the error's shape, not the wait branch

- The stop guard keyed on which wait was chosen, so an exhausted budget
  without a reset header took the generic 1 s / 2 s branch and, after
  disconnect(), still fired its remaining attempts into the spent budget.
  Rate-limit answers are now recognised by their shape (403/429 with an
  exhausted budget, retry-after or the secondary wording) for the wait, the
  stop guard and the definite no-write check alike; such an answer without a
  usable retry-after or reset waits the one-minute minimum.
- x-ratelimit-remaining must be a count: an empty value no longer reads as
  an exhausted budget and turns a permission 403 into a stored retry.
- The pending pass starts no further record once a stop is requested, even
  while another write in flight defers its abort.
- Tests pin the stop guard for every wait, reconnecting, a kept record
  between the stop and a posted one, and an infinite retry-after.

* fix(channels): let an answer that lands after a stop decide the pending reply

- The start-up pass returned on an aborted signal before looking at the
  error. A stop defers the abort until the in-flight write ends, so a 502 or
  network failure from that write (GitHub may have posted the comment) left
  the record and its reply_pending task for the next start, which posted the
  reply again. The pass now classifies first; only an attempt that was never
  sent leaves its record, and githubApi marks that abort as not sent.
- A rate limit without GitHub's header (a 429 with only retry-after) is
  still dropped as ambiguous, but it now stops the pass like a definite one
  instead of spending every later record against the same limit.
- x-ratelimit-remaining as a number must be a non-negative integer, and the
  watch pass judges a 403 with the same rate-limit rule as the reply.
- A call that starts after the stop gives up before logging or sleeping.

* fix(channels): treat a past GitHub budget reset as clock skew

GitHub answering with an exhausted budget means its window has not reset, so
a reset time that already looks past says the local clock is ahead; the wait
fell to a 1 s retry and spent all three attempts in about two seconds. It now
waits the one-minute rate-limit minimum instead. Tests give the secondary
limit GitHub's usual reset header and pin the start-up pass keeping its
records when a stop, deferred by another write, lets the wait run out.

* test(channels): pin the ambiguous audit, the watch 429 abort and the quiet stop

Lock three behaviours the suite let drift: a header-less rate limit on the
final reply is audited failed_ambiguous (the key stays consumed), a bare 429
aborts the whole PR watch pass, and a reply that starts after the stop logs
no retry it will not make.

* docs(channels): name the header a GitHub no-write answer needs

The no-write rule needs a numeric x-ratelimit-remaining on the 403 or
429, so a rate-limit answer without it (a 429 with only retry-after,
say) is ambiguous and its reply is not kept; the guide now says so. The
publication test helper's comment also said the opposite of what
stopping the poll loop does to retry waits.

* docs(channels): bring the GitHub publication contract up to the retry rules

The contract still said a definite no-write goes straight to the pending
file. It now names what counts as one, the two in-process retries and
their waits, what an ambiguous answer to a retry does, the stop that
keeps the reply pending, and how the start-up pass stops at the first
reply GitHub still rate-limits. The first bullet now speaks of one final
response rather than one delivery attempt.

* docs(channels): add the Chinese version of the GitHub publication contract

docs/design/README.md asks a change to a single-language design to add
the missing version. The publication contract gains its zh-CN twin,
reciprocal links under both titles, and a Verification line for the
retry tests.
This commit is contained in:
Shaojin Wen authored and GitHub committed 2026-11-05 22:37:41 +08:00
1 parent 6611b8aeb2
commit ef6914c1d9
5 files changed
+1202 -151

No files matched your search

@@ -1,17 +1,19 @@
# GitHub Channel Publication Contract
[English](2026-07-29-github-channel-publication-contract.md) | [简体中文](2026-07-29-github-channel-publication-contract.zh-CN.md)
## Goal
Make GitHub-channel replies safe to publish automatically and traceable after the fact. The channel publishes only the agent's final response through the adapter; intermediate reasoning, tool output, and streaming chunks never become GitHub comments.
## Contract
- The GitHub adapter disables block streaming, so each accepted inbound event produces at most one final response-delivery attempt.
- The GitHub adapter disables block streaming, so each accepted inbound event produces at most one final response to deliver.
- Final delivery uses the active prompt's issue/PR thread rather than a potentially stale shared-session target.
- Channel instructions tell the agent not to use `gh` or the GitHub API to create comments or reviews. The adapter owns public delivery.
- A final response whose trimmed content is only the `<no-reply/>` sentinel is intentionally suppressed. Whitespace, case, a space before `/>`, and a single wrapping code fence are normalized; any other content is published unchanged.
- Suppression and publication are recorded in a local append-only JSONL audit file at `~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-audit.jsonl`. Records contain time, channel, session, source message, thread, outcome, GitHub comment identity/URL when present, and a SHA-256 plus character count of the reply. They never contain reply text, credentials, or a GitHub token.
- Audit writes are best effort. An audit failure is logged without changing the publication result. An ambiguous GitHub API failure remains a delivery failure and is not retried; definite no-write responses are written to a private pending-delivery file and retried on the next channel start.
- Audit writes are best effort. An audit failure is logged without changing the publication result. An ambiguous GitHub API failure remains a delivery failure and is not retried. A definite no-write response — GitHub's rate limiter refusing the comment with a 403 or 429 that carries a numeric `x-ratelimit-remaining` and shows the budget exhausted, sends `retry-after`, or names a secondary rate limit in its message — is retried in process and, if GitHub still refuses it, written to a private pending-delivery file and retried on the next channel start. A rate-limit answer without that header is ambiguous.
## Flow
@@ -20,7 +22,7 @@ Make GitHub-channel replies safe to publish automatically and traceable after th
3. The agent returns one final response.
4. The adapter either suppresses the exact sentinel or creates one issue comment.
5. The adapter appends a publication audit record. The terminal task lifecycle remains owned by `ChannelBase`.
6. If final delivery fails with a definite no-write response, the adapter stores the final text in `~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-pending-deliveries.json` with private file permissions and retries it after restart without rerunning the agent.
6. If final delivery fails with a definite no-write response, the adapter tries twice more, waiting for `retry-after` when it is a positive number of seconds, otherwise until the budget resets when it is exhausted, and otherwise one minute and then two (also when the reset already looks past). An ambiguous answer to one of those attempts ends them like any ambiguous failure, with no pending record. If GitHub still refuses, or the channel stops during a wait, the adapter stores the final text in `~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-pending-deliveries.json` with private file permissions and retries it after restart without rerunning the agent. That start-up pass stops sending at the first stored reply GitHub still rate-limits and keeps the later ones for the following start, together with that reply when the answer is a definite no-write; replies the audit already records as posted are dropped.
## Non-goals
@@ -30,4 +32,4 @@ Make GitHub-channel replies safe to publish automatically and traceable after th
## Verification
Focused GitHub adapter tests cover sentinel suppression, normal final-comment delivery, audit fields without reply text, and non-blocking audit write failures. Existing routing and delivery tests remain unchanged.
Focused GitHub adapter tests cover sentinel suppression, normal final-comment delivery, audit fields without reply text, and non-blocking audit write failures. Existing routing and delivery tests remain unchanged. Further adapter tests cover which rejections count as a definite no-write, the in-process waits and the stop during them, and the start-up pass over pending deliveries.
@@ -0,0 +1,35 @@
# GitHub Channel 发布契约
[English](2026-07-29-github-channel-publication-contract.md) | [简体中文](2026-07-29-github-channel-publication-contract.zh-CN.md)
## 目标
让 GitHub channel 的回复可以安全地自动发布,并在事后可追溯。channel 只通过适配器发布 agent 的最终回复;中间推理、工具输出和流式分块永远不会变成 GitHub 评论。
## 契约
- GitHub 适配器关闭块流式输出,因此每个被接受的入站事件至多产生一个待投递的最终回复。
- 最终投递使用当前 prompt 所在的 issue/PR 线程,而不是可能已过期的共享会话目标。
- channel 指令要求 agent 不得用 `gh` 或 GitHub API 创建评论或 review,公开投递由适配器负责。
- 去除首尾空白后只剩 `<no-reply/>` 哨兵的最终回复会被有意抑制。空白、大小写、`/>` 前的空格以及单层包裹的代码围栏会被规范化;其它内容原样发布。
- 抑制和发布都记录在本地只追加的 JSONL 审计文件 `~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-audit.jsonl` 中。记录包含时间、channel、会话、来源消息、线程、结果、存在时的 GitHub 评论标识/URL,以及回复的 SHA-256 和字符数。记录中绝不包含回复正文、凭据或 GitHub token。
- 审计写入是尽力而为的。审计失败只记日志,不改变发布结果。歧义的 GitHub API 失败仍算投递失败,不会重试。确定未写入的响应——GitHub 的限流器以 403 或 429 拒绝评论,响应带有数值形式的 `x-ratelimit-remaining`,并且配额已耗尽、带有 `retry-after`,或报错文案写明是二级限流——会先在进程内重试;如果 GitHub 仍然拒绝,就写入私有的待发投递文件,并在下次 channel 启动时重试。缺少该头的限流答复按歧义处理。
## 流程
1. GitHub 适配器把被接受的事件派发进 `ChannelBase`。
2. 当前 prompt 在最终投递结束前一直保留入站消息和 issue/PR 线程。
3. agent 返回一个最终回复。
4. 适配器要么抑制精确匹配的哨兵,要么创建一条 issue 评论。
5. 适配器追加一条发布审计记录。任务的终态生命周期仍由 `ChannelBase` 负责。
6. 如果最终投递遇到确定未写入的响应,适配器会再尝试两次:`retry-after` 是正的秒数时按它等待;否则在配额耗尽时等到配额重置;再否则先等一分钟、再等两分钟(重置时间看起来已经过去时也是如此)。其中某次尝试拿到歧义答复时,按普通歧义失败处理,结束尝试且不留待发记录。如果 GitHub 仍然拒绝,或 channel 在等待期间停止,适配器会把最终文本以私有文件权限存入 `~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-pending-deliveries.json`,重启后重试投递,不重新运行 agent。启动时的这轮重试遇到 GitHub 仍在限流的第一条待发回复即停止发送,后面的回复留到下次启动;答复为确定未写入时,这一条也一并保留;审计中已记录为 posted 的回复会被丢弃。
## 非目标
- 不重试歧义的发布失败,不创建状态评论,也不启用流式回复。这些属于 issue #8012 的其它部分。
- 禁止 agent 直接用 `gh`/API 发布的指令只是对 agent 的操作边界,不是沙箱强制。工具级 GitHub 写入限制的强制执行属于运行时权限模型。
- 待发投递的保留策略,包括最大尝试次数、最长保留时间、大小上限、过期回复处理和遗留临时文件清理,另由 #8142 跟踪。
## 验证
GitHub 适配器的针对性测试覆盖哨兵抑制、正常的最终评论投递、不含回复正文的审计字段,以及不阻塞的审计写入失败。已有的路由和投递测试保持不变。另有适配器测试覆盖哪些拒绝算作确定未写入、进程内等待及等待期间的停止,以及启动时对待发投递的那轮重试。
+21 -6
View File
@@ -163,13 +163,28 @@ For an accepted issue or pull-request comment, the channel adds GitHub's `👀`
The GitHub channel publishes only completed responses. Intermediate model chunks are never published as separate comments.
If GitHub returns a definite no-write delivery failure, such as a rate-limit
response, the channel stores the final reply in
If GitHub returns a definite no-write delivery failure — its rate limiter
rejecting the comment with a 403 or 429 that carries a numeric
`x-ratelimit-remaining` header and shows the hourly budget exhausted, sends
`retry-after`, or names a secondary rate limit in its message — the channel
retries the comment twice more, waiting as
long as GitHub asks: `retry-after` when GitHub sends it, the budget reset when
the hourly budget is exhausted, and otherwise one minute and then two (also
when the reset already looks past, which means the local clock is ahead). If it
still fails, or the channel stops during the wait, the channel stores the
final reply in
`~/.qwen/channels/<workspace-scope>/<channel>-<name-hash>-github-pending-deliveries.json`
with private file permissions and retries it on the next channel start. The
corresponding inbound task remains in `reply_pending` state until that delivery
succeeds or reaches a definite terminal failure. Ambiguous delivery failures are
not retried automatically because GitHub may have created the comment.
with private file permissions and retries it on the next channel start. That
retry stops sending at the first stored reply GitHub still rate-limits and
keeps the later ones for the following start, together with that reply when
GitHub's headers prove it was not posted; replies the audit already records as
posted are dropped. The corresponding inbound task
remains in `reply_pending` state until that delivery succeeds or reaches a
definite terminal failure. Ambiguous delivery failures are not retried
automatically because GitHub may have created the comment: the audit records
`failed_ambiguous` and the reply is not kept. A rate-limit answer without a
numeric `x-ratelimit-remaining` header, such as a 429 with only `retry-after`
from a proxy in front of GitHub, is one of them.
## PR-Bound Session Delivery
File diff suppressed because it is too large. Load diff
+105 -41
View File
@@ -1001,17 +1001,55 @@ function isNoReplySentinel(text: string): boolean {
return NO_REPLY_SENTINEL_PATTERN.test((fenced?.[1] ?? trimmed).trim());
}
function isDefiniteNoWriteGithubError(err: unknown): boolean {
const e = err as {
status?: number;
response?: { headers?: Record<string, string | number> };
};
/** GitHub's wording for a secondary (abuse) rate limit. */
const SECONDARY_RATE_LIMIT_MESSAGE = /secondary rate limit|abuse detection/i;
type GithubRequestError = {
status?: number;
message?: unknown;
response?: { headers?: Record<string, string | number> };
};
/** `x-ratelimit-remaining` as a count; absent or malformed is undefined. */
function rateLimitRemaining(err: unknown): number | undefined {
const value = (err as GithubRequestError).response?.headers?.[
'x-ratelimit-remaining'
];
if (typeof value === 'number') {
return Number.isInteger(value) && value >= 0 ? value : undefined;
}
return typeof value === 'string' && /^\d+$/.test(value)
? Number(value)
: undefined;
}
/**
* A 403 or 429 carrying one of GitHub's rate-limit signals: an exhausted
* budget, `retry-after`, or the secondary-limit wording. GitHub sends a
* secondary limit with quota left and not always with `retry-after`, and its
* wording has changed before, so no single signal is required.
*/
function isRateLimitAnswer(err: unknown): boolean {
const e = err as GithubRequestError;
return (
(e.status === 403 || e.status === 429) &&
Number(e.response?.headers?.['x-ratelimit-remaining']) === 0
(rateLimitRemaining(err) === 0 ||
e.response?.headers?.['retry-after'] !== undefined ||
SECONDARY_RATE_LIMIT_MESSAGE.test(String(e.message)))
);
}
/**
* GitHub's rate limiter rejected the write before creating anything: a rate
* limit answer that also carries `x-ratelimit-remaining`, which shows GitHub's
* API answered rather than, say, a bare 429 from a proxy in front of it. A
* component that copied GitHub's headers onto its own 403/429 could still be
* misread.
*/
function isDefiniteNoWriteGithubError(err: unknown): boolean {
return isRateLimitAnswer(err) && rateLimitRemaining(err) !== undefined;
}
/**
* Watch-pass fail-fast predicate. The watch is best-effort detection that runs
* serially before the notification lane, so it must not sleep through
@@ -1036,9 +1074,6 @@ function isWatchFailFastError(err: unknown): boolean {
);
}
/** GitHub's wording for a secondary (abuse) rate limit. */
const SECONDARY_RATE_LIMIT_MESSAGE = /secondary rate limit|abuse detection/i;
/**
* Fail-fast statuses that condemn the WHOLE pass, not just one PR: a bad
* credential or an exhausted rate budget is shared by every watched PR, so
@@ -1058,20 +1093,10 @@ function isWatchPassAbortError(err: unknown): boolean {
response?: { headers?: Record<string, string | number> };
};
if (e.status === 401 || e.status === 429) return true;
if (e.status === 403) {
// GitHub sends a secondary/abuse limit with quota left and not always
// with retry-after, so an exhausted budget, retry-after or the limit's
// wording each mark a shared rate limit. A bare permission 403 (quota
// left, no retry-after, no limit wording) stays per-PR so it cannot
// starve the other watched PRs.
const headers = e.response?.headers;
return (
Number(headers?.['x-ratelimit-remaining']) === 0 ||
headers?.['retry-after'] !== undefined ||
SECONDARY_RATE_LIMIT_MESSAGE.test(String(e.message))
);
}
return false;
// A 403 aborts only as a rate limit; a bare permission 403 (quota left, no
// retry-after, no limit wording) stays per-PR so it cannot starve the other
// watched PRs.
return e.status === 403 && isRateLimitAnswer(err);
}
/**
@@ -1701,8 +1726,13 @@ export class GithubChannel extends PollingChannelBase<GithubCursor> {
signal?: AbortSignal,
): Promise<void> {
const pending = this.readPendingFinalDeliveries();
let rateLimited = false;
for (const record of pending) {
if (signal?.aborted) return;
// A stop defers the abort while another write is in flight; start no
// new record meanwhile.
if (signal?.aborted || this.pendingFinalDeliveryRetryStopRequested) {
return;
}
const auditBase = this.buildPublicationAuditBase({
chatId: record.chatId,
threadId: record.threadId,
@@ -1730,6 +1760,10 @@ export class GithubChannel extends PollingChannelBase<GithubCursor> {
}
continue;
}
// A rate limit answers every later record the same way: keep them for
// the next start instead of spending three more attempts on each. The
// already-posted check above needs no request, so it still runs.
if (rateLimited) continue;
try {
const comment = await this.createIssueComment(
record.chatId,
@@ -1763,16 +1797,24 @@ export class GithubChannel extends PollingChannelBase<GithubCursor> {
}
this.removeReplyPendingInboundTask(record);
} catch (err) {
if (signal?.aborted) return;
// The abort waits for an in-flight write, so an answer that arrives
// with the signal aborted still decides the record; only an attempt
// that was never sent leaves it for the next start.
if ((err as { notSent?: boolean }).notSent) return;
if (isDefiniteNoWriteGithubError(err)) {
rateLimited = true;
continue;
}
// A rate limit without GitHub's header is no proof the write was
// refused, so this record is dropped below, but it would answer the
// later records the same way: keep those for the next start.
if (isRateLimitAnswer(err)) rateLimited = true;
this.recordPublicationAudit({
...auditBase,
at: new Date().toISOString(),
type: 'github_publication',
// Ambiguous by construction: the definite-no-write branch above
// already continued. Keep the key consumed (at-most-once).
// Ambiguous by construction: a definite no-write never reaches
// here. Keep the key consumed (at-most-once).
outcome: 'failed_ambiguous',
failurePhase: 'delivery',
failureError: sanitizeLogText(
@@ -4070,7 +4112,11 @@ export class GithubChannel extends PollingChannelBase<GithubCursor> {
signal?: AbortSignal,
): Promise<T> {
for (let attempt = 1; attempt <= retries; attempt++) {
if (signal?.aborted) throw new Error(`${label} aborted`);
if (signal?.aborted) {
// Thrown before this attempt was sent: callers can tell it from the
// answer of a request that went out.
throw Object.assign(new Error(`${label} aborted`), { notSent: true });
}
try {
return await fn();
} catch (err: unknown) {
@@ -4084,31 +4130,49 @@ export class GithubChannel extends PollingChannelBase<GithubCursor> {
};
const headers = e.response?.headers ?? {};
// A retry-after that is not a positive number of seconds (an HTTP
// date, "0") is ignored, so the budget reset or the rate-limit
// minimum below sets the wait instead.
const retryAfter = Number(headers['retry-after'] ?? NaN);
const usableRetryAfter = Number.isFinite(retryAfter) && retryAfter > 0;
const rateLimited = isRateLimitAnswer(err);
// GitHub answering with an exhausted budget means its window has not
// reset yet, so a reset that looks past is clock skew: the minimum
// below applies instead of a 1 s retry.
const untilReset =
Number(headers['x-ratelimit-reset']) * 1000 - Date.now();
let cooldown: number;
if (headers['retry-after']) {
const retryAfter = Number(headers['retry-after']);
cooldown = Number.isFinite(retryAfter)
? retryAfter * 1000
: 1000 * 2 ** (attempt - 1);
if (usableRetryAfter) {
cooldown = retryAfter * 1000;
} else if (
(e.status === 403 || e.status === 429) &&
Number(headers['x-ratelimit-remaining']) === 0 &&
Number(headers['x-ratelimit-reset']) > 0
rateLimited &&
rateLimitRemaining(err) === 0 &&
untilReset > 0
) {
cooldown =
Math.max(
0,
Number(headers['x-ratelimit-reset']) * 1000 - Date.now(),
) + 1000;
cooldown = untilReset + 1000;
} else if (rateLimited) {
// GitHub's guidance for a rate limit without a usable retry-after
// or reset: wait at least a minute, longer on each repeat.
cooldown = 60_000 * 2 ** (attempt - 1);
} else {
cooldown = 1000 * 2 ** (attempt - 1);
}
cooldown = Math.min(cooldown, MAX_TIMER_DELAY_MS);
// After disconnect() — which ends the wait early, skips it for a call
// that started after the stop, or lets it run out while an in-flight
// write defers the abort — give up with GitHub's last answer instead
// of retrying into the same limit. A definite no-write stays
// definite, so a final reply is kept as a pending delivery.
const stopped = () =>
(usableRetryAfter || rateLimited) &&
this.pendingFinalDeliveryRetryStopRequested;
if (stopped()) throw err;
process.stderr.write(
`[Channel:${this.name}] ${label} failed (attempt ${attempt}/${retries}, status=${e.status}), retrying in ${cooldown}ms: ${sanitizeLogText(e.message ?? String(err), 200)}\n`,
);
await this.sleepForRetry(cooldown, signal);
if (stopped()) throw err;
}
}
throw new Error('unreachable');