feat(ai): fallback to a secondary model when the provider keeps failing

2026-10-12 02:02:42 +00:00 · 2026-09-24 05:45:55 -04:00
parent 1eb4896ac3
commit a9ab2861d8
7 changed files with 164 additions and 29 deletions
@@ -7,7 +7,7 @@
    "asset-curator",
    "qa-playtester"
  ],
-  "last_cycle": 261,
+  "last_cycle": 262,
  "completed_targets": [
    {
      "cycle": 75,
@@ -1540,6 +1540,12 @@
      "target": "Phase 5: adversarial-invariant audit + full editor-suite gate (no change)",
      "metric_delta": "No code change. (1) Adversarial scan across the repo and the fork AI surface found no hardcoded secrets (sk-/Bearer/api_key literals), no provider-locked hostnames (openrouter/openai/deepseek/anthropic) in newIDE/app/src/AI or AiGeneration, and no key in the committed dev scripts (the mock only names the CORS 'authorization' header) - invariant 1 and the no-secrets rule hold at the tree level. (2) Ran the FULL editor suite, not just the AI surface: 187 suites, 2437 passed, 1 skipped, 114 snapshots - green, confirming the session's many AI-surface edits did not regress the wider app (previously only the 83-suite AI-surface subset was re-run each cycle).",
      "commit": "docs(autonomous): adversarial-invariant audit and full editor-suite gate"
+    },
+    {
+      "cycle": 262,
+      "target": "Tier 2 fallback routing: a secondary model used when the primary exhausts its provider retries",
+      "metric_delta": "Implemented the objective's Tier 2 'fallback routing'. (1) CustomAIConfig gains fallbackModel (sanitized); (2) a 'Fallback model (optional)' field in the AI Settings Preferences, wired through PreferencesContext/Provider/Dialog exactly like aiCustomModel (PreferencesStorage strips only apiKey, so it persists); (3) the three turn paths (create/continue/sub-agent) now accept a per-attempt model override, and runTurnWithProviderRetry makes one final attempt with the fallback model after the 2 provider retries fail on a retryable 408/429/502/503/504. +1 spec (3x503 then a 4th call with the fallback model => ready, 4 posts, 4th uses 'fallback-model'). Full editor suite 187 suites / 2438 passed; build green; ESLint/Prettier/divergence clean. Live-verified with a model-aware provider: primary-model -> 503 x3 then fallback-model -> 200, request ended ready, console logged 'Provider still failing (503); falling back to model \"fallback-model\"', and the Preferences field renders.",
+      "commit": "feat(ai): fallback to a secondary model when the provider keeps failing"
    }
  ],
  "backlog": [
@@ -1761,6 +1767,7 @@
    "A role-based agent must get BOTH its role prompt and its role tool subset at the top level too, not only for sub-agents. The BYOK orchestrator ran with every tool plus a generic prompt, so the 'manager' edited the project directly (feedback-loop Run 5); resolveStudioRoleId now maps orchestrator mode to the manager role while a stale role id still fails closed.",
    "A fresh @feedback-loop run is the right validation for an orchestration fix: the deterministic spec pinned the tool subset, but only the real run proved the manager delegates (4 spawns, 0 direct edits) instead of building the project itself.",
    "Sub-agent failure recovery works (feedback-loop Run 6): an errored child is treated as finished, its failed report is delivered to the manager, and the manager spawned a replacement. The underlying 503 is now retried client-side, so a transient provider blip no longer kills a sub-agent turn. Keep 500 non-retryable (a genuine provider error should surface as-is).",
-    "Periodically run the FULL newIDE/app suite (187 suites), not only the AI-surface subset: the fast per-cycle gate is a subset and cannot prove the wider app still passes after many edits. The full suite is ~10s here."
+    "Periodically run the FULL newIDE/app suite (187 suites), not only the AI-surface subset: the fast per-cycle gate is a subset and cannot prove the wider app still passes after many edits. The full suite is ~10s here.",
+    "Fallback routing belongs after the provider retries, not instead of them: retry the same model (transient blips often clear), then switch models once. Keep the model list user-configurable (AI Settings) and persisted via the preferences store; only the apiKey is stripped before persistence."
  ]
 }
@@ -35,6 +35,11 @@ export type CustomAIConfig = {|
  baseUrl: string,
  apiKey: string,
  model: string,
+  /**
+   * Optional secondary model used when the primary model exhausts its provider
+   * retries on a transient 5xx (fallback routing). Empty/absent disables it.
+   */
+  fallbackModel?: string,
  temperature: number,
  maxTokens?: number,
  customHeaders?: { [string]: string },
@@ -142,6 +147,10 @@ const sanitizeCustomAIConfig = (input: mixed): CustomAIConfig => {
    baseUrl: typeof parsed.baseUrl === 'string' ? parsed.baseUrl : '',
    apiKey: typeof parsed.apiKey === 'string' ? parsed.apiKey : '',
    model: typeof parsed.model === 'string' ? parsed.model : '',
+    fallbackModel:
+      typeof parsed.fallbackModel === 'string'
+        ? parsed.fallbackModel.trim()
+        : undefined,
    temperature:
      typeof parsed.temperature === 'number' &&
      Number.isFinite(parsed.temperature)
@@ -1048,6 +1057,7 @@ export const trimMessagesToBudget = (

 /** Reset CustomAIClient state (for testing). */
 export const _resetCustomAiClientForTesting = () => {
+  providerRetryDelayMs = PROVIDER_RETRY_DELAY_MS;
  cachedConfig = null;
  for (const key of Object.keys(localAiRequestsCache)) {
    delete localAiRequestsCache[key];
@@ -3121,6 +3131,13 @@ const delay = (ms: number): Promise<void> =>
 const PROVIDER_RETRY_DELAY_MS = 1500;
 const MAX_PROVIDER_RETRIES = 2;

+// Overridable in tests so the retry/fallback path does not wait for real
+// backoff. Production never changes it.
+let providerRetryDelayMs = PROVIDER_RETRY_DELAY_MS;
+export const _setProviderRetryDelayForTesting = (ms: number): void => {
+  providerRetryDelayMs = ms;
+};
+
 /** The HTTP status of a thrown provider error, or null. */
 export const getProviderErrorStatus = (error: any): number | null => {
  const status = error && (error: any).providerStatus;
@@ -3141,33 +3158,50 @@ export const isRetryableProviderError = (error: any): boolean => {
 };

 /**
- * Run a turn, retrying a transient provider error (429/5xx) with backoff.
- * Aborted turns are never retried.
+ * Run a turn, retrying a transient provider error (429/5xx) with backoff, and
+ * finally falling back to an optional secondary model before giving up. Aborted
+ * turns are never retried.
 */
 const runTurnWithProviderRetry = async (
-  sendTurn: () => Promise<any>,
-  abortSignal: { aborted: boolean }
+  sendTurn: (?string) => Promise<any>,
+  abortSignal: { aborted: boolean },
+  fallbackModel?: string | null
 ): Promise<any> => {
  let attempt = 0;
+  let triedFallback = false;
  for (;;) {
    try {
-      return await sendTurn();
+      return await sendTurn(triedFallback ? fallbackModel : undefined);
    } catch (error) {
-      if (
-        abortSignal.aborted ||
-        !isRetryableProviderError(error) ||
-        attempt >= MAX_PROVIDER_RETRIES
-      ) {
+      if (abortSignal.aborted || !isRetryableProviderError(error)) {
        throw error;
      }
-      attempt++;
-      const delayMs = PROVIDER_RETRY_DELAY_MS * attempt;
-      console.warn(
-        `[CustomAIClient] Transient provider error (${getProviderErrorStatus(
-          error
-        )}); retrying in ${delayMs}ms (attempt ${attempt}/${MAX_PROVIDER_RETRIES}).`
-      );
-      await delay(delayMs);
+      if (attempt < MAX_PROVIDER_RETRIES) {
+        attempt++;
+        const delayMs = providerRetryDelayMs * attempt;
+        console.warn(
+          `[CustomAIClient] Transient provider error (${getProviderErrorStatus(
+            error
+          )}); retrying in ${delayMs}ms (attempt ${attempt}/${MAX_PROVIDER_RETRIES}).`
+        );
+        await delay(delayMs);
+        continue;
+      }
+      const canFallBack =
+        !triedFallback &&
+        typeof fallbackModel === 'string' &&
+        !!fallbackModel.trim();
+      if (canFallBack) {
+        triedFallback = true;
+        console.warn(
+          `[CustomAIClient] Provider still failing (${getProviderErrorStatus(
+            error
+          )}); falling back to model "${fallbackModel}".`
+        );
+        await delay(providerRetryDelayMs);
+        continue;
+      }
+      throw error;
    }
  }
 };
@@ -4359,11 +4393,13 @@ export const customCreateAiRequest = async ({

  const abortController = registerTurnAbortController(reqId);
  pendingCreateAiRequestIds[reqId] = true;
-  const sendTurn = () =>
+  const sendTurn = (modelOverride?: string) =>
    sendChatCompletion({
      messages: budgetedMessages,
      tools: createTools,
-      config: createConfig,
+      config: modelOverride
+        ? { ...createConfig, model: modelOverride }
+        : createConfig,
      signal: abortController.signal,
      // The first turn is the one most likely to hit a model still loading, so
      // it must publish partial content like addMessage does: the chat's
@@ -4386,7 +4422,8 @@ export const customCreateAiRequest = async ({
    try {
      assistantResponse = await runTurnWithProviderRetry(
        sendTurn,
-        abortController.signal
+        abortController.signal,
+        createConfig.fallbackModel
      );
    } catch (error) {
      // The first turn embeds the project structure, so it is the most likely
@@ -4628,11 +4665,14 @@ export const customAddMessageToAiRequest = async ({
    }

    const abortController = registerTurnAbortController(aiRequestId);
-    const sendTurn = () =>
+    const continueConfig = getEffectiveConfigForRequest(aiRequestId);
+    const sendTurn = (modelOverride?: string) =>
      sendChatCompletion({
        messages: budgetedMessages,
        tools: roleTools,
-        config: getEffectiveConfigForRequest(aiRequestId),
+        config: modelOverride
+          ? { ...continueConfig, model: modelOverride }
+          : continueConfig,
        signal: abortController.signal,
        onStreamDelta: partialContent => {
          localAiRequestPartialContent[aiRequestId] = partialContent;
@@ -4649,7 +4689,8 @@ export const customAddMessageToAiRequest = async ({
      try {
        assistantResponse = await runTurnWithProviderRetry(
          sendTurn,
-          abortController.signal
+          abortController.signal,
+          continueConfig.fallbackModel
        );
      } catch (error) {
        // A context-overflow 400 means the request still exceeded the window
@@ -4911,11 +4952,16 @@ export const customCreateSubAgentAiRequest = async ({
      reqId,
      parentAiRequestId
    );
-    const sendTurn = () =>
+    const subAgentConfig = getEffectiveConfigForRequest(
+      parentAiRequestId || reqId
+    );
+    const sendTurn = (modelOverride?: string) =>
      sendChatCompletion({
        messages: budgetedMessages,
        tools: subAgentTools,
-        config: getEffectiveConfigForRequest(parentAiRequestId || reqId),
+        config: modelOverride
+          ? { ...subAgentConfig, model: modelOverride }
+          : subAgentConfig,
        signal: abortController.signal,
        // Attribute the partial content to the parent chat, which is what the
        // UI polls while a sub-agent runs (the sub-agent request is internal).
@@ -4938,7 +4984,8 @@ export const customCreateSubAgentAiRequest = async ({
      try {
        assistantResponse = await runTurnWithProviderRetry(
          sendTurn,
-          abortController.signal
+          abortController.signal,
+          subAgentConfig.fallbackModel
        );
      } catch (error) {
        // Sub-agents receive the project JSON too, so they can overflow the
@@ -46,6 +46,7 @@ import {
  isNetworkLevelError,
  isRetryableProviderError,
  getProviderErrorStatus,
+  _setProviderRetryDelayForTesting,
  estimateTokens,
  estimateToolsTokens,
  getMessageBudget,
@@ -1852,6 +1853,42 @@ describe('CustomAIClient', () => {
      expect(axios.post).toHaveBeenCalledTimes(2);
    });

+    it('falls back to the secondary model when the primary keeps failing', async () => {
+      // Tier 2 fallback routing: after the provider retries are exhausted, a
+      // configured fallback model gets one more attempt.
+      _setProviderRetryDelayForTesting(0);
+      setCustomEndpointConfig({
+        enabled: true,
+        baseUrl: 'http://localhost:11434/v1',
+        apiKey: '',
+        model: 'primary-model',
+        fallbackModel: 'fallback-model',
+        temperature: 0.7,
+      });
+      for (let i = 0; i < 3; i++) {
+        // $FlowFixMe
+        axios.post.mockRejectedValueOnce({
+          response: {
+            status: 503,
+            data: { error: { message: 'unavailable' } },
+          },
+        });
+      }
+      // $FlowFixMe
+      axios.post.mockResolvedValueOnce({
+        status: 200,
+        data: { choices: [{ message: { role: 'assistant', content: 'ok' } }] },
+      });
+      const result = await customCreateAiRequest({
+        userRequest: 'start',
+        mode: 'chat',
+      });
+      expect(result.status).not.toBe('error');
+      // initial + 2 retries + 1 fallback.
+      expect(axios.post).toHaveBeenCalledTimes(4);
+      expect(axios.post.mock.calls[3][1].model).toBe('fallback-model');
+    });
+
    it('does not retry a real client error (400)', async () => {
      setCustomEndpointConfig({
        enabled: true,
@@ -262,6 +262,7 @@ export type PreferencesValues = {|
  aiCustomBaseUrl: string,
  aiCustomApiKey: string,
  aiCustomModel: string,
+  aiCustomFallbackModel: string,
  aiCustomTemperature: number,
  aiCustomTimeoutMs: number,
  aiCustomStreaming: boolean,
@@ -402,6 +403,7 @@ export type Preferences = {|
  setAiCustomBaseUrl: (baseUrl: string) => void,
  setAiCustomApiKey: (apiKey: string) => void,
  setAiCustomModel: (model: string) => void,
+  setAiCustomFallbackModel: (model: string) => void,
  setAiCustomTemperature: (temperature: number) => void,
  setAiCustomTimeoutMs: (timeoutMs: number) => void,
  setAiCustomStreaming: (streaming: boolean) => void,
@@ -478,6 +480,7 @@ export const initialPreferences = {
    aiCustomBaseUrl: 'http://localhost:11434/v1',
    aiCustomApiKey: '',
    aiCustomModel: 'qwen2.5-coder',
+    aiCustomFallbackModel: '',
    aiCustomTemperature: 0.7,
    aiCustomTimeoutMs: 120000,
    aiCustomStreaming: false,
@@ -581,6 +584,7 @@ export const initialPreferences = {
  setAiCustomBaseUrl: (baseUrl: string) => {},
  setAiCustomApiKey: (apiKey: string) => {},
  setAiCustomModel: (model: string) => {},
+  setAiCustomFallbackModel: (model: string) => {},
  setAiCustomTemperature: (temperature: number) => {},
  setAiCustomTimeoutMs: (timeoutMs: number) => {},
  setAiCustomStreaming: (streaming: boolean) => {},
@@ -453,6 +453,7 @@ const PreferencesDialog = ({
    setAiCustomBaseUrl,
    setAiCustomApiKey,
    setAiCustomModel,
+    setAiCustomFallbackModel,
    setAiCustomTemperature,
    setAiCustomTimeoutMs,
    setAiCustomStreaming,
@@ -1417,6 +1418,15 @@ const PreferencesDialog = ({
            value={values.aiCustomModel}
            onChange={(e, value) => setAiCustomModel(value)}
          />
+          <TextField
+            floatingLabelText={<Trans>Fallback model (optional)</Trans>}
+            helperMarkdownText={i18n._(
+              t`Used once if the primary model keeps returning a transient provider error (502/503/429). Leave empty to disable.`
+            )}
+            fullWidth
+            value={values.aiCustomFallbackModel}
+            onChange={(e, value) => setAiCustomFallbackModel(value)}
+          />
          <TextField
            type="number"
            floatingLabelText={<Trans>Temperature (0.0 to 1.0)</Trans>}
@@ -421,6 +421,8 @@ export default class PreferencesProvider extends React.Component<Props, State> {
    // $FlowFixMe[method-unbinding]
    setAiCustomModel: (this._setAiCustomModel.bind(this): any),
    // $FlowFixMe[method-unbinding]
+    setAiCustomFallbackModel: (this._setAiCustomFallbackModel.bind(this): any),
+    // $FlowFixMe[method-unbinding]
    setAiCustomTemperature: (this._setAiCustomTemperature.bind(this): any),
    // $FlowFixMe[method-unbinding]
    setAiCustomTimeoutMs: (this._setAiCustomTimeoutMs.bind(this): any),
@@ -435,6 +437,7 @@ export default class PreferencesProvider extends React.Component<Props, State> {
      baseUrl: this.state.values.aiCustomBaseUrl,
      apiKey: this.state.values.aiCustomApiKey,
      model: this.state.values.aiCustomModel,
+      fallbackModel: this.state.values.aiCustomFallbackModel,
      temperature: this.state.values.aiCustomTemperature,
      timeoutMs: this.state.values.aiCustomTimeoutMs,
      streaming: this.state.values.aiCustomStreaming,
@@ -1555,6 +1558,21 @@ export default class PreferencesProvider extends React.Component<Props, State> {
    );
  }

+  _setAiCustomFallbackModel(newValue: string) {
+    this.setState(
+      state => ({
+        values: {
+          ...state.values,
+          aiCustomFallbackModel: newValue,
+        },
+      }),
+      () => {
+        this._persistValuesToLocalStorage(this.state);
+        setCustomEndpointConfig({ fallbackModel: newValue });
+      }
+    );
+  }
+
  _setAiCustomTemperature(newValue: number) {
    this.setState(
      state => ({
@@ -98,6 +98,18 @@ work; `endpoint` is the AI endpoint under test for that cycle.
  looping until the guard tripped (cycles 239-241)..

 ## Proven workflows
+- Tier 2 fallback routing (cycle 262): a configurable secondary model used when
+  the primary model exhausts its provider retries. `CustomAIConfig.fallbackModel`
+  (sanitized), a "Fallback model (optional)" field in the AI Settings
+  Preferences (context/provider/dialog wired like `aiCustomModel`; the key store
+  strips only the API key, so it persists), and `runTurnWithProviderRetry` now
+  makes one final attempt with the fallback model after the 2 provider retries
+  fail. The three turn paths (create/continue/sub-agent) accept a per-attempt
+  model override. Verified live with a model-aware provider: `primary-model →
+  503` ×3, then `fallback-model → 200`, the chat request ended `ready`, and the
+  console logged the retries then `Provider still failing (503); falling back to
+  model "fallback-model"`; the Preferences field renders. +1 spec (3×503 then a
+  fallback 200 ⇒ 4 posts, 4th uses the fallback model).
 - Adversarial-invariant audit, clean (cycle 261): repo-wide scan for hardcoded
  secrets (`sk-…`, `Bearer …`, long `api_key` literals), provider locks
  (openrouter/openai/deepseek/anthropic hostnames) in the fork AI surface, and