mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-11 01:51:34 -10:00
feat(models): refresh configured catalogs and load Codex account models
Add models --update/--refresh with exact provider scoping, durable endpoint catalogs, partial failure receipts, safe offline listing, and picker reuse. Query Codex through its documented stdio account/read and model/list API; preserve pagination and supported efforts while distinguishing observation time from upstream fetch time. Do not import credentials or run inference. Validation (macOS, local, locked/offline Cargo dependencies): - catalog: 172 passed, 0 failed, 2 ignored - provider_lake: 24 passed, 0 failed - codex_model_cache: 14 passed, 0 failed - model_picker: 15 passed, 0 failed - provider_picker: 142 passed, 0 failed - localization: 49 passed, 0 failed - 15 locales x 6 additions: 90 key/placeholder checks passed - CLI check/build, cargo fmt check, and git diff --check passed - CLI fixture smoke passed: offline reuse, scoped refresh, partial 403 retention, alias/help, unchanged config, and secret-free caches - Installed Codex 0.150.1 account listing succeeded: 7 visible models and matching offline roster. No exact gpt-6-astra ID was advertised. The model/list API exposes no force-refresh field, so receipts say loaded. Requested root npm gate could not run: npm test and npm run check:web both fail with Missing script (2 command failures, no npm tests executed). Rust linker warned that __eh_frame exceeds compact unwind offsets. No hosted CI, inference, public push, or deployment was performed.
This commit is contained in:
CodeWhale Bot committed
1 parent
8da0d9e4dc
commit
83c890658c
25 files changed
+1724
-107
No files matched your search
@@ -188,7 +188,10 @@ enum Commands {
|
||||
Run(RunArgs),
|
||||
/// Run Codewhale diagnostics.
|
||||
Doctor(TuiPassthroughArgs),
|
||||
/// List live models from the selected provider.
|
||||
/// List cached models; use --update to refresh configured provider catalogs.
|
||||
#[command(
|
||||
after_help = "Examples:\n codewhale models --update\n codewhale models --update --provider openai\n codewhale models --provider openai-codex --json\n\n--update (alias: --refresh) refreshes configured provider catalogs. --provider ID limits the scope."
|
||||
)]
|
||||
Models(TuiPassthroughArgs),
|
||||
/// Generate speech audio with Xiaomi MiMo TTS models.
|
||||
#[command(visible_alias = "tts")]
|
||||
@@ -1944,7 +1947,8 @@ fn run() -> Result<()> {
|
||||
run_tui_in_process(&cli, &resolved_runtime, tui_args("doctor", args))
|
||||
}
|
||||
Some(Commands::Models(args)) => {
|
||||
let resolved_runtime = resolve_runtime_for_dispatch(&mut store, &runtime_overrides);
|
||||
let resolved_runtime =
|
||||
resolve_runtime_for_diagnostic_dispatch(&store, &runtime_overrides);
|
||||
run_tui_in_process(&cli, &resolved_runtime, tui_args("models", args))
|
||||
}
|
||||
Some(Commands::Speech(args)) => {
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Canvia els permisos o obre el selector de mode",
|
||||
"CmdModelDescription": "Canvia o mostra el model actual",
|
||||
"CmdModelsDescription": "Obtén els IDs de model en viu de l'API activa",
|
||||
"ModelsListHeader": "Models de {provider} (predeterminat: {model})",
|
||||
"ModelsListHint": "Catàleg en memòria cau; executa `codewhale models --update` per actualitzar els proveïdors configurats.",
|
||||
"ModelsUpdateSummary": "Catàlegs de models: {updated} actualitzats, {loaded} carregats des de Codex, {failed} amb errors, {skipped} omesos.",
|
||||
"ModelsUpdatePartial": "No s’han pogut actualitzar alguns catàlegs. Els models anteriors i configurats continuen disponibles.",
|
||||
"ModelsCodexHint": "Els models de ChatGPT es carreguen a través de Codex. L’hora de consulta no confirma una actualització del catàleg del proveïdor.",
|
||||
"ModelsSourceFallback": "Models inclosos o configurats; disponibilitat no verificada.",
|
||||
"CmdModelDbDescription": "Explora la base de dades de models inclosa",
|
||||
"CmdNetworkDescription": "Gestiona les regles de xarxa de permís i denegació",
|
||||
"CmdUpdateDescription": "Comprova i instal·la una nova versió de Codewhale",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Berechtigungen wechseln oder die Modusauswahl öffnen",
|
||||
"CmdModelDescription": "Modell wechseln oder aktuelles anzeigen",
|
||||
"CmdModelsDescription": "Live-Modell-IDs von der aktiven API abrufen",
|
||||
"ModelsListHeader": "Modelle von {provider} (Standard: {model})",
|
||||
"ModelsListHint": "Zwischengespeicherter Katalog; mit `codewhale models --update` die eingerichteten Anbieter aktualisieren.",
|
||||
"ModelsUpdateSummary": "Modellkataloge: {updated} aktualisiert, {loaded} aus Codex geladen, {failed} fehlgeschlagen, {skipped} übersprungen.",
|
||||
"ModelsUpdatePartial": "Einige Kataloge konnten nicht aktualisiert werden. Bisherige und konfigurierte Modelle bleiben verfügbar.",
|
||||
"ModelsCodexHint": "ChatGPT-Modelle werden über Codex geladen. Der Abfragezeitpunkt bestätigt keine Aktualisierung des Anbieterkatalogs.",
|
||||
"ModelsSourceFallback": "Mitgelieferte oder konfigurierte Modelle; Verfügbarkeit nicht geprüft.",
|
||||
"CmdModelDbDescription": "Mitgelieferte Modelldatenbank durchsuchen",
|
||||
"CmdNetworkDescription": "Netzwerk-Allow- und Deny-Regeln verwalten",
|
||||
"CmdUpdateDescription": "Neue Codewhale-Version suchen und installieren",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "Switch permissions or open the mode picker",
|
||||
"CmdModelDescription": "Switch or view current model",
|
||||
"CmdModelsDescription": "Fetch live model IDs from the active API",
|
||||
"ModelsListHeader": "{provider} models (default: {model})",
|
||||
"ModelsListHint": "Cached catalog; run `codewhale models --update` to refresh configured providers.",
|
||||
"ModelsUpdateSummary": "Model catalogs: {updated} updated, {loaded} loaded from Codex, {failed} failed, {skipped} skipped.",
|
||||
"ModelsUpdatePartial": "Some catalogs could not be refreshed. Previous and configured models remain available.",
|
||||
"ModelsCodexHint": "ChatGPT models are loaded through Codex. The lookup time does not confirm an upstream catalog refresh.",
|
||||
"ModelsSourceFallback": "Bundled or configured models; availability not verified.",
|
||||
"CmdModelDbDescription": "Browse the bundled model database",
|
||||
"CmdNetworkDescription": "Manage network allow and deny rules",
|
||||
"CmdUpdateDescription": "Check for and install a new Codewhale release",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "Cambiar permisos o abrir el selector de modo",
|
||||
"CmdModelDescription": "Cambiar o mostrar el modelo actual",
|
||||
"CmdModelsDescription": "Obtener IDs de modelo en vivo de la API activa",
|
||||
"ModelsListHeader": "Modelos de {provider} (predeterminado: {model})",
|
||||
"ModelsListHint": "Catálogo en caché; ejecuta `codewhale models --update` para actualizar los proveedores configurados.",
|
||||
"ModelsUpdateSummary": "Catálogos de modelos: {updated} actualizados, {loaded} cargados desde Codex, {failed} fallidos, {skipped} omitidos.",
|
||||
"ModelsUpdatePartial": "No se pudieron actualizar algunos catálogos. Los modelos anteriores y configurados siguen disponibles.",
|
||||
"ModelsCodexHint": "Los modelos de ChatGPT se cargan mediante Codex. La hora de consulta no confirma una actualización del catálogo del proveedor.",
|
||||
"ModelsSourceFallback": "Modelos incluidos o configurados; disponibilidad sin verificar.",
|
||||
"CmdModelDbDescription": "Explorar la base de datos de modelos integrada",
|
||||
"CmdNetworkDescription": "Gestionar reglas de red permitidas y bloqueadas",
|
||||
"CmdUpdateDescription": "Buscar e instalar una nueva versión de Codewhale",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Changer les permissions ou ouvrir le sélecteur de mode",
|
||||
"CmdModelDescription": "Changer ou afficher le modèle actuel",
|
||||
"CmdModelsDescription": "Récupérer les IDs de modèles en direct depuis l'API active",
|
||||
"ModelsListHeader": "Modèles de {provider} (par défaut : {model})",
|
||||
"ModelsListHint": "Catalogue en cache ; lancez `codewhale models --update` pour actualiser les fournisseurs configurés.",
|
||||
"ModelsUpdateSummary": "Catalogues de modèles : {updated} actualisés, {loaded} chargés depuis Codex, {failed} en échec, {skipped} ignorés.",
|
||||
"ModelsUpdatePartial": "Certains catalogues n’ont pas pu être actualisés. Les modèles précédents et configurés restent disponibles.",
|
||||
"ModelsCodexHint": "Les modèles ChatGPT sont chargés via Codex. L’heure de consultation ne confirme pas une actualisation du catalogue du fournisseur.",
|
||||
"ModelsSourceFallback": "Modèles fournis ou configurés ; disponibilité non vérifiée.",
|
||||
"CmdModelDbDescription": "Parcourir la base de modèles intégrée",
|
||||
"CmdNetworkDescription": "Gérer les règles réseau d'autorisation et de refus",
|
||||
"CmdUpdateDescription": "Rechercher et installer une nouvelle version de Codewhale",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "अनुमतियाँ बदलें या मोड पिकर खोलें",
|
||||
"CmdModelDescription": "वर्तमान मॉडल बदलें या देखें",
|
||||
"CmdModelsDescription": "सक्रिय API से लाइव मॉडल ID लाएँ",
|
||||
"ModelsListHeader": "{provider} के मॉडल (डिफ़ॉल्ट: {model})",
|
||||
"ModelsListHint": "कैश में सहेजी सूची; कॉन्फ़िगर किए प्रदाताओं को अपडेट करने के लिए `codewhale models --update` चलाएँ।",
|
||||
"ModelsUpdateSummary": "मॉडल कैटलॉग: {updated} अपडेट हुए, {loaded} Codex से लोड हुए, {failed} विफल हुए, {skipped} छोड़े गए।",
|
||||
"ModelsUpdatePartial": "कुछ सूचियाँ अपडेट नहीं हो सकीं। पिछले और कॉन्फ़िगर किए मॉडल उपलब्ध हैं।",
|
||||
"ModelsCodexHint": "ChatGPT मॉडल Codex के ज़रिए लोड होते हैं। खोज का समय प्रदाता के कैटलॉग के अपडेट होने की पुष्टि नहीं करता।",
|
||||
"ModelsSourceFallback": "शामिल या कॉन्फ़िगर किए मॉडल; उपलब्धता सत्यापित नहीं है।",
|
||||
"CmdModelDbDescription": "बंडल किया मॉडल डेटाबेस ब्राउज़ करें",
|
||||
"CmdNetworkDescription": "नेटवर्क अनुमति और निषेध नियम प्रबंधित करें",
|
||||
"CmdUpdateDescription": "नया Codewhale रिलीज़ जाँचें और इंस्टॉल करें",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Ganti izin atau buka pemilih mode",
|
||||
"CmdModelDescription": "Ganti atau lihat model saat ini",
|
||||
"CmdModelsDescription": "Ambil ID model langsung dari API aktif",
|
||||
"ModelsListHeader": "Model {provider} (bawaan: {model})",
|
||||
"ModelsListHint": "Katalog tersimpan; jalankan `codewhale models --update` untuk memperbarui penyedia yang dikonfigurasi.",
|
||||
"ModelsUpdateSummary": "Katalog model: {updated} diperbarui, {loaded} dimuat dari Codex, {failed} gagal, {skipped} dilewati.",
|
||||
"ModelsUpdatePartial": "Beberapa katalog tidak dapat diperbarui. Model sebelumnya dan yang dikonfigurasi tetap tersedia.",
|
||||
"ModelsCodexHint": "Model ChatGPT dimuat melalui Codex. Waktu pencarian tidak memastikan katalog penyedia telah diperbarui.",
|
||||
"ModelsSourceFallback": "Model bawaan atau yang dikonfigurasi; ketersediaan belum diverifikasi.",
|
||||
"CmdModelDbDescription": "Telusuri database model bawaan",
|
||||
"CmdNetworkDescription": "Kelola aturan izinkan dan tolak jaringan",
|
||||
"CmdUpdateDescription": "Periksa dan pasang rilis Codewhale baru",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "権限を切り替えるか、モードピッカーを開く",
|
||||
"CmdModelDescription": "現在のモデルを切り替え・確認",
|
||||
"CmdModelsDescription": "アクティブな API からライブのモデル ID を取得する",
|
||||
"ModelsListHeader": "{provider} のモデル（既定: {model}）",
|
||||
"ModelsListHint": "キャッシュ済みカタログ。設定済みプロバイダーを更新するには `codewhale models --update` を実行してください。",
|
||||
"ModelsUpdateSummary": "モデルカタログ: {updated} 件更新、{loaded} 件を Codex から読み込み、{failed} 件失敗、{skipped} 件スキップ。",
|
||||
"ModelsUpdatePartial": "一部のカタログを更新できませんでした。以前のモデルと設定済みモデルは引き続き利用できます。",
|
||||
"ModelsCodexHint": "ChatGPT モデルは Codex 経由で読み込みます。照会時刻はプロバイダーのカタログ更新を保証するものではありません。",
|
||||
"ModelsSourceFallback": "同梱または設定済みのモデル。利用可能かどうかは未確認です。",
|
||||
"CmdModelDbDescription": "内蔵のモデルデータベースを閲覧する",
|
||||
"CmdNetworkDescription": "ネットワーク許可・拒否ルールを管理",
|
||||
"CmdUpdateDescription": "新しい Codewhale リリースを確認してインストールします",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "권한 전환 또는 모드 선택기 열기",
|
||||
"CmdModelDescription": "현재 모델을 전환하거나 확인합니다",
|
||||
"CmdModelsDescription": "현재 API에서 실시간 모델 ID를 가져옵니다",
|
||||
"ModelsListHeader": "{provider} 모델 (기본값: {model})",
|
||||
"ModelsListHint": "캐시된 카탈로그입니다. 설정된 제공업체를 갱신하려면 `codewhale models --update`를 실행하세요.",
|
||||
"ModelsUpdateSummary": "모델 카탈로그: {updated}개 업데이트, {loaded}개 Codex에서 불러옴, {failed}개 실패, {skipped}개 건너뜀.",
|
||||
"ModelsUpdatePartial": "일부 카탈로그를 갱신하지 못했습니다. 이전 모델과 설정된 모델은 계속 사용할 수 있습니다.",
|
||||
"ModelsCodexHint": "ChatGPT 모델은 Codex를 통해 불러옵니다. 조회 시각이 공급자 카탈로그의 갱신을 보장하지는 않습니다.",
|
||||
"ModelsSourceFallback": "기본 제공 또는 설정된 모델입니다. 사용 가능 여부는 확인되지 않았습니다.",
|
||||
"CmdModelDbDescription": "내장 모델 데이터베이스를 조회합니다",
|
||||
"CmdNetworkDescription": "네트워크 허용/차단 규칙을 관리합니다",
|
||||
"CmdUpdateDescription": "새 Codewhale 릴리스를 확인하고 설치합니다",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "Alternar permissões ou abrir o seletor de modo",
|
||||
"CmdModelDescription": "Trocar ou exibir o modelo atual",
|
||||
"CmdModelsDescription": "Buscar IDs de modelo ao vivo da API ativa",
|
||||
"ModelsListHeader": "Modelos de {provider} (padrão: {model})",
|
||||
"ModelsListHint": "Catálogo em cache; execute `codewhale models --update` para atualizar os provedores configurados.",
|
||||
"ModelsUpdateSummary": "Catálogos de modelos: {updated} atualizados, {loaded} carregados do Codex, {failed} com falha, {skipped} ignorados.",
|
||||
"ModelsUpdatePartial": "Não foi possível atualizar alguns catálogos. Os modelos anteriores e configurados continuam disponíveis.",
|
||||
"ModelsCodexHint": "Os modelos do ChatGPT são carregados pelo Codex. O horário da consulta não confirma uma atualização do catálogo do provedor.",
|
||||
"ModelsSourceFallback": "Modelos incluídos ou configurados; disponibilidade não verificada.",
|
||||
"CmdModelDbDescription": "Navegar pelo banco de dados de modelos integrado",
|
||||
"CmdNetworkDescription": "Gerenciar regras de rede permitidas e bloqueadas",
|
||||
"CmdUpdateDescription": "Verificar e instalar uma nova versão do Codewhale",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Переключить разрешения или открыть выбор режима",
|
||||
"CmdModelDescription": "Сменить или показать текущую модель",
|
||||
"CmdModelsDescription": "Получить актуальные ID моделей из активного API",
|
||||
"ModelsListHeader": "Модели {provider} (по умолчанию: {model})",
|
||||
"ModelsListHint": "Сохранённый каталог; выполните `codewhale models --update`, чтобы обновить настроенных провайдеров.",
|
||||
"ModelsUpdateSummary": "Каталоги моделей: обновлено {updated}, загружено из Codex {loaded}, ошибок {failed}, пропущено {skipped}.",
|
||||
"ModelsUpdatePartial": "Не удалось обновить некоторые каталоги. Предыдущие и настроенные модели остаются доступны.",
|
||||
"ModelsCodexHint": "Модели ChatGPT загружаются через Codex. Время запроса не подтверждает обновление каталога у провайдера.",
|
||||
"ModelsSourceFallback": "Встроенные или настроенные модели; доступность не проверена.",
|
||||
"CmdModelDbDescription": "Просмотр встроенной базы моделей",
|
||||
"CmdNetworkDescription": "Управление правилами разрешения и запрета сети",
|
||||
"CmdUpdateDescription": "Проверить и установить новый выпуск Codewhale",
|
||||
|
||||
@@ -483,6 +483,12 @@
|
||||
"CmdModeDescription": "Перемкнути дозволи або відкрити вибір режиму",
|
||||
"CmdModelDescription": "Змінити або переглянути поточну модель",
|
||||
"CmdModelsDescription": "Отримати актуальні ID моделей з активного API",
|
||||
"ModelsListHeader": "Моделі {provider} (типова: {model})",
|
||||
"ModelsListHint": "Збережений каталог; виконайте `codewhale models --update`, щоб оновити налаштованих провайдерів.",
|
||||
"ModelsUpdateSummary": "Каталоги моделей: оновлено {updated}, завантажено з Codex {loaded}, помилок {failed}, пропущено {skipped}.",
|
||||
"ModelsUpdatePartial": "Не вдалося оновити деякі каталоги. Попередні та налаштовані моделі залишаються доступними.",
|
||||
"ModelsCodexHint": "Моделі ChatGPT завантажуються через Codex. Час запиту не підтверджує оновлення каталогу постачальника.",
|
||||
"ModelsSourceFallback": "Вбудовані або налаштовані моделі; доступність не перевірено.",
|
||||
"CmdModelDbDescription": "Переглянути вбудовану базу моделей",
|
||||
"CmdNetworkDescription": "Керувати мережевими правилами дозволу та заборони",
|
||||
"CmdUpdateDescription": "Перевірити та встановити новий випуск Codewhale",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "Chuyển quyền hạn hoặc mở trình chọn chế độ",
|
||||
"CmdModelDescription": "Chuyển đổi hoặc xem mô hình AI hiện tại",
|
||||
"CmdModelsDescription": "Lấy ID mô hình trực tiếp từ API đang dùng",
|
||||
"ModelsListHeader": "Mô hình của {provider} (mặc định: {model})",
|
||||
"ModelsListHint": "Danh mục đã lưu; chạy `codewhale models --update` để cập nhật các nhà cung cấp đã cấu hình.",
|
||||
"ModelsUpdateSummary": "Danh mục mô hình: {updated} đã cập nhật, {loaded} đã tải từ Codex, {failed} thất bại, {skipped} đã bỏ qua.",
|
||||
"ModelsUpdatePartial": "Không thể cập nhật một số danh mục. Các mô hình trước đó và đã cấu hình vẫn có sẵn.",
|
||||
"ModelsCodexHint": "Các mô hình ChatGPT được tải qua Codex. Thời điểm truy vấn không xác nhận danh mục của nhà cung cấp đã được cập nhật.",
|
||||
"ModelsSourceFallback": "Mô hình đi kèm hoặc đã cấu hình; chưa xác minh tính khả dụng.",
|
||||
"CmdModelDbDescription": "Duyệt cơ sở dữ liệu mô hình tích hợp",
|
||||
"CmdNetworkDescription": "Quản lý các quy tắc cho phép và từ chối mạng",
|
||||
"CmdUpdateDescription": "Kiểm tra và cài đặt bản phát hành Codewhale mới",
|
||||
|
||||
@@ -486,6 +486,12 @@
|
||||
"CmdModeDescription": "切换权限或打开模式选择器",
|
||||
"CmdModelDescription": "切换或查看当前模型",
|
||||
"CmdModelsDescription": "工具箱：从当前 API 获取实时模型 ID",
|
||||
"ModelsListHeader": "{provider} 的模型（默认：{model}）",
|
||||
"ModelsListHint": "已缓存的目录；运行 `codewhale models --update` 可更新已配置的提供商。",
|
||||
"ModelsUpdateSummary": "模型目录：{updated} 个已更新，{loaded} 个从 Codex 加载，{failed} 个失败，{skipped} 个已跳过。",
|
||||
"ModelsUpdatePartial": "部分目录无法更新。此前的模型和已配置的模型仍然可用。",
|
||||
"ModelsCodexHint": "ChatGPT 模型通过 Codex 加载。查询时间并不代表提供商已刷新模型目录。",
|
||||
"ModelsSourceFallback": "内置或已配置的模型；尚未验证可用性。",
|
||||
"CmdModelDbDescription": "参考：浏览内置的模型数据库",
|
||||
"CmdNetworkDescription": "管理网络允许和拒绝规则",
|
||||
"CmdUpdateDescription": "检查并安装新的 Codewhale 版本",
|
||||
|
||||
@@ -359,6 +359,12 @@
|
||||
"CmdModelDbDescription": "參考：瀏覽內置的模型資料庫",
|
||||
"CmdModelDescription": "切換或檢視目前模型",
|
||||
"CmdModelsDescription": "工具箱：從目前 API 取得實時模型 ID",
|
||||
"ModelsListHeader": "{provider} 的模型（預設：{model}）",
|
||||
"ModelsListHint": "已快取的目錄；執行 `codewhale models --update` 可更新已設定的提供者。",
|
||||
"ModelsUpdateSummary": "模型目錄：{updated} 個已更新，{loaded} 個從 Codex 載入，{failed} 個失敗，{skipped} 個已略過。",
|
||||
"ModelsUpdatePartial": "部分目錄無法更新。先前的模型和已設定的模型仍然可用。",
|
||||
"ModelsCodexHint": "ChatGPT 模型透過 Codex 載入。查詢時間並不代表供應商已重新整理模型目錄。",
|
||||
"ModelsSourceFallback": "內建或已設定的模型；尚未驗證可用性。",
|
||||
"CmdNetworkDescription": "管理網路允許和拒絕規則",
|
||||
"CmdUpdateDescription": "檢查並安裝新的 Codewhale 版本",
|
||||
"CmdNewDescription": "開始一個新的已儲存工作階段",
|
||||
|
||||
@@ -2497,10 +2497,27 @@ impl DeepSeekClient {
|
||||
});
|
||||
}
|
||||
|
||||
let body = response
|
||||
.text()
|
||||
// Catalogs are untrusted remote data, retained locally by `models --update`.
|
||||
const MAX_CATALOG_BYTES: usize = 8 * 1024 * 1024;
|
||||
if response
|
||||
.content_length()
|
||||
.is_some_and(|size| size > MAX_CATALOG_BYTES as u64)
|
||||
{
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
let mut response = response;
|
||||
let mut body = Vec::new();
|
||||
while let Some(chunk) = response
|
||||
.chunk()
|
||||
.await
|
||||
.map_err(|_| CatalogRefreshError::Network)?;
|
||||
.map_err(|_| CatalogRefreshError::Network)?
|
||||
{
|
||||
if body.len().saturating_add(chunk.len()) > MAX_CATALOG_BYTES {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
body.extend_from_slice(&chunk);
|
||||
}
|
||||
let body = String::from_utf8(body).map_err(|_| CatalogRefreshError::InvalidResponse)?;
|
||||
|
||||
let provider = self.catalog_provider_id();
|
||||
let fingerprint = base_url_fingerprint(&self.base_url);
|
||||
@@ -2580,6 +2597,15 @@ impl DeepSeekClient {
|
||||
.collect()
|
||||
};
|
||||
|
||||
if offerings.iter().any(|row| {
|
||||
!crate::provider_lake::valid_catalog_model_id(&row.wire_model_id)
|
||||
|| self
|
||||
.model_bound_secret_values
|
||||
.iter()
|
||||
.any(|secret| !secret.is_empty() && row.wire_model_id.contains(secret))
|
||||
}) {
|
||||
return Err(CatalogRefreshError::InvalidResponse);
|
||||
}
|
||||
Ok(ProviderCatalogDelta {
|
||||
provider,
|
||||
base_url_fingerprint: fingerprint,
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
//! Secret-free OpenAI Codex / ChatGPT OAuth model roster discovery.
|
||||
//!
|
||||
//! The Codex CLI keeps its account-scoped roster in `models_cache.json`.
|
||||
//! CodeWhale reads only the cache timestamp and model identifiers; it never
|
||||
//! opens the adjacent OAuth credential file and never logs cache contents.
|
||||
//! CodeWhale reads model metadata only. An explicit update can ask the Codex
|
||||
//! CLI's documented stdio API for its ChatGPT account roster; credentials stay
|
||||
//! with Codex. The query time is not proof of an upstream catalog refresh.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Stdio;
|
||||
use std::sync::Mutex;
|
||||
use std::time::SystemTime;
|
||||
|
||||
@@ -14,7 +16,9 @@ use std::time::SystemTime;
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
|
||||
use chrono::{DateTime, Duration, Utc};
|
||||
use serde::Deserialize;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Value, json};
|
||||
use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
|
||||
|
||||
use crate::config::DEFAULT_OPENAI_CODEX_MODEL;
|
||||
|
||||
@@ -51,9 +55,11 @@ pub(crate) struct CodexModelRoster {
|
||||
pub(crate) models: Vec<CodexModelMetadata>,
|
||||
pub(crate) freshness: CodexModelCacheFreshness,
|
||||
pub(crate) fetched_at: Option<DateTime<Utc>>,
|
||||
pub(crate) observed_at: Option<DateTime<Utc>>,
|
||||
pub(crate) source: &'static str,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub(crate) struct CodexModelMetadata {
|
||||
pub(crate) id: String,
|
||||
pub(crate) context_window: Option<u32>,
|
||||
@@ -76,6 +82,8 @@ impl CodexModelRoster {
|
||||
}],
|
||||
freshness,
|
||||
fetched_at,
|
||||
observed_at: None,
|
||||
source: "codex_cli_cache",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -156,22 +164,20 @@ pub(crate) fn codex_home_path() -> PathBuf {
|
||||
/// The key is the resolved path plus the file's mtime and length, so a Codex
|
||||
/// refresh is picked up on the next call and a test pointing `CODEX_HOME`
|
||||
/// somewhere else never reads another test's entry.
|
||||
type RosterCacheKey = (PathBuf, Option<SystemTime>, u64);
|
||||
type RosterCacheKey = Vec<(PathBuf, Option<SystemTime>, u64)>;
|
||||
static ROSTER_MEMO: Mutex<Option<(RosterCacheKey, CodexModelRoster)>> = Mutex::new(None);
|
||||
|
||||
#[must_use]
|
||||
pub(crate) fn model_roster() -> CodexModelRoster {
|
||||
let home = codex_home_path();
|
||||
let path = home.join(MODEL_CACHE_FILE);
|
||||
let key: Option<RosterCacheKey> = std::fs::symlink_metadata(&path)
|
||||
.ok()
|
||||
.map(|metadata| (path, metadata.modified().ok(), metadata.len()));
|
||||
|
||||
// A missing or unreadable cache is cheap to re-check, so it stays uncached
|
||||
// and keeps reporting live freshness.
|
||||
let Some(key) = key else {
|
||||
return load_model_roster_from_home_at(&home, Utc::now());
|
||||
};
|
||||
let snapshot_path = cli_snapshot_path(&home);
|
||||
let key: RosterCacheKey = std::iter::once(home.join(MODEL_CACHE_FILE))
|
||||
.chain(snapshot_path.iter().cloned())
|
||||
.map(|path| match std::fs::symlink_metadata(&path) {
|
||||
Ok(metadata) => (path, metadata.modified().ok(), metadata.len()),
|
||||
Err(_) => (path, None, 0),
|
||||
})
|
||||
.collect();
|
||||
|
||||
// A `Fresh` roster ages into `Stale` on the clock, not on a file change,
|
||||
// so the memo is only trusted while the entry it was built from still
|
||||
@@ -182,12 +188,20 @@ pub(crate) fn model_roster() -> CodexModelRoster {
|
||||
&& roster.freshness == CodexModelCacheFreshness::Fresh
|
||||
&& roster
|
||||
.fetched_at
|
||||
.or(roster.observed_at)
|
||||
.is_some_and(|fetched| Utc::now().signed_duration_since(fetched) <= MODEL_CACHE_MAX_AGE)
|
||||
{
|
||||
return roster.clone();
|
||||
}
|
||||
|
||||
let roster = load_model_roster_from_home_at(&home, Utc::now());
|
||||
let now = Utc::now();
|
||||
let mut roster = load_model_roster_from_home_at(&home, now);
|
||||
if let Some(snapshot) = snapshot_path.and_then(|path| load_cli_snapshot(&path, now))
|
||||
&& (roster.freshness != CodexModelCacheFreshness::Fresh
|
||||
|| snapshot.observed_at > roster.fetched_at)
|
||||
{
|
||||
roster = snapshot;
|
||||
}
|
||||
if let Ok(mut memo) = ROSTER_MEMO.lock() {
|
||||
*memo = Some((key, roster.clone()));
|
||||
}
|
||||
@@ -196,38 +210,10 @@ pub(crate) fn model_roster() -> CodexModelRoster {
|
||||
|
||||
fn load_model_roster_from_home_at(home: &Path, now: DateTime<Utc>) -> CodexModelRoster {
|
||||
let path = home.join(MODEL_CACHE_FILE);
|
||||
let path_metadata = match std::fs::symlink_metadata(&path) {
|
||||
Ok(metadata) => metadata,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
|
||||
return CodexModelRoster::fallback(CodexModelCacheFreshness::Missing, None);
|
||||
}
|
||||
Err(_) => return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None),
|
||||
let bytes = match read_cache_bytes(&path) {
|
||||
Ok(bytes) => bytes,
|
||||
Err(freshness) => return CodexModelRoster::fallback(freshness, None),
|
||||
};
|
||||
if !path_metadata.file_type().is_file() || path_metadata.len() > MAX_MODEL_CACHE_BYTES {
|
||||
return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None);
|
||||
}
|
||||
let mut file = match open_cache_file(&path) {
|
||||
Ok(file) => file,
|
||||
Err(_) => return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None),
|
||||
};
|
||||
let metadata = match file.metadata() {
|
||||
Ok(metadata) => metadata,
|
||||
Err(_) => return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None),
|
||||
};
|
||||
if !metadata.file_type().is_file() || metadata.len() > MAX_MODEL_CACHE_BYTES {
|
||||
return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None);
|
||||
}
|
||||
|
||||
let mut bytes = Vec::with_capacity(metadata.len().min(MAX_MODEL_CACHE_BYTES) as usize);
|
||||
if file
|
||||
.by_ref()
|
||||
.take(MAX_MODEL_CACHE_BYTES + 1)
|
||||
.read_to_end(&mut bytes)
|
||||
.is_err()
|
||||
|| bytes.len() as u64 > MAX_MODEL_CACHE_BYTES
|
||||
{
|
||||
return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None);
|
||||
}
|
||||
let cache: CacheFile = match serde_json::from_slice(&bytes) {
|
||||
Ok(cache) => cache,
|
||||
Err(_) => return CodexModelRoster::fallback(CodexModelCacheFreshness::Invalid, None),
|
||||
@@ -254,8 +240,6 @@ fn load_model_roster_from_home_at(home: &Path, now: DateTime<Utc>) -> CodexModel
|
||||
if !valid_model_id(slug) {
|
||||
continue;
|
||||
}
|
||||
// A roster entry the vendor hides is not a model the operator can
|
||||
// pick. Listing it puts a route in the picker that is not on offer.
|
||||
if model
|
||||
.visibility
|
||||
.as_deref()
|
||||
@@ -301,9 +285,47 @@ fn load_model_roster_from_home_at(home: &Path, now: DateTime<Utc>) -> CodexModel
|
||||
models,
|
||||
freshness: CodexModelCacheFreshness::Fresh,
|
||||
fetched_at: Some(cache.fetched_at),
|
||||
observed_at: None,
|
||||
source: "codex_cli_cache",
|
||||
}
|
||||
}
|
||||
|
||||
fn read_cache_bytes(path: &Path) -> Result<Vec<u8>, CodexModelCacheFreshness> {
|
||||
let path_metadata = match std::fs::symlink_metadata(&path) {
|
||||
Ok(metadata) => metadata,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Err(CodexModelCacheFreshness::Missing);
|
||||
}
|
||||
Err(_) => return Err(CodexModelCacheFreshness::Invalid),
|
||||
};
|
||||
if !path_metadata.file_type().is_file() || path_metadata.len() > MAX_MODEL_CACHE_BYTES {
|
||||
return Err(CodexModelCacheFreshness::Invalid);
|
||||
}
|
||||
let mut file = match open_cache_file(&path) {
|
||||
Ok(file) => file,
|
||||
Err(_) => return Err(CodexModelCacheFreshness::Invalid),
|
||||
};
|
||||
let metadata = match file.metadata() {
|
||||
Ok(metadata) => metadata,
|
||||
Err(_) => return Err(CodexModelCacheFreshness::Invalid),
|
||||
};
|
||||
if !metadata.file_type().is_file() || metadata.len() > MAX_MODEL_CACHE_BYTES {
|
||||
return Err(CodexModelCacheFreshness::Invalid);
|
||||
}
|
||||
|
||||
let mut bytes = Vec::with_capacity(metadata.len().min(MAX_MODEL_CACHE_BYTES) as usize);
|
||||
if file
|
||||
.by_ref()
|
||||
.take(MAX_MODEL_CACHE_BYTES + 1)
|
||||
.read_to_end(&mut bytes)
|
||||
.is_err()
|
||||
|| bytes.len() as u64 > MAX_MODEL_CACHE_BYTES
|
||||
{
|
||||
return Err(CodexModelCacheFreshness::Invalid);
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
fn open_cache_file(path: &Path) -> std::io::Result<std::fs::File> {
|
||||
let mut options = std::fs::OpenOptions::new();
|
||||
options.read(true);
|
||||
@@ -321,6 +343,332 @@ fn valid_model_id(value: &str) -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
struct CliSnapshot {
|
||||
observed_at: DateTime<Utc>,
|
||||
models: Vec<CodexModelMetadata>,
|
||||
}
|
||||
|
||||
fn cli_snapshot_path(home: &Path) -> Option<PathBuf> {
|
||||
let catalog_path = crate::models_dev_live::cache_path()?;
|
||||
let identity = codewhale_config::catalog::base_url_fingerprint(&home.to_string_lossy());
|
||||
Some(
|
||||
catalog_path
|
||||
.parent()?
|
||||
.join(format!("codex-{identity}.json")),
|
||||
)
|
||||
}
|
||||
|
||||
fn valid_effort(effort: &str) -> bool {
|
||||
!effort.is_empty()
|
||||
&& effort.len() <= 32
|
||||
&& effort
|
||||
.bytes()
|
||||
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'_' | b'-'))
|
||||
}
|
||||
|
||||
fn load_cli_snapshot(path: &Path, now: DateTime<Utc>) -> Option<CodexModelRoster> {
|
||||
let snapshot: CliSnapshot = serde_json::from_slice(&read_cache_bytes(path).ok()?).ok()?;
|
||||
let age = now.signed_duration_since(snapshot.observed_at);
|
||||
if age < -MAX_FUTURE_CLOCK_SKEW
|
||||
|| age > MODEL_CACHE_MAX_AGE
|
||||
|| snapshot.models.is_empty()
|
||||
|| snapshot.models.iter().any(|model| {
|
||||
!valid_model_id(&model.id)
|
||||
|| model.efforts.len() > 16
|
||||
|| model.efforts.iter().any(|effort| !valid_effort(effort))
|
||||
|| model
|
||||
.context_window
|
||||
.is_some_and(|window| !(1..=16_000_000).contains(&window))
|
||||
})
|
||||
{
|
||||
return None;
|
||||
}
|
||||
Some(CodexModelRoster {
|
||||
models: snapshot.models,
|
||||
freshness: CodexModelCacheFreshness::Fresh,
|
||||
fetched_at: None,
|
||||
observed_at: Some(snapshot.observed_at),
|
||||
source: "codex_app_server",
|
||||
})
|
||||
}
|
||||
|
||||
/// Ask the credential-owning CLI for its account roster. This does not import
|
||||
/// tokens, force token refresh, start a thread, or make an inference request.
|
||||
pub(crate) async fn update_from_codex_cli() -> Result<CodexModelRoster, &'static str> {
|
||||
update_from_codex_command(
|
||||
tokio::process::Command::new("codex"),
|
||||
std::time::Duration::from_secs(20),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn update_from_codex_command(
|
||||
command: tokio::process::Command,
|
||||
timeout: std::time::Duration,
|
||||
) -> Result<CodexModelRoster, &'static str> {
|
||||
let home = codex_home_path();
|
||||
let path = cli_snapshot_path(&home).ok_or("cache_path_unavailable")?;
|
||||
let roster = query_codex_cli(command, timeout, &model_roster()).await?;
|
||||
let snapshot = CliSnapshot {
|
||||
observed_at: roster.observed_at.ok_or("codex_invalid_response")?,
|
||||
models: roster.models.clone(),
|
||||
};
|
||||
let encoded = serde_json::to_vec(&snapshot).map_err(|_| "cache_write_failed")?;
|
||||
if encoded.len() as u64 > MAX_MODEL_CACHE_BYTES {
|
||||
return Err("codex_response_too_large");
|
||||
}
|
||||
codewhale_config::persistence::atomic_write(&path, &encoded)
|
||||
.map_err(|_| "cache_write_failed")?;
|
||||
if let Ok(mut memo) = ROSTER_MEMO.lock() {
|
||||
*memo = None;
|
||||
}
|
||||
Ok(roster)
|
||||
}
|
||||
|
||||
struct CodexProcess(tokio::process::Child);
|
||||
|
||||
impl CodexProcess {
|
||||
fn terminate(&mut self) {
|
||||
// npm-installed CLIs can be wrappers. Kill only the process group
|
||||
// created for this invocation, including a wrapped app-server.
|
||||
#[cfg(unix)]
|
||||
if let Some(pid) = self.0.id().and_then(|id| i32::try_from(id).ok()) {
|
||||
// SAFETY: process_group(0) below gives this child its own group.
|
||||
unsafe {
|
||||
libc::kill(-pid, libc::SIGKILL);
|
||||
}
|
||||
}
|
||||
let _ = self.0.start_kill();
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for CodexProcess {
|
||||
fn drop(&mut self) {
|
||||
self.terminate();
|
||||
}
|
||||
}
|
||||
|
||||
async fn query_codex_cli(
|
||||
mut command: tokio::process::Command,
|
||||
timeout: std::time::Duration,
|
||||
previous: &CodexModelRoster,
|
||||
) -> Result<CodexModelRoster, &'static str> {
|
||||
let directory = tempfile::tempdir().map_err(|_| "codex_temporary_directory_failed")?;
|
||||
// Never forward Codewhale/provider keys or source paths to the CLI. Its
|
||||
// own HOME/CODEX_HOME still selects its login and configuration.
|
||||
command.env_clear();
|
||||
for name in [
|
||||
"PATH",
|
||||
"HOME",
|
||||
"USERPROFILE",
|
||||
"CODEX_HOME",
|
||||
"TMPDIR",
|
||||
"TMP",
|
||||
"TEMP",
|
||||
"LANG",
|
||||
"LC_ALL",
|
||||
"SYSTEMROOT",
|
||||
"WINDIR",
|
||||
"PATHEXT",
|
||||
] {
|
||||
if let Some(value) = std::env::var_os(name) {
|
||||
command.env(name, value);
|
||||
}
|
||||
}
|
||||
command
|
||||
.args([
|
||||
"app-server",
|
||||
"--listen",
|
||||
"stdio://",
|
||||
"-c",
|
||||
"analytics.enabled=false",
|
||||
])
|
||||
.current_dir(directory.path())
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::null())
|
||||
.kill_on_drop(true);
|
||||
#[cfg(unix)]
|
||||
command.process_group(0);
|
||||
crate::utils::suppress_tokio_console_window(&mut command);
|
||||
let mut child = CodexProcess(command.spawn().map_err(|_| "codex_cli_unavailable")?);
|
||||
let mut stdin = child.0.stdin.take().ok_or("codex_stdio_unavailable")?;
|
||||
let stdout = child.0.stdout.take().ok_or("codex_stdio_unavailable")?;
|
||||
let mut stdout = BufReader::new(stdout.take(MAX_MODEL_CACHE_BYTES + 1));
|
||||
let mut bytes_read = 0usize;
|
||||
let result =
|
||||
tokio::time::timeout(timeout, async {
|
||||
send_rpc(&mut stdin, &json!({"id": 1, "method": "initialize", "params": {
|
||||
"clientInfo": {"name": "codewhale_models", "version": env!("CARGO_PKG_VERSION")},
|
||||
"capabilities": {"experimentalApi": false}
|
||||
}})).await?;
|
||||
read_rpc_result(&mut stdout, 1, &mut bytes_read).await?;
|
||||
send_rpc(&mut stdin, &json!({"method": "initialized", "params": {}})).await?;
|
||||
send_rpc(
|
||||
&mut stdin,
|
||||
&json!({"id": 2, "method": "account/read", "params": {"refreshToken": false}}),
|
||||
)
|
||||
.await?;
|
||||
let account = read_rpc_result(&mut stdout, 2, &mut bytes_read).await?;
|
||||
if account.pointer("/account/type").and_then(Value::as_str) != Some("chatgpt")
|
||||
|| account.get("requiresOpenaiAuth").and_then(Value::as_bool) != Some(true)
|
||||
{
|
||||
return Err("codex_chatgpt_login_required");
|
||||
}
|
||||
let mut models = Vec::new();
|
||||
let mut cursor: Option<String> = None;
|
||||
let mut cursors = HashSet::new();
|
||||
for page in 0..20u64 {
|
||||
let id = page + 3;
|
||||
send_rpc(
|
||||
&mut stdin,
|
||||
&json!({"id": id, "method": "model/list", "params": {
|
||||
"cursor": cursor, "limit": 100, "includeHidden": false
|
||||
}}),
|
||||
)
|
||||
.await?;
|
||||
let result = read_rpc_result(&mut stdout, id, &mut bytes_read).await?;
|
||||
let page: CliModelPage =
|
||||
serde_json::from_value(result).map_err(|_| "codex_invalid_response")?;
|
||||
models.extend(page.data);
|
||||
let Some(next) = page.next_cursor else {
|
||||
return roster_from_cli_models(models, previous);
|
||||
};
|
||||
if next.is_empty() || next.len() > 4096 || !cursors.insert(next.clone()) {
|
||||
return Err("codex_invalid_pagination");
|
||||
}
|
||||
cursor = Some(next);
|
||||
}
|
||||
Err("codex_pagination_limit")
|
||||
})
|
||||
.await
|
||||
.unwrap_or(Err("codex_timeout"));
|
||||
child.terminate();
|
||||
// Bound both the conversation and process reaping. Drop also covers
|
||||
// cancellation or errors before this cleanup point.
|
||||
let _ = tokio::time::timeout(std::time::Duration::from_secs(2), child.0.wait()).await;
|
||||
result
|
||||
}
|
||||
|
||||
async fn send_rpc(
|
||||
stdin: &mut tokio::process::ChildStdin,
|
||||
value: &Value,
|
||||
) -> Result<(), &'static str> {
|
||||
let mut bytes = serde_json::to_vec(value).map_err(|_| "codex_invalid_request")?;
|
||||
bytes.push(b'\n');
|
||||
stdin
|
||||
.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|_| "codex_stdio_failed")?;
|
||||
stdin.flush().await.map_err(|_| "codex_stdio_failed")
|
||||
}
|
||||
|
||||
async fn read_rpc_result<R: tokio::io::AsyncBufRead + Unpin>(
|
||||
stdout: &mut R,
|
||||
id: u64,
|
||||
bytes_read: &mut usize,
|
||||
) -> Result<Value, &'static str> {
|
||||
for _ in 0..256 {
|
||||
let mut line = Vec::new();
|
||||
let count = stdout
|
||||
.read_until(b'\n', &mut line)
|
||||
.await
|
||||
.map_err(|_| "codex_stdio_failed")?;
|
||||
*bytes_read += count;
|
||||
if *bytes_read as u64 > MAX_MODEL_CACHE_BYTES {
|
||||
return Err("codex_response_too_large");
|
||||
}
|
||||
if count == 0 {
|
||||
return Err("codex_stdio_closed");
|
||||
}
|
||||
let response: Value =
|
||||
serde_json::from_slice(&line).map_err(|_| "codex_invalid_response")?;
|
||||
if let Some(response_id) = response.get("id") {
|
||||
if response_id.as_u64() != Some(id) || response.get("method").is_some() {
|
||||
return Err("codex_unexpected_request");
|
||||
}
|
||||
if response.get("error").is_some() {
|
||||
return Err("codex_request_failed");
|
||||
}
|
||||
return response
|
||||
.get("result")
|
||||
.cloned()
|
||||
.ok_or("codex_invalid_response");
|
||||
}
|
||||
if response.get("method").and_then(Value::as_str).is_none() {
|
||||
return Err("codex_invalid_response");
|
||||
}
|
||||
}
|
||||
Err("codex_notification_limit")
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct CliModelPage {
|
||||
data: Vec<CliModel>,
|
||||
next_cursor: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct CliModel {
|
||||
model: String,
|
||||
hidden: bool,
|
||||
is_default: bool,
|
||||
supported_reasoning_efforts: Vec<CliReasoningEffort>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct CliReasoningEffort {
|
||||
reasoning_effort: String,
|
||||
}
|
||||
|
||||
fn roster_from_cli_models(
|
||||
mut entries: Vec<CliModel>,
|
||||
previous: &CodexModelRoster,
|
||||
) -> Result<CodexModelRoster, &'static str> {
|
||||
entries.sort_by_key(|model| !model.is_default);
|
||||
let mut seen = HashSet::new();
|
||||
let mut models = Vec::new();
|
||||
for entry in entries.into_iter().filter(|model| !model.hidden) {
|
||||
if !valid_model_id(&entry.model) || entry.supported_reasoning_efforts.len() > 16 {
|
||||
return Err("codex_invalid_response");
|
||||
}
|
||||
let mut efforts = Vec::new();
|
||||
for effort in entry.supported_reasoning_efforts {
|
||||
let effort = effort.reasoning_effort.trim().to_ascii_lowercase();
|
||||
if !valid_effort(&effort) {
|
||||
return Err("codex_invalid_response");
|
||||
}
|
||||
if !efforts.contains(&effort) {
|
||||
efforts.push(effort);
|
||||
}
|
||||
}
|
||||
if seen.insert(entry.model.to_ascii_lowercase()) {
|
||||
models.push(CodexModelMetadata {
|
||||
context_window: previous
|
||||
.metadata_for(&entry.model)
|
||||
.and_then(|model| model.context_window),
|
||||
id: entry.model,
|
||||
reasoning: Some(!efforts.is_empty()),
|
||||
efforts,
|
||||
});
|
||||
}
|
||||
}
|
||||
if models.is_empty() {
|
||||
return Err("codex_empty_catalog");
|
||||
}
|
||||
Ok(CodexModelRoster {
|
||||
models,
|
||||
freshness: CodexModelCacheFreshness::Fresh,
|
||||
fetched_at: None,
|
||||
observed_at: Some(Utc::now()),
|
||||
source: "codex_app_server",
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -336,6 +684,257 @@ mod tests {
|
||||
std::fs::write(home.join(MODEL_CACHE_FILE), FIXTURE).expect("write fixture");
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn fake_codex(script: &str, trace: &Path) -> tokio::process::Command {
|
||||
let mut command = tokio::process::Command::new("/bin/sh");
|
||||
command.args(["-c", script, "fake-codex"]).arg(trace);
|
||||
command
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
const CLI_FIXTURE: &str = r#"
|
||||
test -z "${OPENAI_API_KEY:-}${OPENAI_CODEX_ACCESS_TOKEN:-}${CODEWHALE_HOME:-}${PRIVATE_TASK_SECRET:-}" || exit 71
|
||||
pwd > "$1"
|
||||
while IFS= read -r line; do
|
||||
printf '%s\n' "$line" >> "$1"
|
||||
case "$line" in
|
||||
*'"id":1,'*) printf '%s\n' '{"id":1,"result":{}}' ;;
|
||||
*'"id":2,'*) printf '%s\n' '{"id":2,"result":{"account":{"type":"chatgpt","email":"[email protected]","planType":"pro"},"requiresOpenaiAuth":true}}' ;;
|
||||
*'"id":3,'*) printf '%s\n' '{"method":"account/updated","params":{}}' '{"id":3,"result":{"data":[{"model":"gpt-test-secondary","hidden":false,"isDefault":false,"supportedReasoningEfforts":[]},{"model":"hidden-test-model","hidden":true,"isDefault":false,"supportedReasoningEfforts":[]}],"nextCursor":"page-two"}}' ;;
|
||||
*'"id":4,'*) printf '%s\n' '{"id":4,"result":{"data":[{"model":"gpt-new-account-model","hidden":false,"isDefault":true,"supportedReasoningEfforts":[{"reasoningEffort":"high"},{"reasoningEffort":"ultra"},{"reasoningEffort":"high"}]}],"nextCursor":null}}' ;;
|
||||
esac
|
||||
done
|
||||
"#;
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn codex_cli_loads_paginated_roster_without_forwarding_credentials_and_persists_offline()
|
||||
{
|
||||
let _lock = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _codex = crate::test_support::EnvVarGuard::set("CODEX_HOME", home.path().join("codex"));
|
||||
let _codewhale =
|
||||
crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path().join("codewhale"));
|
||||
let _key = crate::test_support::EnvVarGuard::set("OPENAI_API_KEY", "private-test-key");
|
||||
let _token = crate::test_support::EnvVarGuard::set(
|
||||
"OPENAI_CODEX_ACCESS_TOKEN",
|
||||
"private-test-token",
|
||||
);
|
||||
let _secret =
|
||||
crate::test_support::EnvVarGuard::set("PRIVATE_TASK_SECRET", "private-task-value");
|
||||
let trace = home.path().join("trace");
|
||||
let roster = update_from_codex_command(
|
||||
fake_codex(CLI_FIXTURE, &trace),
|
||||
std::time::Duration::from_secs(3),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
roster.model_ids(),
|
||||
["gpt-new-account-model", "gpt-test-secondary"]
|
||||
);
|
||||
assert_eq!(
|
||||
roster
|
||||
.metadata_for("gpt-new-account-model")
|
||||
.unwrap()
|
||||
.efforts,
|
||||
["high", "ultra"]
|
||||
);
|
||||
assert_eq!(roster.fetched_at, None);
|
||||
assert!(roster.observed_at.is_some());
|
||||
assert_eq!(roster.source, "codex_app_server");
|
||||
let trace = std::fs::read_to_string(trace).unwrap();
|
||||
let mut lines = trace.lines();
|
||||
assert_ne!(
|
||||
Path::new(lines.next().unwrap()),
|
||||
std::env::current_dir().unwrap()
|
||||
);
|
||||
let requests: Vec<Value> = lines
|
||||
.map(|line| serde_json::from_str(line).unwrap())
|
||||
.collect();
|
||||
assert_eq!(
|
||||
requests
|
||||
.iter()
|
||||
.map(|r| r["method"].as_str().unwrap())
|
||||
.collect::<Vec<_>>(),
|
||||
[
|
||||
"initialize",
|
||||
"initialized",
|
||||
"account/read",
|
||||
"model/list",
|
||||
"model/list"
|
||||
]
|
||||
);
|
||||
assert_eq!(requests[2]["params"]["refreshToken"], false);
|
||||
assert_eq!(requests[4]["params"]["cursor"], "page-two");
|
||||
assert_eq!(requests[3]["params"]["includeHidden"], false);
|
||||
let path = cli_snapshot_path(&codex_home_path()).unwrap();
|
||||
let bytes = std::fs::read_to_string(&path).unwrap();
|
||||
assert!(!bytes.contains("private-"));
|
||||
assert!(!bytes.contains("hidden-test-model"));
|
||||
*ROSTER_MEMO.lock().unwrap() = None;
|
||||
assert_eq!(model_roster(), roster);
|
||||
let before = std::fs::read(&path).unwrap();
|
||||
let failure = update_from_codex_command(
|
||||
tokio::process::Command::new(home.path().join("missing-codex")),
|
||||
std::time::Duration::from_secs(1),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(failure.unwrap_err(), "codex_cli_unavailable");
|
||||
assert_eq!(std::fs::read(&path).unwrap(), before);
|
||||
assert_eq!(model_roster(), roster);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn codex_cli_rejects_other_auth_modes_and_invalid_pagination() {
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let fallback = CodexModelRoster::fallback(CodexModelCacheFreshness::Missing, None);
|
||||
for (script, expected) in [
|
||||
(
|
||||
CLI_FIXTURE.replace("\"type\":\"chatgpt\"", "\"type\":\"apiKey\""),
|
||||
"codex_chatgpt_login_required",
|
||||
),
|
||||
(
|
||||
CLI_FIXTURE.replace(
|
||||
"\"requiresOpenaiAuth\":true",
|
||||
"\"requiresOpenaiAuth\":false",
|
||||
),
|
||||
"codex_chatgpt_login_required",
|
||||
),
|
||||
(
|
||||
CLI_FIXTURE.replace("\"nextCursor\":null", "\"nextCursor\":\"page-two\""),
|
||||
"codex_invalid_pagination",
|
||||
),
|
||||
(
|
||||
CLI_FIXTURE.replace("gpt-new-account-model", "bad model"),
|
||||
"codex_invalid_response",
|
||||
),
|
||||
(
|
||||
CLI_FIXTURE.replace(
|
||||
"\"reasoningEffort\":\"ultra\"",
|
||||
"\"reasoningEffort\":\"bad effort\"",
|
||||
),
|
||||
"codex_invalid_response",
|
||||
),
|
||||
] {
|
||||
let trace = home.path().join("trace");
|
||||
let error = query_codex_cli(
|
||||
fake_codex(&script, &trace),
|
||||
std::time::Duration::from_secs(3),
|
||||
&fallback,
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert_eq!(error, expected);
|
||||
if expected == "codex_chatgpt_login_required" {
|
||||
assert!(
|
||||
!std::fs::read_to_string(trace)
|
||||
.unwrap()
|
||||
.contains("model/list")
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn codex_rpc_bounds_output_and_sanitizes_protocol_errors() {
|
||||
for (bytes, expected) in [
|
||||
(
|
||||
b"{\"id\":3,\"error\":{\"message\":\"private-secret\"}}\n".to_vec(),
|
||||
"codex_request_failed",
|
||||
),
|
||||
(
|
||||
b"{\"id\":3,\"method\":\"account/chatgptAuthTokens/refresh\"}\n".to_vec(),
|
||||
"codex_unexpected_request",
|
||||
),
|
||||
(
|
||||
b"{\"id\":9,\"result\":{}}\n".to_vec(),
|
||||
"codex_unexpected_request",
|
||||
),
|
||||
(
|
||||
vec![b'x'; MAX_MODEL_CACHE_BYTES as usize + 1],
|
||||
"codex_response_too_large",
|
||||
),
|
||||
(
|
||||
b"{\"method\":\"noise\"}\n".repeat(256),
|
||||
"codex_notification_limit",
|
||||
),
|
||||
] {
|
||||
let mut input = bytes.as_slice();
|
||||
assert_eq!(
|
||||
read_rpc_result(&mut input, 3, &mut 0).await.unwrap_err(),
|
||||
expected
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn codex_cli_timeout_and_cancellation_stop_the_wrapper_and_child() {
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let fallback = CodexModelRoster::fallback(CodexModelCacheFreshness::Missing, None);
|
||||
for cancel in [false, true] {
|
||||
let trace = home.path().join(if cancel { "cancel" } else { "timeout" });
|
||||
let command = fake_codex(
|
||||
"sleep 300 &\nprintf '%s %s' \"$$\" \"$!\" > \"$1\"\nwait",
|
||||
&trace,
|
||||
);
|
||||
let query = query_codex_cli(command, std::time::Duration::from_millis(500), &fallback);
|
||||
if cancel {
|
||||
let mut query = Box::pin(query);
|
||||
tokio::select! {
|
||||
_ = &mut query => panic!("CLI should remain pending"),
|
||||
_ = tokio::time::sleep(std::time::Duration::from_millis(250)) => {}
|
||||
}
|
||||
drop(query);
|
||||
} else {
|
||||
assert_eq!(query.await.unwrap_err(), "codex_timeout");
|
||||
}
|
||||
let pids = std::fs::read_to_string(trace).unwrap();
|
||||
for pid in pids.split_whitespace() {
|
||||
let mut running = true;
|
||||
for _ in 0..40 {
|
||||
let status = tokio::process::Command::new("/bin/ps")
|
||||
.args(["-o", "stat=", "-p", pid])
|
||||
.output()
|
||||
.await
|
||||
.unwrap();
|
||||
let state = String::from_utf8_lossy(&status.stdout);
|
||||
running = !state.trim().is_empty() && !state.trim().starts_with('Z');
|
||||
if !running {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(25)).await;
|
||||
}
|
||||
assert!(!running, "owned Codex child survived cleanup");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn codex_cli_snapshot_age_and_ids_are_validated() {
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let path = home.path().join("roster.json");
|
||||
let snapshot = CliSnapshot {
|
||||
observed_at: fixture_time(),
|
||||
models: vec![CodexModelMetadata {
|
||||
id: "gpt-account-model".to_string(),
|
||||
context_window: None,
|
||||
reasoning: Some(true),
|
||||
efforts: vec!["ultra".to_string()],
|
||||
}],
|
||||
};
|
||||
codewhale_config::persistence::atomic_write_json(&path, &snapshot).unwrap();
|
||||
assert!(load_cli_snapshot(&path, fixture_time()).is_some());
|
||||
assert!(load_cli_snapshot(&path, fixture_time() + Duration::hours(25)).is_none());
|
||||
assert!(load_cli_snapshot(&path, fixture_time() - Duration::hours(1)).is_none());
|
||||
let mut invalid = snapshot;
|
||||
invalid.models[0].id = "bad\u{1b}model".to_string();
|
||||
codewhale_config::persistence::atomic_write_json(&path, &invalid).unwrap();
|
||||
assert!(load_cli_snapshot(&path, fixture_time()).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn valid_cache_uses_priority_order_and_drops_vendor_hidden_models() {
|
||||
let home = tempfile::tempdir().expect("temp CODEX_HOME");
|
||||
|
||||
+8
-30
@@ -312,7 +312,7 @@ enum Commands {
|
||||
Logout,
|
||||
/// Manage provider authentication flows.
|
||||
Auth(TuiAuthArgs),
|
||||
/// List available models from the configured API endpoint
|
||||
/// List cached models, or update catalogs for configured providers
|
||||
Models(ModelsArgs),
|
||||
/// Generate speech audio with Xiaomi MiMo TTS models
|
||||
#[command(visible_alias = "tts")]
|
||||
@@ -1154,6 +1154,12 @@ struct ModelsArgs {
|
||||
/// Print models as pretty JSON
|
||||
#[arg(long, default_value_t = false)]
|
||||
json: bool,
|
||||
/// Refresh catalogs for all configured providers (no inference requests)
|
||||
#[arg(long, visible_alias = "refresh")]
|
||||
update: bool,
|
||||
/// Limit listing or refresh to this exact provider identity
|
||||
#[arg(long, value_name = "ID")]
|
||||
provider: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Args, Debug, Clone)]
|
||||
@@ -7549,35 +7555,7 @@ fn run_features_command(config: &Config, command: FeaturesCli) -> Result<()> {
|
||||
}
|
||||
|
||||
async fn run_models(config: &Config, args: ModelsArgs) -> Result<()> {
|
||||
use crate::client::DeepSeekClient;
|
||||
|
||||
let client = DeepSeekClient::new(config)?;
|
||||
let mut models = client.list_models().await?;
|
||||
models.sort_by(|a, b| a.id.cmp(&b.id));
|
||||
|
||||
if args.json {
|
||||
println!("{}", serde_json::to_string_pretty(&models)?);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
if models.is_empty() {
|
||||
println!("No models returned by the API.");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let default_model = config.default_model();
|
||||
|
||||
println!("Available models (default: {default_model})");
|
||||
for model in models {
|
||||
let marker = if model.id == default_model { "*" } else { " " };
|
||||
if let Some(owner) = model.owned_by {
|
||||
println!("{marker} {} ({owner})", model.id);
|
||||
} else {
|
||||
println!("{marker} {}", model.id);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
crate::provider_lake::run_models(config, args.update, args.provider.as_deref(), args.json).await
|
||||
}
|
||||
|
||||
async fn run_speech(config: &Config, args: SpeechArgs) -> Result<()> {
|
||||
|
||||
@@ -611,6 +611,12 @@ pub enum MessageId {
|
||||
CmdModeDescription,
|
||||
CmdModelDescription,
|
||||
CmdModelsDescription,
|
||||
ModelsListHeader,
|
||||
ModelsListHint,
|
||||
ModelsUpdateSummary,
|
||||
ModelsUpdatePartial,
|
||||
ModelsCodexHint,
|
||||
ModelsSourceFallback,
|
||||
CmdModelDbDescription,
|
||||
CmdNetworkDescription,
|
||||
CmdUpdateDescription,
|
||||
@@ -2745,6 +2751,12 @@ pub const ALL_MESSAGE_IDS: &[MessageId] = &[
|
||||
MessageId::CmdModeDescription,
|
||||
MessageId::CmdModelDescription,
|
||||
MessageId::CmdModelsDescription,
|
||||
MessageId::ModelsListHeader,
|
||||
MessageId::ModelsListHint,
|
||||
MessageId::ModelsUpdateSummary,
|
||||
MessageId::ModelsUpdatePartial,
|
||||
MessageId::ModelsCodexHint,
|
||||
MessageId::ModelsSourceFallback,
|
||||
MessageId::CmdModelDbDescription,
|
||||
MessageId::CmdNetworkDescription,
|
||||
MessageId::CmdUpdateDescription,
|
||||
|
||||
@@ -289,7 +289,7 @@ pub async fn refresh(force_network: bool) -> Result<usize, ModelsDevRefreshError
|
||||
ModelsDevFreshness::Live,
|
||||
)?;
|
||||
if let Some(path) = cache_path() {
|
||||
let _ = save_cache_file(
|
||||
save_cache_file(
|
||||
&path,
|
||||
&PersistedModelsDevCache {
|
||||
schema_version: CACHE_SCHEMA_VERSION,
|
||||
@@ -298,7 +298,8 @@ pub async fn refresh(force_network: bool) -> Result<usize, ModelsDevRefreshError
|
||||
source_label: url,
|
||||
body,
|
||||
},
|
||||
);
|
||||
)
|
||||
.inspect_err(|error| mark_failed(error.clone()))?;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
@@ -348,7 +349,7 @@ async fn refresh_from_path(path: &Path) -> Result<usize, ModelsDevRefreshError>
|
||||
ModelsDevFreshness::Live,
|
||||
)?;
|
||||
if let Some(cache) = cache_path() {
|
||||
let _ = save_cache_file(
|
||||
save_cache_file(
|
||||
&cache,
|
||||
&PersistedModelsDevCache {
|
||||
schema_version: CACHE_SCHEMA_VERSION,
|
||||
@@ -357,7 +358,8 @@ async fn refresh_from_path(path: &Path) -> Result<usize, ModelsDevRefreshError>
|
||||
source_label: label,
|
||||
body,
|
||||
},
|
||||
);
|
||||
)
|
||||
.inspect_err(|error| mark_failed(error.clone()))?;
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
@@ -12,15 +12,20 @@
|
||||
//! them).
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::io::Read;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Arc, RwLock};
|
||||
|
||||
use codewhale_config::catalog::{CatalogOffering, CatalogSnapshot, bundled_catalog_offerings};
|
||||
use codewhale_config::catalog::{
|
||||
CatalogOffering, CatalogSnapshot, CatalogStatus, ProviderCatalogCache, base_url_fingerprint,
|
||||
bundled_catalog_offerings, now_unix,
|
||||
};
|
||||
|
||||
use crate::codex_model_cache;
|
||||
use crate::config::{
|
||||
ApiProvider, Config, model_completion_names_for_provider, opencode_go_chat_model_id,
|
||||
provider_is_configured_for_active,
|
||||
ApiProvider, Config, ProviderIdentity, model_completion_names_for_provider,
|
||||
opencode_go_chat_model_id, provider_is_configured_for_active,
|
||||
};
|
||||
|
||||
static BUNDLED_SNAPSHOT: std::sync::OnceLock<CatalogSnapshot> = std::sync::OnceLock::new();
|
||||
@@ -495,18 +500,854 @@ pub fn models_for_provider(
|
||||
provider: ApiProvider,
|
||||
) -> Vec<String> {
|
||||
if provider_is_configured_for_active(config, provider, active) {
|
||||
all_catalog_models_for_provider(provider)
|
||||
catalog_models_for_route(
|
||||
provider,
|
||||
&config.provider_identity_for(provider),
|
||||
&config.base_url_for_route(provider),
|
||||
)
|
||||
} else {
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
|
||||
const PROVIDER_CATALOG_MAX_BYTES: u64 = 8 * 1024 * 1024;
|
||||
const PROVIDER_CATALOG_TTL_SECS: u64 = 24 * 60 * 60;
|
||||
type ProviderCacheMemo = BTreeMap<
|
||||
PathBuf,
|
||||
(
|
||||
Option<std::time::SystemTime>,
|
||||
u64,
|
||||
Arc<ProviderCatalogCache>,
|
||||
),
|
||||
>;
|
||||
static PROVIDER_CACHE_MEMO: RwLock<ProviderCacheMemo> = RwLock::new(BTreeMap::new());
|
||||
|
||||
fn provider_catalog_path(identity: &str, base_url: &str) -> anyhow::Result<PathBuf> {
|
||||
let catalog = crate::models_dev_live::cache_path()
|
||||
.and_then(|path| path.parent().map(std::path::Path::to_path_buf))
|
||||
.ok_or_else(|| anyhow::anyhow!("catalog directory unavailable"))?;
|
||||
// One file per exact configured identity + endpoint. Concurrent refreshes
|
||||
// of different routes never overwrite one another, and URLs/keys are not
|
||||
// written into filenames or receipts.
|
||||
Ok(catalog.join(format!(
|
||||
"provider-{}-{}.json",
|
||||
base_url_fingerprint(identity),
|
||||
base_url_fingerprint(base_url),
|
||||
)))
|
||||
}
|
||||
|
||||
fn load_provider_catalog(
|
||||
identity: &str,
|
||||
base_url: &str,
|
||||
) -> anyhow::Result<Arc<ProviderCatalogCache>> {
|
||||
let path = provider_catalog_path(identity, base_url)?;
|
||||
let metadata = match std::fs::symlink_metadata(&path) {
|
||||
Ok(metadata) => metadata,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Ok(Arc::new(ProviderCatalogCache::new()));
|
||||
}
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
anyhow::ensure!(
|
||||
metadata.is_file() && metadata.len() <= PROVIDER_CATALOG_MAX_BYTES,
|
||||
"invalid catalog cache"
|
||||
);
|
||||
let modified = metadata.modified().ok();
|
||||
if let Ok(memo) = PROVIDER_CACHE_MEMO.read()
|
||||
&& let Some((cached_modified, cached_len, cache)) = memo.get(&path)
|
||||
&& *cached_modified == modified
|
||||
&& *cached_len == metadata.len()
|
||||
{
|
||||
return Ok(Arc::clone(cache));
|
||||
}
|
||||
let mut options = std::fs::OpenOptions::new();
|
||||
options.read(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.custom_flags(libc::O_NOFOLLOW);
|
||||
}
|
||||
let file = options.open(&path)?;
|
||||
let mut body = Vec::new();
|
||||
file.take(PROVIDER_CATALOG_MAX_BYTES + 1)
|
||||
.read_to_end(&mut body)?;
|
||||
anyhow::ensure!(
|
||||
body.len() as u64 <= PROVIDER_CATALOG_MAX_BYTES,
|
||||
"catalog cache too large"
|
||||
);
|
||||
let cache: ProviderCatalogCache = serde_json::from_slice(&body)?;
|
||||
let fingerprint = base_url_fingerprint(base_url);
|
||||
anyhow::ensure!(
|
||||
cache.entries.values().all(|entry| {
|
||||
entry.provider == identity
|
||||
&& entry.base_url_fingerprint == fingerprint
|
||||
&& entry
|
||||
.offerings
|
||||
.iter()
|
||||
.all(|row| valid_catalog_model_id(&row.wire_model_id))
|
||||
}),
|
||||
"catalog scope mismatch"
|
||||
);
|
||||
let cache = Arc::new(cache);
|
||||
if let Ok(mut memo) = PROVIDER_CACHE_MEMO.write() {
|
||||
memo.insert(path, (modified, metadata.len(), Arc::clone(&cache)));
|
||||
}
|
||||
Ok(cache)
|
||||
}
|
||||
|
||||
pub(crate) fn valid_catalog_model_id(value: &str) -> bool {
|
||||
!value.is_empty()
|
||||
&& value.len() <= 256
|
||||
&& value.bytes().any(|byte| byte.is_ascii_alphanumeric())
|
||||
&& value.bytes().all(|byte| {
|
||||
byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b':' | b'/' | b'-')
|
||||
})
|
||||
}
|
||||
|
||||
/// Endpoint-scoped roster for CLI, pickers and inventory. A cached provider
|
||||
/// listing is authoritative for IDs, including removals. Failed/stale rows
|
||||
/// remain usable offline; configured models are retained by the caller.
|
||||
#[must_use]
|
||||
pub(crate) fn catalog_models_for_route(
|
||||
provider: ApiProvider,
|
||||
identity: &str,
|
||||
base_url: &str,
|
||||
) -> Vec<String> {
|
||||
if provider == ApiProvider::OpenaiCodex {
|
||||
return codex_model_cache::model_roster().model_ids();
|
||||
}
|
||||
if let Ok(cache) = load_provider_catalog(identity, base_url)
|
||||
&& let Some(entry) = cache.get(identity, &base_url_fingerprint(base_url))
|
||||
&& entry.fetched_at > 0
|
||||
{
|
||||
return entry
|
||||
.offerings
|
||||
.iter()
|
||||
.map(|row| row.wire_model_id.clone())
|
||||
.collect();
|
||||
}
|
||||
// Custom endpoints must never inherit models from another configured
|
||||
// endpoint that happens to have the same generic provider kind.
|
||||
if provider == ApiProvider::Custom {
|
||||
Vec::new()
|
||||
} else {
|
||||
all_catalog_models_for_provider(provider)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(serde::Serialize)]
|
||||
struct CatalogUpdateReceipt {
|
||||
provider: String,
|
||||
source: &'static str,
|
||||
outcome: &'static str,
|
||||
status: CatalogStatus,
|
||||
fetched_at: Option<u64>,
|
||||
observed_at: Option<u64>,
|
||||
base_url_fingerprint: Option<String>,
|
||||
model_count: usize,
|
||||
error: Option<&'static str>,
|
||||
}
|
||||
|
||||
fn cached_receipt(config: &Config, identity: &ProviderIdentity) -> CatalogUpdateReceipt {
|
||||
let base_url = config.base_url_for_route_identity(identity.provider, &identity.key);
|
||||
let fingerprint = base_url_fingerprint(&base_url);
|
||||
let cache = load_provider_catalog(&identity.key, &base_url);
|
||||
let entry = cache
|
||||
.as_ref()
|
||||
.ok()
|
||||
.and_then(|cache| cache.get(&identity.key, &fingerprint));
|
||||
CatalogUpdateReceipt {
|
||||
provider: identity.key.clone(),
|
||||
source: "provider_models",
|
||||
outcome: "cached",
|
||||
status: cache.as_ref().map_or(CatalogStatus::Unknown, |cache| {
|
||||
cache.status(&identity.key, &fingerprint, now_unix())
|
||||
}),
|
||||
fetched_at: entry
|
||||
.map(|entry| entry.fetched_at)
|
||||
.filter(|timestamp| *timestamp > 0),
|
||||
observed_at: None,
|
||||
base_url_fingerprint: Some(fingerprint),
|
||||
model_count: entry.map_or(0, |entry| entry.offerings.len()),
|
||||
error: cache.is_err().then_some("cache_read_failed"),
|
||||
}
|
||||
}
|
||||
|
||||
fn catalog_identities(
|
||||
config: &Config,
|
||||
selected: Option<&str>,
|
||||
update: bool,
|
||||
) -> anyhow::Result<Vec<ProviderIdentity>> {
|
||||
if let Some(selected) = selected {
|
||||
return Ok(vec![
|
||||
config
|
||||
.resolve_provider_identity(selected)
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
]);
|
||||
}
|
||||
let active = config
|
||||
.active_provider_identity(config.api_provider())
|
||||
.map_err(anyhow::Error::msg)?;
|
||||
if !update {
|
||||
return Ok(vec![active]);
|
||||
}
|
||||
let mut identities = vec![active];
|
||||
for provider in configured_providers(config, config.api_provider()) {
|
||||
if provider != ApiProvider::Custom {
|
||||
identities.push(
|
||||
config
|
||||
.active_provider_identity(provider)
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
);
|
||||
}
|
||||
}
|
||||
if let Some(providers) = &config.providers {
|
||||
for (name, entry) in &providers.custom {
|
||||
if !entry.is_openai_compatible_custom() {
|
||||
continue;
|
||||
}
|
||||
identities.push(
|
||||
config
|
||||
.resolve_provider_identity(name)
|
||||
.map_err(anyhow::Error::msg)?,
|
||||
);
|
||||
}
|
||||
}
|
||||
identities.sort_by(|a, b| a.key.cmp(&b.key));
|
||||
identities.dedup_by(|a, b| a.key == b.key);
|
||||
Ok(identities)
|
||||
}
|
||||
|
||||
fn codex_receipt(identity: &ProviderIdentity) -> CatalogUpdateReceipt {
|
||||
let roster = codex_model_cache::model_roster();
|
||||
codex_roster_receipt(identity, &roster)
|
||||
}
|
||||
|
||||
fn codex_roster_receipt(
|
||||
identity: &ProviderIdentity,
|
||||
roster: &codex_model_cache::CodexModelRoster,
|
||||
) -> CatalogUpdateReceipt {
|
||||
let fresh = roster.freshness == codex_model_cache::CodexModelCacheFreshness::Fresh;
|
||||
CatalogUpdateReceipt {
|
||||
provider: identity.key.clone(),
|
||||
source: roster.source,
|
||||
outcome: "cached",
|
||||
status: if fresh {
|
||||
CatalogStatus::Fresh
|
||||
} else {
|
||||
CatalogStatus::Unknown
|
||||
},
|
||||
fetched_at: roster
|
||||
.fetched_at
|
||||
.and_then(|timestamp| u64::try_from(timestamp.timestamp()).ok()),
|
||||
observed_at: roster
|
||||
.observed_at
|
||||
.and_then(|timestamp| u64::try_from(timestamp.timestamp()).ok()),
|
||||
base_url_fingerprint: None,
|
||||
model_count: roster.models.len(),
|
||||
error: (!fresh).then_some("codex_cache_unavailable"),
|
||||
}
|
||||
}
|
||||
|
||||
fn codex_route_matches_cli_account(config: &Config) -> bool {
|
||||
if config.provider_uses_custom_endpoint(ApiProvider::OpenaiCodex)
|
||||
|| [
|
||||
"OPENAI_CODEX_ACCESS_TOKEN",
|
||||
"CODEX_ACCESS_TOKEN",
|
||||
"OPENAI_CODEX_ACCOUNT_ID",
|
||||
"CODEX_ACCOUNT_ID",
|
||||
]
|
||||
.iter()
|
||||
.any(|name| std::env::var(name).is_ok_and(|value| !value.trim().is_empty()))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
let cli_auth_path = codex_model_cache::codex_home_path().join("auth.json");
|
||||
let cli_auth_path = std::fs::canonicalize(&cli_auth_path).unwrap_or(cli_auth_path);
|
||||
crate::oauth::auth_file_path() == cli_auth_path
|
||||
}
|
||||
|
||||
async fn update_provider_catalog(
|
||||
config: &Config,
|
||||
identity: &ProviderIdentity,
|
||||
) -> CatalogUpdateReceipt {
|
||||
if identity.provider == ApiProvider::OpenaiCodex {
|
||||
if !codex_route_matches_cli_account(config) {
|
||||
let mut receipt = codex_receipt(identity);
|
||||
receipt.outcome = "skipped";
|
||||
receipt.error = Some("codex_account_route_mismatch");
|
||||
return receipt;
|
||||
}
|
||||
return match codex_model_cache::update_from_codex_cli().await {
|
||||
Ok(roster) => {
|
||||
let mut receipt = codex_roster_receipt(identity, &roster);
|
||||
receipt.outcome = "loaded";
|
||||
receipt
|
||||
}
|
||||
Err(error) => {
|
||||
let mut receipt = codex_receipt(identity);
|
||||
receipt.outcome = "failed";
|
||||
receipt.error = Some(error);
|
||||
receipt
|
||||
}
|
||||
};
|
||||
}
|
||||
let mut route_config = config.clone();
|
||||
route_config.scope_to_provider_identity(identity);
|
||||
let base_url = route_config.deepseek_base_url();
|
||||
let fingerprint = base_url_fingerprint(&base_url);
|
||||
let mut receipt = cached_receipt(&route_config, identity);
|
||||
if identity.provider == ApiProvider::Antigravity {
|
||||
receipt.outcome = "skipped";
|
||||
receipt.error = Some("provider_retired");
|
||||
return receipt;
|
||||
}
|
||||
if crate::config::explicit_cli_api_key_override().is_some()
|
||||
&& config
|
||||
.active_provider_identity(config.api_provider())
|
||||
.ok()
|
||||
.as_ref()
|
||||
!= Some(identity)
|
||||
{
|
||||
receipt.outcome = "skipped";
|
||||
receipt.error = Some("cli_key_is_scoped_to_active_provider");
|
||||
return receipt;
|
||||
}
|
||||
if route_config
|
||||
.auth_mode_for_provider(identity.provider)
|
||||
.is_some_and(|mode| mode.eq_ignore_ascii_case("oauth"))
|
||||
{
|
||||
receipt.outcome = "skipped";
|
||||
receipt.error = Some("oauth_catalog_unavailable");
|
||||
return receipt;
|
||||
}
|
||||
// Ordinary model listing never constructs a client. Explicit refresh uses
|
||||
// the existing read-only resolver: no secret migration or OAuth refresh.
|
||||
let client = route_config
|
||||
.with_read_only_api_key_for_diagnostic()
|
||||
.and_then(|config| crate::client::DeepSeekClient::new(&config));
|
||||
let client = match client {
|
||||
Ok(client) => client,
|
||||
Err(_) => {
|
||||
receipt.outcome = "skipped";
|
||||
receipt.error = Some("credentials_or_route_unavailable");
|
||||
return receipt;
|
||||
}
|
||||
};
|
||||
let mut cache = match load_provider_catalog(&identity.key, &base_url) {
|
||||
Ok(cache) => (*cache).clone(),
|
||||
Err(_) => {
|
||||
receipt.outcome = "failed";
|
||||
// Do not overwrite an unreadable/corrupt prior cache.
|
||||
return receipt;
|
||||
}
|
||||
};
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(20),
|
||||
client.fetch_catalog_delta(),
|
||||
)
|
||||
.await
|
||||
.unwrap_or(Err(codewhale_config::catalog::CatalogRefreshError::Network));
|
||||
match result {
|
||||
Ok(mut delta) => {
|
||||
if delta.base_url_fingerprint != fingerprint {
|
||||
receipt.outcome = "failed";
|
||||
receipt.error = Some("catalog_endpoint_mismatch");
|
||||
return receipt;
|
||||
}
|
||||
delta.provider = identity.key.clone();
|
||||
cache.record_success(delta, PROVIDER_CATALOG_TTL_SECS);
|
||||
receipt.outcome = "updated";
|
||||
}
|
||||
Err(reason) => {
|
||||
cache.record_failure(&identity.key, &fingerprint, reason);
|
||||
receipt.outcome = "failed";
|
||||
}
|
||||
}
|
||||
let save = provider_catalog_path(&identity.key, &base_url).and_then(|path| {
|
||||
codewhale_config::persistence::atomic_write_json(&path, &cache)?;
|
||||
if let Ok(mut memo) = PROVIDER_CACHE_MEMO.write() {
|
||||
memo.remove(&path);
|
||||
}
|
||||
Ok(())
|
||||
});
|
||||
if save.is_err() {
|
||||
receipt.outcome = "failed";
|
||||
receipt.error = Some("cache_write_failed");
|
||||
return receipt;
|
||||
}
|
||||
let outcome = receipt.outcome;
|
||||
receipt = cached_receipt(&route_config, identity);
|
||||
receipt.outcome = outcome;
|
||||
receipt
|
||||
}
|
||||
|
||||
pub(crate) async fn run_models(
|
||||
config: &Config,
|
||||
update: bool,
|
||||
selected: Option<&str>,
|
||||
json: bool,
|
||||
) -> anyhow::Result<()> {
|
||||
use crate::localization::{MessageId, resolve_locale, tr};
|
||||
let locale = resolve_locale(
|
||||
&crate::settings::Settings::load_persisted()
|
||||
.unwrap_or_default()
|
||||
.locale,
|
||||
);
|
||||
let identities = catalog_identities(config, selected, update)?;
|
||||
crate::models_dev_live::maybe_load_persisted_cache();
|
||||
if update {
|
||||
let mut receipts = Vec::new();
|
||||
if selected.is_none() {
|
||||
let result = crate::models_dev_live::refresh(true).await;
|
||||
let status = crate::models_dev_live::status();
|
||||
receipts.push(CatalogUpdateReceipt {
|
||||
provider: "models.dev".to_string(),
|
||||
source: "models.dev",
|
||||
outcome: if result.is_ok() { "updated" } else { "failed" },
|
||||
status: if result.is_ok() {
|
||||
CatalogStatus::Fresh
|
||||
} else {
|
||||
CatalogStatus::Unknown
|
||||
},
|
||||
fetched_at: status.fetched_at,
|
||||
observed_at: None,
|
||||
base_url_fingerprint: None,
|
||||
model_count: status.offering_count,
|
||||
error: result.err().map(|error| match error {
|
||||
crate::models_dev_live::ModelsDevRefreshError::Disabled => "fetch_disabled",
|
||||
crate::models_dev_live::ModelsDevRefreshError::Network(_) => "network",
|
||||
crate::models_dev_live::ModelsDevRefreshError::HttpStatus(_) => "http_status",
|
||||
crate::models_dev_live::ModelsDevRefreshError::InvalidResponse(_) => {
|
||||
"invalid_response"
|
||||
}
|
||||
crate::models_dev_live::ModelsDevRefreshError::EmptyCatalog => "empty_catalog",
|
||||
crate::models_dev_live::ModelsDevRefreshError::Io(_) => "cache_io",
|
||||
}),
|
||||
});
|
||||
}
|
||||
use futures_util::StreamExt;
|
||||
receipts.extend(
|
||||
futures_util::stream::iter(&identities)
|
||||
.map(|identity| update_provider_catalog(config, identity))
|
||||
.buffered(4)
|
||||
.collect::<Vec<_>>()
|
||||
.await,
|
||||
);
|
||||
let updated = receipts
|
||||
.iter()
|
||||
.filter(|receipt| receipt.outcome == "updated")
|
||||
.count();
|
||||
let loaded = receipts
|
||||
.iter()
|
||||
.filter(|receipt| receipt.outcome == "loaded")
|
||||
.count();
|
||||
let failed = receipts
|
||||
.iter()
|
||||
.filter(|receipt| receipt.outcome == "failed")
|
||||
.count();
|
||||
let skipped = receipts
|
||||
.iter()
|
||||
.filter(|receipt| receipt.outcome == "skipped")
|
||||
.count();
|
||||
if json {
|
||||
println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(&serde_json::json!({
|
||||
"updated": updated, "loaded": loaded, "failed": failed, "skipped": skipped,
|
||||
"catalogs": receipts,
|
||||
}))?
|
||||
);
|
||||
} else {
|
||||
println!(
|
||||
"{}",
|
||||
tr(locale, MessageId::ModelsUpdateSummary)
|
||||
.replace("{updated}", &updated.to_string())
|
||||
.replace("{loaded}", &loaded.to_string())
|
||||
.replace("{failed}", &failed.to_string())
|
||||
.replace("{skipped}", &skipped.to_string())
|
||||
);
|
||||
for receipt in &receipts {
|
||||
println!(
|
||||
"{}\t{}\tmodels={}\tfetched_at={}\tobserved_at={}\tsource={}\tstatus={}{}",
|
||||
receipt.provider,
|
||||
receipt.outcome,
|
||||
receipt.model_count,
|
||||
receipt
|
||||
.fetched_at
|
||||
.map_or_else(|| "unknown".to_string(), |timestamp| timestamp.to_string()),
|
||||
receipt
|
||||
.observed_at
|
||||
.map_or_else(|| "unknown".to_string(), |timestamp| timestamp.to_string()),
|
||||
receipt.source,
|
||||
serde_json::to_string(&receipt.status)?,
|
||||
receipt
|
||||
.error
|
||||
.map_or_else(String::new, |error| format!("\terror={error}"))
|
||||
);
|
||||
}
|
||||
if receipts
|
||||
.iter()
|
||||
.any(|receipt| matches!(receipt.source, "codex_cli_cache" | "codex_app_server"))
|
||||
{
|
||||
println!("{}", tr(locale, MessageId::ModelsCodexHint));
|
||||
}
|
||||
}
|
||||
if failed > 0 {
|
||||
anyhow::bail!("{}", tr(locale, MessageId::ModelsUpdatePartial));
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
let identity = &identities[0];
|
||||
let mut route_config = config.clone();
|
||||
route_config.scope_to_provider_identity(identity);
|
||||
let mut models = catalog_models_for_route(
|
||||
identity.provider,
|
||||
&identity.key,
|
||||
&route_config.deepseek_base_url(),
|
||||
);
|
||||
let default_model = route_config.default_model();
|
||||
if !default_model.is_empty() && !default_model.eq_ignore_ascii_case("auto") {
|
||||
push_unique_model(&mut models, &default_model);
|
||||
}
|
||||
models.sort();
|
||||
models.dedup();
|
||||
if json {
|
||||
// Preserve the existing array + AvailableModel field shape.
|
||||
let rows: Vec<_> = models
|
||||
.iter()
|
||||
.map(|id| crate::client::AvailableModel {
|
||||
id: id.clone(),
|
||||
owned_by: None,
|
||||
created: None,
|
||||
})
|
||||
.collect();
|
||||
println!("{}", serde_json::to_string_pretty(&rows)?);
|
||||
} else {
|
||||
println!(
|
||||
"{}",
|
||||
tr(locale, MessageId::ModelsListHeader)
|
||||
.replace("{provider}", &identity.key)
|
||||
.replace("{model}", &default_model)
|
||||
);
|
||||
let receipt = if identity.provider == ApiProvider::OpenaiCodex {
|
||||
codex_receipt(identity)
|
||||
} else {
|
||||
cached_receipt(&route_config, identity)
|
||||
};
|
||||
println!(
|
||||
"source={}\tstatus={}\tfetched_at={}\tobserved_at={}",
|
||||
receipt.source,
|
||||
serde_json::to_string(&receipt.status)?,
|
||||
receipt
|
||||
.fetched_at
|
||||
.map_or_else(|| "unknown".to_string(), |timestamp| timestamp.to_string()),
|
||||
receipt
|
||||
.observed_at
|
||||
.map_or_else(|| "unknown".to_string(), |timestamp| timestamp.to_string())
|
||||
);
|
||||
if receipt.model_count == 0 {
|
||||
println!("{}", tr(locale, MessageId::ModelsSourceFallback));
|
||||
}
|
||||
for model in models {
|
||||
println!("{} {model}", if model == default_model { "*" } else { " " });
|
||||
}
|
||||
println!("{}", tr(locale, MessageId::ModelsListHint));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::{DEFAULT_TOGETHER_FLASH_MODEL, DEFAULT_TOGETHER_MODEL};
|
||||
use codewhale_config::catalog::CatalogSource;
|
||||
|
||||
fn catalog_test_config(first_url: &str, second_url: &str) -> Config {
|
||||
use crate::config::{ProviderConfig, ProvidersConfig};
|
||||
Config {
|
||||
provider: Some("catalog-first".to_string()),
|
||||
providers: Some(ProvidersConfig {
|
||||
custom: [
|
||||
("catalog-first", first_url, "first-route-test-key"),
|
||||
("catalog-second", second_url, "second-route-test-key"),
|
||||
]
|
||||
.into_iter()
|
||||
.map(|(name, base_url, key)| {
|
||||
(
|
||||
name.to_string(),
|
||||
ProviderConfig {
|
||||
kind: Some("openai-compatible".to_string()),
|
||||
base_url: Some(base_url.to_string()),
|
||||
api_key: Some(key.to_string()),
|
||||
model: Some("saved-model".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
})
|
||||
.collect(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
async fn catalog_mock(
|
||||
server: &wiremock::MockServer,
|
||||
key: &str,
|
||||
status: u16,
|
||||
body: serde_json::Value,
|
||||
) {
|
||||
use wiremock::matchers::{header, method, path};
|
||||
wiremock::Mock::given(method("GET"))
|
||||
.and(path("/v1/models"))
|
||||
.and(header("Authorization", format!("Bearer {key}")))
|
||||
.respond_with(wiremock::ResponseTemplate::new(status).set_body_json(body))
|
||||
.expect(1)
|
||||
.mount(server)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn models_update_persists_exact_routes_and_keeps_prior_rows_after_failure() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _cli_key = crate::test_support::EnvVarGuard::remove(codewhale_config::CLI_API_KEY_ENV);
|
||||
let first = wiremock::MockServer::start().await;
|
||||
let second = wiremock::MockServer::start().await;
|
||||
let config = catalog_test_config(&first.uri(), &second.uri());
|
||||
let first_id = config.resolve_provider_identity("catalog-first").unwrap();
|
||||
let second_id = config.resolve_provider_identity("catalog-second").unwrap();
|
||||
catalog_mock(
|
||||
&first,
|
||||
"first-route-test-key",
|
||||
200,
|
||||
serde_json::json!({"data":[{"id":"new-first-model"}]}),
|
||||
)
|
||||
.await;
|
||||
catalog_mock(
|
||||
&second,
|
||||
"second-route-test-key",
|
||||
200,
|
||||
serde_json::json!({"data":[{"id":"new-second-model"}]}),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
update_provider_catalog(&config, &first_id).await.outcome,
|
||||
"updated"
|
||||
);
|
||||
assert_eq!(
|
||||
update_provider_catalog(&config, &second_id).await.outcome,
|
||||
"updated"
|
||||
);
|
||||
assert_eq!(config.provider.as_deref(), Some("catalog-first"));
|
||||
assert_eq!(config.default_model(), "saved-model");
|
||||
// Simulate restart: remove only this memo, not any persistent state.
|
||||
PROVIDER_CACHE_MEMO.write().unwrap().clear();
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, "catalog-first", &first.uri()),
|
||||
["new-first-model"]
|
||||
);
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, "catalog-second", &second.uri()),
|
||||
["new-second-model"]
|
||||
);
|
||||
assert!(
|
||||
catalog_models_for_route(ApiProvider::Custom, "catalog-first", &second.uri())
|
||||
.is_empty()
|
||||
);
|
||||
let prior = cached_receipt(&config, &first_id).fetched_at;
|
||||
first.reset().await;
|
||||
catalog_mock(
|
||||
&first,
|
||||
"first-route-test-key",
|
||||
401,
|
||||
serde_json::json!({"error":"first-route-test-key"}),
|
||||
)
|
||||
.await;
|
||||
let receipt = update_provider_catalog(&config, &first_id).await;
|
||||
assert_eq!(receipt.outcome, "failed");
|
||||
assert_eq!(receipt.fetched_at, prior);
|
||||
assert!(matches!(
|
||||
receipt.status,
|
||||
CatalogStatus::Failed {
|
||||
reason: codewhale_config::catalog::CatalogRefreshError::Unauthorized
|
||||
}
|
||||
));
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, "catalog-first", &first.uri()),
|
||||
["new-first-model"]
|
||||
);
|
||||
let body =
|
||||
std::fs::read_to_string(provider_catalog_path("catalog-first", &first.uri()).unwrap())
|
||||
.unwrap();
|
||||
assert!(!body.contains("first-route-test-key"));
|
||||
assert!(!body.contains(&first.uri()));
|
||||
assert!(
|
||||
!serde_json::to_string(&receipt)
|
||||
.unwrap()
|
||||
.contains("first-route-test-key")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn models_update_removes_withdrawn_ids_and_rejects_secret_or_control_ids() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _cli_key = crate::test_support::EnvVarGuard::remove(codewhale_config::CLI_API_KEY_ENV);
|
||||
let server = wiremock::MockServer::start().await;
|
||||
let config = catalog_test_config(&server.uri(), &server.uri());
|
||||
let identity = config.resolve_provider_identity("catalog-first").unwrap();
|
||||
for model in [
|
||||
"old-model",
|
||||
"new-model",
|
||||
"first-route-test-key",
|
||||
"bad\u{1b}[31m-model",
|
||||
] {
|
||||
server.reset().await;
|
||||
catalog_mock(
|
||||
&server,
|
||||
"first-route-test-key",
|
||||
200,
|
||||
serde_json::json!({"data":[{"id": model}]}),
|
||||
)
|
||||
.await;
|
||||
let receipt = update_provider_catalog(&config, &identity).await;
|
||||
if model.starts_with("old-") || model.starts_with("new-") {
|
||||
assert_eq!(receipt.outcome, "updated");
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, &identity.key, &server.uri()),
|
||||
[model]
|
||||
);
|
||||
} else {
|
||||
assert_eq!(receipt.outcome, "failed");
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, &identity.key, &server.uri()),
|
||||
["new-model"]
|
||||
);
|
||||
}
|
||||
}
|
||||
server.reset().await;
|
||||
catalog_mock(
|
||||
&server,
|
||||
"first-route-test-key",
|
||||
200,
|
||||
serde_json::json!({"data":[]}),
|
||||
)
|
||||
.await;
|
||||
let receipt = update_provider_catalog(&config, &identity).await;
|
||||
// The existing provider adapter treats an empty list as a failed
|
||||
// refresh. Preserve the last usable rows and disclose that failure.
|
||||
assert_eq!(receipt.outcome, "failed");
|
||||
assert_eq!(
|
||||
catalog_models_for_route(ApiProvider::Custom, &identity.key, &server.uri()),
|
||||
["new-model"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn codex_update_does_not_attribute_another_accounts_roster_to_overridden_credentials() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEX_HOME", home.path());
|
||||
let _overrides: Vec<_> = [
|
||||
"OPENAI_CODEX_AUTH_FILE",
|
||||
"OPENAI_CODEX_ACCESS_TOKEN",
|
||||
"CODEX_ACCESS_TOKEN",
|
||||
"OPENAI_CODEX_ACCOUNT_ID",
|
||||
"CODEX_ACCOUNT_ID",
|
||||
]
|
||||
.iter()
|
||||
.map(|name| crate::test_support::EnvVarGuard::remove(name))
|
||||
.collect();
|
||||
let config = Config {
|
||||
provider: Some("openai-codex".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
assert!(codex_route_matches_cli_account(&config));
|
||||
{
|
||||
let _token =
|
||||
crate::test_support::EnvVarGuard::set("CODEX_ACCESS_TOKEN", "standalone-token");
|
||||
assert!(!codex_route_matches_cli_account(&config));
|
||||
}
|
||||
let _other = crate::test_support::EnvVarGuard::set(
|
||||
"OPENAI_CODEX_AUTH_FILE",
|
||||
home.path().join("other-auth.json"),
|
||||
);
|
||||
assert!(!codex_route_matches_cli_account(&config));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn models_listing_is_offline_and_update_never_forwards_another_routes_cli_key() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _source =
|
||||
crate::test_support::EnvVarGuard::set(codewhale_config::CLI_API_KEY_SOURCE_ENV, "cli");
|
||||
let _key = crate::test_support::EnvVarGuard::set(
|
||||
codewhale_config::CLI_API_KEY_ENV,
|
||||
"active-route-cli-key",
|
||||
);
|
||||
let server = wiremock::MockServer::start().await;
|
||||
let config = catalog_test_config(&server.uri(), &server.uri());
|
||||
run_models(&config, false, Some("catalog-first"), true)
|
||||
.await
|
||||
.unwrap();
|
||||
let other = config.resolve_provider_identity("catalog-second").unwrap();
|
||||
let receipt = update_provider_catalog(&config, &other).await;
|
||||
assert_eq!(receipt.outcome, "skipped");
|
||||
assert_eq!(receipt.error, Some("cli_key_is_scoped_to_active_provider"));
|
||||
assert!(server.received_requests().await.unwrap().is_empty());
|
||||
assert!(
|
||||
!provider_catalog_path("catalog-first", &server.uri())
|
||||
.unwrap()
|
||||
.exists()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn models_update_reports_io_failure_without_claiming_persistence() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let home = tempfile::tempdir().unwrap();
|
||||
let _home = crate::test_support::EnvVarGuard::set("CODEWHALE_HOME", home.path());
|
||||
let _cli_key = crate::test_support::EnvVarGuard::remove(codewhale_config::CLI_API_KEY_ENV);
|
||||
let server = wiremock::MockServer::start().await;
|
||||
let config = catalog_test_config(&server.uri(), &server.uri());
|
||||
let identity = config.resolve_provider_identity("catalog-first").unwrap();
|
||||
let path = provider_catalog_path(&identity.key, &server.uri()).unwrap();
|
||||
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
|
||||
std::fs::write(&path, b"broken cache").unwrap();
|
||||
let receipt = update_provider_catalog(&config, &identity).await;
|
||||
assert_eq!(receipt.outcome, "failed");
|
||||
assert_eq!(receipt.error, Some("cache_read_failed"));
|
||||
assert_eq!(std::fs::read(&path).unwrap(), b"broken cache");
|
||||
assert!(server.received_requests().await.unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn models_update_scope_includes_every_named_identity_once() {
|
||||
let _env = crate::test_support::lock_test_env();
|
||||
let config = catalog_test_config("http://localhost:1", "http://localhost:2");
|
||||
let identities = catalog_identities(&config, None, true).unwrap();
|
||||
for name in ["catalog-first", "catalog-second"] {
|
||||
assert_eq!(
|
||||
identities
|
||||
.iter()
|
||||
.filter(|identity| identity.key == name)
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
catalog_identities(&config, Some("catalog-second"), true)
|
||||
.unwrap()
|
||||
.len(),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn together_catalog_includes_flash_from_bundled_asset() {
|
||||
let _live = lock_live_snapshot();
|
||||
|
||||
@@ -36,9 +36,7 @@ use crate::model_profile::{
|
||||
use crate::model_registry;
|
||||
use crate::models_dev_live::{self, ModelsDevFreshness};
|
||||
use crate::palette;
|
||||
use crate::provider_lake::{
|
||||
all_catalog_models_for_provider, catalog_offering_for_model, configured_providers,
|
||||
};
|
||||
use crate::provider_lake::{catalog_offering_for_model, configured_providers};
|
||||
use crate::settings::PinnedModel;
|
||||
use crate::tui::app::{App, ReasoningEffort};
|
||||
use crate::tui::menu_style;
|
||||
@@ -1336,7 +1334,11 @@ fn picker_model_rows_for_app(app: &App, config: &Config) -> Vec<ModelPickerRow>
|
||||
let mut model_ids = if provider == ApiProvider::OpenaiCodex {
|
||||
codex_roster.model_ids()
|
||||
} else {
|
||||
provider_catalog_model_ids(provider)
|
||||
provider_catalog_model_ids(
|
||||
provider,
|
||||
&config.provider_identity_for(provider),
|
||||
&config.base_url_for_route(provider),
|
||||
)
|
||||
};
|
||||
if let Some(model) = app
|
||||
.provider_models
|
||||
@@ -1604,11 +1606,15 @@ fn push_configured_provider_model(
|
||||
}
|
||||
}
|
||||
|
||||
fn provider_catalog_model_ids(provider: ApiProvider) -> Vec<String> {
|
||||
fn provider_catalog_model_ids(
|
||||
provider: ApiProvider,
|
||||
identity: &str,
|
||||
base_url: &str,
|
||||
) -> Vec<String> {
|
||||
let mut models = Vec::new();
|
||||
for id in all_catalog_models_for_provider(provider) {
|
||||
// The catalog describes the built-in provider route. A custom route's
|
||||
// endpoint-owned current/configured model is appended separately.
|
||||
for id in crate::provider_lake::catalog_models_for_route(provider, identity, base_url) {
|
||||
// Cached IDs belong to this exact endpoint. The configured/current
|
||||
// model is appended separately so users can still select saved IDs.
|
||||
push_model_id(&mut models, picker_visible_model_id(provider, &id, false));
|
||||
}
|
||||
models
|
||||
@@ -1619,7 +1625,11 @@ fn provider_scoped_model_ids_for_app(app: &App, include_current_model: bool) ->
|
||||
// separate custom/current-model row.
|
||||
let mut models = Vec::new();
|
||||
push_model_id(&mut models, "auto");
|
||||
for id in provider_catalog_model_ids(app.api_provider) {
|
||||
for id in provider_catalog_model_ids(
|
||||
app.api_provider,
|
||||
app.provider_identity_for_persistence(),
|
||||
&app.active_route_base_url,
|
||||
) {
|
||||
push_model_id(&mut models, &id);
|
||||
}
|
||||
|
||||
|
||||
@@ -43,9 +43,7 @@ use crate::model_profile::{
|
||||
};
|
||||
use crate::models_dev_live::{self, ModelsDevFreshness};
|
||||
use crate::palette;
|
||||
use crate::provider_lake::{
|
||||
all_catalog_models_for_provider, catalog_model_count_for_provider, catalog_offering_for_model,
|
||||
};
|
||||
use crate::provider_lake::{catalog_model_count_for_provider, catalog_offering_for_model};
|
||||
use crate::provider_readiness::{
|
||||
CredentialState, ProviderReadinessSnapshot, ProviderRouteIdentity, ResolvedProviderReadiness,
|
||||
credential_state_for_provider, route_identity_for_model,
|
||||
@@ -1462,7 +1460,11 @@ fn model_cost_label(provider: ApiProvider, model: &str) -> String {
|
||||
/// alphabetical. Falls back to the default route when the catalog has no rows
|
||||
/// for the provider, so the pane never renders empty.
|
||||
fn provider_pane_models(row: &ProviderDashboardRow, limit: usize) -> Vec<(String, String, bool)> {
|
||||
let mut models = all_catalog_models_for_provider(row.provider);
|
||||
let mut models = crate::provider_lake::catalog_models_for_route(
|
||||
row.provider,
|
||||
&row.provider_id,
|
||||
&row.base_url,
|
||||
);
|
||||
if models.is_empty() && !row.default_route.logical_model.trim().is_empty() {
|
||||
models.push(row.default_route.logical_model.clone());
|
||||
}
|
||||
@@ -2181,7 +2183,12 @@ impl ProviderPickerView {
|
||||
} else {
|
||||
route.logical_model.clone()
|
||||
};
|
||||
let mut models = crate::provider_lake::all_catalog_models_for_provider(provider);
|
||||
let row = &self.rows[self.selected_idx];
|
||||
let mut models = crate::provider_lake::catalog_models_for_route(
|
||||
provider,
|
||||
&row.provider_id,
|
||||
&row.base_url,
|
||||
);
|
||||
if kimi_code_k3
|
||||
&& !preferred.trim().is_empty()
|
||||
&& !models
|
||||
@@ -2944,7 +2951,12 @@ impl ProviderPickerView {
|
||||
}
|
||||
lines.push(Line::from(spans));
|
||||
}
|
||||
let total_models = all_catalog_models_for_provider(row.provider).len();
|
||||
let total_models = crate::provider_lake::catalog_models_for_route(
|
||||
row.provider,
|
||||
&row.provider_id,
|
||||
&row.base_url,
|
||||
)
|
||||
.len();
|
||||
if total_models > pane_models.len() {
|
||||
lines.push(Line::from(Span::styled(
|
||||
format!(" +{} more · M for all", total_models - pane_models.len()),
|
||||
|
||||
@@ -42,6 +42,49 @@ Sources to keep in sync:
|
||||
|
||||
## Provider Selection
|
||||
|
||||
Refresh model catalogs without installing a new Codewhale release:
|
||||
|
||||
```sh
|
||||
codewhale models --update
|
||||
codewhale models --update --provider openai
|
||||
codewhale models --provider openai --json
|
||||
```
|
||||
|
||||
`models --update` (also `--refresh`) updates the shared Models.dev metadata
|
||||
and calls the existing `/models` endpoint for each configured provider with
|
||||
its own credentials. `--provider ID` restricts the refresh to that exact
|
||||
provider, including named custom endpoints. It makes no inference requests
|
||||
and never changes the saved provider or model. A command-line API key is
|
||||
confined to the active provider; other routes are reported as skipped for
|
||||
that invocation.
|
||||
|
||||
Plain `models` lists the active provider's saved catalog without provider requests or
|
||||
authentication checks. Successful refreshes are saved under Codewhale's
|
||||
catalog directory and used by the model/provider pickers. Cache files are
|
||||
scoped to provider identity and endpoint; a failed refresh preserves prior
|
||||
rows. Text output reports source, last successful fetch time (Unix seconds),
|
||||
and freshness. `--update --json` adds per-source receipts and aggregate counts;
|
||||
partial failures return a nonzero exit code after writing those receipts.
|
||||
Ordinary `models --json` keeps its model-array format. Bundled/configured
|
||||
fallbacks are not proof that an account can use every listed model.
|
||||
|
||||
`codewhale models --update --provider openai-codex` asks the installed Codex
|
||||
CLI for its signed-in ChatGPT account's model list through the documented
|
||||
[app-server stdio API](https://learn.chatgpt.com/docs/app-server). Pagination
|
||||
and supported reasoning efforts are preserved. This requires a Codex version
|
||||
with `account/read` and `model/list` support. It starts no conversation, imports
|
||||
no tokens, and sends no Codewhale provider keys to Codex. Standalone credential
|
||||
overrides or custom endpoints that could select a different account are skipped.
|
||||
|
||||
Codex controls its own upstream cache policy; `model/list` does not expose a
|
||||
force-refresh option. Receipts therefore count the result as `loaded`, with an
|
||||
`observed_at` lookup time and no claimed upstream `fetched_at`. The observed
|
||||
roster is saved for offline listing and pickers, alongside the existing Codex
|
||||
cache fallback. A new model, such as GPT-6 Astra, appears only if that account's
|
||||
roster supplies its exact ID. Codewhale never guesses availability or substitutes
|
||||
a different billing route. Missing/stale rosters, CLI failures, and unsupported
|
||||
OAuth catalogs are reported explicitly.
|
||||
|
||||
The canonical provider IDs are the 42 entries of `ProviderKind::ALL`
|
||||
(`crates/config/src/provider_kind.rs`), in that order:
|
||||
|
||||
|
||||
Reference in new issue
Block a user