Merge remote-tracking branch 'origin/main' into test/compute-release-completion

* feat: Ollama System One (Jev) decision models and triage routing

Add /v1/systemone client, ticket triage, CLI, praisonai-ts, optional typesafe-sdk extra, unit tests. Live-validated with tev1:0.8b.

* fix: harden System One decisions (security, routing, parity trackers)

- system_one: scope TYPESAFE_API_KEY to TYPESAFE_BASE_URL hosts only; validate
  base URL (reject non-http(s) and plain-HTTP to non-loopback); fix timeout
  sentinel so explicit 120s is honoured (None is the unset sentinel).
- routing: reject empty/non-choice route questions; trigger fallback when
  confidence is missing; preserve the prompt over extra_state in triage state.
- CLI: validate --route-question is a choice question; print JSON via builtin so
  global --json mode no longer swallows decision output.
- pyproject: include decisions extra in [all].
- TS: normalize bare OLLAMA_HOST with http scheme; use getChoice + own-key route
  check; await async function targets; export routing API from package root.
- Regenerate parity tracker artifacts (ts + rust) for new decision symbols.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-authored-by: Mervin Praison <[email protected]>

* chore: regenerate TS parity tracker after rebase on main

---------

Co-authored-by: Dhivya-Bharathy <[email protected]>
Co-authored-by: praisonai-triage-agent[bot] <272766704+praisonai-triage-agent[bot]@users.noreply.github.com>
Co-authored-by: Mervin Praison <[email protected]>

.github/scripts/merge-gate.js

@@ -16,6 +16,9 @@ const PR_MAX_AUTO_ADDITIONS = 800;
 const PR_MAX_AUTO_FILES = 30;
 const MANUAL_ONLY_LABELS = new Set(['security', 'breaking-change', 'needs-manual-review', 'release']);
 const WORKFLOW_ONLY_LABEL = 'merge-gate-ci-only';
+const EXTERNAL_EXAMPLE_PATH_PREFIX = 'examples/tools/external/';
+const EXTERNAL_EXAMPLE_LABEL = 'pipeline/external-example';
+const MAINTAINER_ACCEPT_EXAMPLE_LABEL = 'maintainer-accept-example';
 const CI_ONLY_PATH_PREFIXES = ['.github/workflows/', '.github/actions/', '.github/scripts/merge-gate'];
 const SDK_PATH_PREFIXES = ['src/praisonai-agents/', 'src/praisonai/', 'src/praisonai-ts/'];
 const TS_SDK_SOURCE_PREFIX = 'src/praisonai-ts/src/';
@@ -328,7 +331,7 @@ function finalClaudeCompletedOnSha(comments, headPushedAt) {
 }
 
 const FINAL_CLAUDE_REVIEW_BODY =
-  '@claude You are the FINAL architecture reviewer. If the branch is under MervinPraison/PraisonAI (not a fork), you are able to make modifications to this branch and push directly. SCOPE: Review changes in this PR. Python SDK: praisonaiagents, praisonai. TypeScript SDK: src/praisonai-ts/. Do NOT modify src/praisonai-rust. Read ALL comments above from Gemini, Qodo, CodeRabbit, and Copilot carefully before responding.\n\n**MANDATORY READ (before reviewing):**\n- Always read src/praisonai-agents/AGENTS.md\n- If this PR touches src/praisonai-ts/, also read src/praisonai-ts/AGENTS.md §2.1.2 (TS triage + PR review checklist)\n\n**Phase 1: Review per AGENTS.md**\n1. Protocol-driven: check heavy implementations vs core SDK\n2. Backward compatible: ensure zero feature regressions\n3. Performance: no hot-path regressions\n4. SDK value: review in depth whether the change genuinely adds value to the SDK — never add features for the sake of adding them. It must strengthen the SDK (simpler, more user-friendly, robust, world-class, secure). If it does not clearly add value, request changes or recommend rejecting/closing rather than merging scope creep\n5. Do not bloat the Agent class with additional params — only if absolutely required; we already support many params.\n6. Repo routing: agent-callable tools → PraisonAI-Tools; lifecycle plugins → PraisonAI-Plugins; optional sandbox backends → PraisonAI-Plugins (`praisonai.sandbox` entry point) — request changes if wrongly added to praisonaiagents/\n\n**MANDATORY COMMENT FORMAT — include this Phase 1 table in your review comment:**\n#### Phase 1 — AGENTS.md review\n| Check | Result |\n|---|---|\n| Protocol-driven / no heavy impl in core | ✅ or ❌ + one-line rationale |\n| Backward compatible | ✅ or ❌ + one-line rationale |\n| Performance (hot path) | ✅ or ❌ + one-line rationale |\n| **SDK value** | ✅ or ❌ + one-line rationale (explicitly judge whether the change strengthens the SDK) |\n| No Agent param bloat | ✅ or ❌ + one-line rationale |\n| Repo routing | ✅ or ❌ + one-line rationale |\n\nFor TypeScript PRs (src/praisonai-ts/), also add:\n| TS types / parity / tests | ✅ or ❌ + one-line rationale (npm run build && npm test) |\n\n**Phase 2: FIX Valid Issues**\n7. For any VALID bugs or architectural flaws found by Gemini, CodeRabbit, Qodo, Copilot, or any other reviewer: implement the fix\n8. Also independently identify and fix any gaps or issues you find in the changed code — do not rely only on prior reviewer feedback\n9. Push all code fixes directly to THIS branch (do NOT create a new PR)\n10. Comment a summary of exact files modified and what you skipped\n\n**Phase 3: Final Verdict**\n11. If all issues are resolved, approve the PR / close the Issue\n12. If blocking issues remain, request changes / leave clear action items';
+  '@claude You are the FINAL architecture reviewer. If the branch is under MervinPraison/PraisonAI (not a fork), you are able to make modifications to this branch and push directly. SCOPE: Review changes in this PR. Python SDK: praisonaiagents, praisonai. TypeScript SDK: src/praisonai-ts/. Do NOT modify src/praisonai-rust. Read ALL comments above from Gemini, Qodo, CodeRabbit, and Copilot carefully before responding.\n\n**MANDATORY READ (before reviewing):**\n- Always read src/praisonai-agents/AGENTS.md\n- If this PR touches src/praisonai-ts/, also read src/praisonai-ts/AGENTS.md §2.1.2 (TS triage + PR review checklist)\n\n**Phase 1: Review per AGENTS.md**\n1. Protocol-driven: check heavy implementations vs core SDK\n2. Backward compatible: ensure zero feature regressions\n3. Performance: no hot-path regressions\n4. SDK value: review in depth whether the change genuinely adds value to the SDK — never add features for the sake of adding them. It must strengthen the SDK (simpler, more user-friendly, robust, world-class, secure). If it does not clearly add value, request changes or recommend rejecting/closing rather than merging scope creep\n5. Do not bloat the Agent class with additional params — only if absolutely required; we already support many params.\n6. Repo routing: agent-callable tools → PraisonAI-Tools; lifecycle plugins → PraisonAI-Plugins; optional sandbox backends → PraisonAI-Plugins (`praisonai.sandbox` entry point) — request changes if wrongly added to praisonaiagents/\n6b. **External tool examples (`examples/tools/external/**`):** If the diff is confined to that path, mark **SDK value** ❌ for monorepo merge — redirect the contributor to PraisonAI-Tools; do **not** Approve for merge unless the PR has the `maintainer-accept-example` label.\n\n**MANDATORY COMMENT FORMAT — include this Phase 1 table in your review comment:**\n#### Phase 1 — AGENTS.md review\n| Check | Result |\n|---|---|\n| Protocol-driven / no heavy impl in core | ✅ or ❌ + one-line rationale |\n| Backward compatible | ✅ or ❌ + one-line rationale |\n| Performance (hot path) | ✅ or ❌ + one-line rationale |\n| **SDK value** | ✅ or ❌ + one-line rationale (explicitly judge whether the change strengthens the SDK) |\n| No Agent param bloat | ✅ or ❌ + one-line rationale |\n| Repo routing | ✅ or ❌ + one-line rationale |\n\nFor TypeScript PRs (src/praisonai-ts/), also add:\n| TS types / parity / tests | ✅ or ❌ + one-line rationale (npm run build && npm test) |\n\n**Phase 2: FIX Valid Issues**\n7. For any VALID bugs or architectural flaws found by Gemini, CodeRabbit, Qodo, Copilot, or any other reviewer: implement the fix\n8. Also independently identify and fix any gaps or issues you find in the changed code — do not rely only on prior reviewer feedback\n9. Push all code fixes directly to THIS branch (do NOT create a new PR)\n10. Comment a summary of exact files modified and what you skipped\n\n**Phase 3: Final Verdict**\n11. If all issues are resolved, approve the PR / close the Issue\n12. If blocking issues remain, request changes / leave clear action items';
 
 async function getMergeState(github, owner, repo, prNumber) {
   const query = `
@@ -583,6 +586,11 @@ function isCiOnlyChange(files) {
   return files.every((f) => CI_ONLY_PATH_PREFIXES.some((p) => f.filename.startsWith(p)));
 }
 
+function isExternalExampleOnlyChange(files) {
+  if (!files.length) return false;
+  return files.every((f) => f.filename.startsWith(EXTERNAL_EXAMPLE_PATH_PREFIX));
+}
+
 function isInternalPullRequestLink(link, owner, repo) {
   if (link?.base?.repo?.full_name === `${owner}/${repo}`) return true;
   const baseUrl = link?.base?.repo?.url || '';
@@ -973,6 +981,15 @@ async function evaluatePipelineQuiescent(github, owner, repo, prNumber, core, op
 
   const pullFiles = await listPullFiles(github, owner, repo, prNumber);
 
+  if (
+    isExternalExampleOnlyChange(pullFiles) &&
+    !ctx.labels.includes(MAINTAINER_ACCEPT_EXAMPLE_LABEL)
+  ) {
+    reasons.push(
+      'external example only — belongs in PraisonAI-Tools (add maintainer-accept-example to opt in)'
+    );
+  }
+
   const agentChange = getAgentPyChangeFromFiles(pullFiles);
   const agentManual = manualReviewReasonForAgentPy(agentChange);
   if (agentManual) reasons.push(agentManual);
@@ -1019,6 +1036,13 @@ async function listPrNumbersForMergeGateScan(github, owner, repo, core) {
   }
   const mergeReady = mergeReadyIssues
     .filter((issue) => issue.pull_request)
+    .filter((issue) => {
+      const labels = (issue.labels || []).map((l) => (typeof l === 'string' ? l : l.name));
+      if (labels.includes(EXTERNAL_EXAMPLE_LABEL) && !labels.includes(MAINTAINER_ACCEPT_EXAMPLE_LABEL)) {
+        return false;
+      }
+      return true;
+    })
     .map((issue) => issue.number)
     .sort((a, b) => a - b);
 
@@ -1146,6 +1170,10 @@ module.exports = {
   claudeRunBlocksPr,
   hasBlockingClaudeRunForPr,
   isCiOnlyChange,
+  isExternalExampleOnlyChange,
+  EXTERNAL_EXAMPLE_PATH_PREFIX,
+  EXTERNAL_EXAMPLE_LABEL,
+  MAINTAINER_ACCEPT_EXAMPLE_LABEL,
   isInternalPullRequestLink,
   resolvePrNumberFromLinkedPullRequests,
   resolvePrNumberFromHeadBranch,

.github/scripts/pipeline-status-selftest.js

@@ -73,4 +73,33 @@ assert(
   )
 );
 
+assert(
+  'external example only path',
+  mg.isExternalExampleOnlyChange([{ filename: 'examples/tools/external/arxiv/tool.py' }])
+);
+assert(
+  'mixed paths not external-example only',
+  !mg.isExternalExampleOnlyChange([
+    { filename: 'examples/tools/external/foo.py' },
+    { filename: 'src/praisonai-agents/foo.py' },
+  ])
+);
+
+assert(
+  'upstream head repo',
+  ps.isUpstreamHeadRepo(
+    { head: { repo: { full_name: 'MervinPraison/PraisonAI' } } },
+    'MervinPraison',
+    'PraisonAI'
+  )
+);
+assert(
+  'fork head repo still synced (not upstream)',
+  !ps.isUpstreamHeadRepo(
+    { head: { repo: { full_name: 'ai-mrscraper/PraisonAI' } } },
+    'MervinPraison',
+    'PraisonAI'
+  )
+);
+
 process.exit(failed ? 1 : 0);

.github/scripts/pipeline-status.js

@@ -37,8 +37,17 @@ const LABEL_SPECS = [
   { name: 'pipeline/blocked:cooldown', color: 'fef2c0', description: 'Blocked: post-push or @claude cooldown' },
   { name: 'pipeline/blocked:stale-final', color: 'f9d0c4', description: 'Blocked: FINAL stale after new commits' },
   { name: 'pipeline/blocked:no-final', color: 'ededed', description: 'Blocked: no FINAL @claude trigger yet' },
+  {
+    name: 'pipeline/external-example',
+    color: 'bfd4f2',
+    description: 'Diff only under examples/tools/external/ — not merge-ready without maintainer-accept-example',
+  },
 ];
 
+// PRs with pipeline/external-example cannot reach pipeline/merge-ready unless
+// maintainers add maintainer-accept-example (enforced in merge-gate evaluatePipelineQuiescent,
+// same fail-closed pattern as fork PRs).
+
 function deriveStage(comments, evalResult) {
   if (evalResult.ready) return 'pipeline/merge-ready';
   if (!chain.chainKickPosted(comments)) return 'pipeline/reviews-pending';
@@ -63,7 +72,8 @@ function reasonToBlockerLabel(reason) {
     r.includes('without test') ||
     r.includes('files changed') ||
     r.includes('possible secret') ||
-    r.includes('agent.py')
+    r.includes('agent.py') ||
+    r.includes('external example only')
   ) return 'pipeline/blocked:manual-review';
   if (r.includes('recent @claude') || r.includes('post-push buffer')) return 'pipeline/blocked:cooldown';
   if (r.includes('stale final')) return 'pipeline/blocked:stale-final';
@@ -133,11 +143,16 @@ async function syncPipelineLabels(github, owner, repo, prNumber, core) {
     github, owner, repo, prNumber, core
   );
   const { stage, blockers, all } = computePipelineLabels(ctx.comments, evalResult);
+  const pullFiles = await mergeGate.listPullFiles(github, owner, repo, prNumber);
+  const auxiliary = mergeGate.isExternalExampleOnlyChange(pullFiles)
+    ? [mergeGate.EXTERNAL_EXAMPLE_LABEL]
+    : [];
+  const labelSet = [...all, ...auxiliary];
 
   const current = ctx.labels.filter((l) => l.startsWith(PIPELINE_PREFIX));
-  const desired = new Set(all);
+  const desired = new Set(labelSet);
   const toRemove = current.filter((l) => !desired.has(l) && l !== 'pipeline/merged');
-  const toAdd = all.filter((l) => !current.includes(l));
+  const toAdd = labelSet.filter((l) => !current.includes(l));
 
   for (const name of toRemove) {
     try {
@@ -199,6 +214,11 @@ async function dispatchMergeGateForOldestReady(github, owner, repo, readyCandida
   return 0;
 }
 
+function isUpstreamHeadRepo(pr, owner, repo) {
+  const headFull = pr.head?.repo?.full_name;
+  return !headFull || headFull === `${owner}/${repo}`;
+}
+
 async function syncOpenPullRequests(github, owner, repo, options, core) {
   const { maxPrs = 20, dispatchMergeGate = true } = options || {};
   await ensurePipelineLabels(github, owner, repo, core);
@@ -223,9 +243,8 @@ async function syncOpenPullRequests(github, owner, repo, options, core) {
   for (const pr of prs) {
     if (synced >= maxPrs) break;
     if (pr.draft) continue;
-    if (pr.head?.repo?.full_name && pr.head.repo.full_name !== `${owner}/${repo}`) continue;
     const result = await syncPipelineLabels(github, owner, repo, pr.number, core);
-    if (result.ready) {
+    if (result.ready && isUpstreamHeadRepo(pr, owner, repo)) {
       readyCandidates.push({
         prNumber: pr.number,
         createdAt: result.createdAt || new Date(pr.created_at).getTime(),
@@ -248,6 +267,7 @@ module.exports = {
   STAGE_LABELS,
   BLOCKER_LABELS,
   ALL_PIPELINE_LABELS,
+  isUpstreamHeadRepo,
   deriveStage,
   deriveBlockerLabels,
   computePipelineLabels,

.github/workflows/agents-suite.yml

@@ -93,7 +93,7 @@ jobs:
           # a CI-shaped environment rather than trusting a developer machine
           # where every sibling is already installed.
           pip install -e src/praisonai
-          pip install pytest pytest-asyncio pytest-timeout
+          pip install pytest pytest-asyncio pytest-timeout aiofiles
       - name: Run the unit suite
         working-directory: src/praisonai-agents
         env:

.github/workflows/merge-conflict-claude.yml

@@ -117,13 +117,25 @@ jobs:
               const labels = issue.labels.map((l) => l.name);
 
               if (status !== 'DIRTY' && labels.includes(LABEL)) {
-                await github.rest.issues.removeLabel({
-                  owner,
-                  repo,
-                  issue_number: prNumber,
-                  name: LABEL,
-                });
-                core.info(`Removed ${LABEL} from PR #${prNumber}`);
+                try {
+                  await github.rest.issues.removeLabel({
+                    owner,
+                    repo,
+                    issue_number: prNumber,
+                    name: LABEL,
+                  });
+                  core.info(`Removed ${LABEL} from PR #${prNumber}`);
+                } catch (err) {
+                  // Fork PRs may return 403/404 when the token cannot mutate labels on the head repo's issue.
+                  // For upstream PRs a 403/404 is a genuine failure that must surface, otherwise the
+                  // merge-blocking label stays in place while the check reports success.
+                  const isFork = headRepo && headRepo !== baseRepo;
+                  if (isFork && (err.status === 403 || err.status === 404)) {
+                    core.info(`Skip remove ${LABEL} on fork PR #${prNumber}: ${err.status}`);
+                  } else {
+                    throw err;
+                  }
+                }
               }
 
               if (status !== 'DIRTY') return { skipped: true, reason: 'not dirty' };

.github/workflows/pipeline-status-sync.yml

@@ -1,6 +1,6 @@
 name: Pipeline Status Sync
 
-# Sync pipeline/* stage and blocker labels on open internal PRs.
+# Sync pipeline/* stage and blocker labels on open PRs (fork heads included; merge gate only for upstream branches).
 
 on:
   schedule:

.github/workflows/praisonai-pr-review.yml

@@ -34,10 +34,24 @@ jobs:
         id: config
         env:
           HAS_APP_CREDENTIALS: ${{ secrets.PRAISONAI_APP_ID != '' && secrets.PRAISONAI_APP_PRIVATE_KEY != '' }}
-        run: echo "configured=$HAS_APP_CREDENTIALS" >> "$GITHUB_OUTPUT"
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: |
+          echo "configured=$HAS_APP_CREDENTIALS" >> "$GITHUB_OUTPUT"
+          openai_ready=true
+          if [ -z "$OPENAI_API_KEY" ] || [ "$OPENAI_API_KEY" = "not-needed" ]; then
+            openai_ready=false
+            if [ "$HAS_APP_CREDENTIALS" = "true" ]; then
+              echo "::notice::Skipping PraisonAI PR review: OPENAI_API_KEY is missing or a placeholder."
+            fi
+          fi
+          if [ "$HAS_APP_CREDENTIALS" = "true" ] && [ "$openai_ready" = "true" ]; then
+            echo "ready=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "ready=false" >> "$GITHUB_OUTPUT"
+          fi
 
       - name: Determine checkout ref
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         id: dest
         env:
           ISSUE_NUMBER: ${{ github.event.issue.number }}
@@ -53,7 +67,7 @@ jobs:
           fi
 
       - name: Checkout Repository
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         uses: actions/checkout@v4
         with:
           persist-credentials: false
@@ -61,7 +75,7 @@ jobs:
           fetch-depth: 0
           
       - name: Generate GitHub App Token
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         id: generate_token
         uses: tibdex/github-app-token@v2
         with:
@@ -69,17 +83,17 @@ jobs:
           private_key: ${{ secrets.PRAISONAI_APP_PRIVATE_KEY }}
           
       - name: Set up Python
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         uses: actions/setup-python@v5
         with:
           python-version: '3.11'
           
       - name: Install PraisonAI
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         run: pip install "praisonai[all]"
         
       - name: Run PraisonAI PR Review
-        if: steps.config.outputs.configured == 'true'
+        if: steps.config.outputs.ready == 'true'
         env:
           GITHUB_TOKEN: ${{ steps.generate_token.outputs.token }}
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

.github/workflows/test-optimized.yml

@@ -42,7 +42,7 @@ jobs:
   # ============================================
   smoke:
     runs-on: ubuntu-latest
-    timeout-minutes: 12
+    timeout-minutes: 16
     steps:
     - uses: actions/checkout@v4
       with:
@@ -773,7 +773,9 @@ jobs:
     env:
       # Bump deliberately: this is the version everything in this job was
       # verified against. Latest at time of writing is v0.33.3.
-      OLLAMA_VERSION: v0.33.2
+      # No leading "v": install.sh passes ?version=$OLLAMA_VERSION and ollama.com
+      # redirects to GitHub tag v$version (vv0.33.2 → 404 if set to v0.33.2).
+      OLLAMA_VERSION: "0.33.2"
       # 522 MB, capabilities: completion, tools, thinking.
       OLLAMA_TEST_MODEL: qwen3:0.6b
 

.gitignore

@@ -122,3 +122,4 @@ jobs/*
 # Local Android SDK/NDK/JDK for building praisonai-mobile on a laptop without
 # Android Studio (how: .github/workflows/mobile-platforms.yml). Several GB.
 .android-sdk/
+.worktrees/

api.md

@@ -776,7 +776,7 @@ Methods:
 * <code title="class AgentTeam">AgentTeam.<a href="./src/praisonai-agents/praisonaiagents/agents/agents.py">wait_for_completions</a>(timeout: Optional[float] = None, agent_ids: Optional[List[str]] = None) -> List[SubAgentCompletionEvent]</code>
 * <code title="class AgentTeam">AgentTeam.<a href="./src/praisonai-agents/praisonaiagents/agents/agents.py">where_does_it_run</a>() -> str</code>
 * <code title="class AutoApproveBackend">AutoApproveBackend.<a href="./src/praisonai-agents/praisonaiagents/approval/backends.py">request_approval</a>(request: ApprovalRequest) -> ApprovalDecision</code>
-* <code title="class AutoApproveBackend">AutoApproveBackend.<a href="./src/praisonai-agents/praisonaiagents/approval/backends.py">request_approval_sync</a>(request: ApprovalRequest) -> ApprovalDecision</code>
+* <code title="class AutoApproveBackend">AutoApproveBackend.<a href="./src/praisonai-agents/praisonaiagents/approval/backends.py">request_approval_sync</a>(request: ApprovalRequest, timeout: Optional[float] = None) -> ApprovalDecision</code>
 * <code title="class RunOutcome">RunOutcome.<a href="./src/praisonai-agents/praisonaiagents/agent/run_outcome.py">completed</a>(output: Optional[str] = None) -> 'RunOutcome'</code>
 * <code title="class RunOutcome">RunOutcome.<a href="./src/praisonai-agents/praisonaiagents/agent/run_outcome.py">from_exception</a>(exc: BaseException, output: Optional[str] = None) -> 'RunOutcome'</code>
 * <code title="class RunOutcome">RunOutcome.<a href="./src/praisonai-agents/praisonaiagents/agent/run_outcome.py">succeeded</a>() -> bool</code>
@@ -1001,6 +1001,8 @@ export type { DbAdapter, DbConfig, DbMessage, DbRun, DbTrace } from "./db";
 export { MemoryPostgresAdapter, NeonPostgresAdapter, PostgresSessionStorage, createMemoryPostgres, createNeonPostgres, createPostgresSessionStorage } from "./db/postgres";
 export { MemoryRedisAdapter, UpstashRedisAdapter, createMemoryRedis, createUpstashRedis } from "./db/redis";
 export { SQLiteAdapter, createSQLiteAdapter } from "./db/sqlite";
+export { KNOWN_DECISION_MODELS, choiceQuestion, defaultTicketTriageQuestions, getChoice, getNoul, getScore, isDecisionModel, noulQuestion, resolveSystemOneBaseUrl, scoreQuestion, systemOne, triagedStart } from "./decisions";
+export type { DecisionAgentLike, DecisionRoutePlan, QuestionSpec, RouteTarget, SystemOneAnswer, SystemOneOptions, SystemOneResult } from "./decisions";
 export { // Functions
   // NB: the unaliased registerDisplayCallback name belongs to the per-type
   // callback registry (callbacks module, // Types
@@ -1018,7 +1020,8 @@ export { // Bot types
   FailoverManager, // Enums
   SandboxStatus, // Gateway types
   type GatewayConfig, // Other types
-  type ProviderStatus, AutonomyLevel, GatewayEventType, RagRetrievalPolicy } from "./gateway";
+  type ProviderStatus, // Session-projection reducer (Issue #5324)
+  SessionProjection, AutonomyLevel, GatewayEventType, RagRetrievalPolicy } from "./gateway";
 export { LLMGuardrail, createLLMGuardrail } from "./guardrails/llm-guardrail";
 export { DisplayTypes, HooksManager, WorkflowHooksExecutor, clearAllCallbacks, clearApprovalCallback, createHooksManager, createLoggingOperationHooks, createLoggingWorkflowHooks, createTimingWorkflowHooks, createValidationOperationHooks, createWorkflowHooks, executeCallback, executeSyncCallback, getRegisteredDisplayTypes, hasApprovalCallback, registerApprovalCallback, registerDisplayCallback, requestApproval, unregisterDisplayCallback } from "./hooks";
 export { // Computer Use

docker/Dockerfile.chat

@@ -16,7 +16,7 @@ RUN mkdir -p /root/.praison
 # Install Python packages (using latest versions)
 RUN pip install --no-cache-dir \
     praisonai_tools \
-    "praisonai>=4.7.10" \
+    "praisonai>=4.7.12" \
     "praisonai[chat]" \
     "embedchain[github,youtube]"
 

docker/Dockerfile.dev

@@ -20,7 +20,7 @@ RUN mkdir -p /root/.praison
 # Install Python packages (using latest versions)
 RUN pip install --no-cache-dir \
     praisonai_tools \
-    "praisonai>=4.7.10" \
+    "praisonai>=4.7.12" \
     "praisonai[ui]" \
     "praisonai[chat]" \
     "praisonai[realtime]" \

docker/Dockerfile.ui

@@ -16,7 +16,7 @@ RUN mkdir -p /root/.praison
 # Install Python packages (using latest versions)
 RUN pip install --no-cache-dir \
     praisonai_tools \
-    "praisonai>=4.7.10" \
+    "praisonai>=4.7.12" \
     "praisonai[ui]" \
     "praisonai[crewai]"
 

src/praisonai-agents/DOCS_PARITY.md

@@ -18,76 +18,76 @@ This report compares **Python SDK feature categories** against **Python document
 | Category | Features | Docs | Lines |
 |----------|----------|------|-------|
 | ✅ AGUI | 1 | 2 | 511 |
-| ✅ Agent | 22 | 59 | 18970 |
+| ✅ Agent | 22 | 59 | 19085 |
 | ✅ Agent-to-Agent (A2A) | 1 | 7 | 2996 |
 | ✅ Agent-to-User (A2U) | 1 | 3 | 701 |
-| ✅ Approval | 1 | 8 | 3941 |
+| ✅ Approval | 1 | 8 | 4028 |
 | ✅ Audio | 2 | 12 | 1000 |
-| ✅ Auto Generation | 5 | 12 | 3879 |
-| ✅ Autonomy | 3 | 7 | 2794 |
-| ✅ Bots | 7 | 36 | 14882 |
+| ✅ Auto Generation | 5 | 12 | 3941 |
+| ✅ Autonomy | 3 | 7 | 2808 |
+| ✅ Bots | 7 | 39 | 16300 |
 | ✅ Budget | 1 | 1 | 287 |
-| ✅ CLI | 5 | 124 | 38936 |
+| ✅ CLI | 5 | 124 | 39199 |
 | ✅ Chunking | 2 | 2 | 435 |
 | ✅ Citations | 2 | 1 | 202 |
 | ✅ Code Execution | 2 | 14 | 5022 |
 | ✅ Conditions | 1 | 3 | 1386 |
-| ✅ Configuration | 3 | 8 | 3341 |
-| ✅ Context Management | 16 | 36 | 14289 |
+| ✅ Configuration | 3 | 8 | 3350 |
+| ✅ Context Management | 16 | 37 | 14687 |
 | ✅ Deep Research | 8 | 2 | 591 |
 | ✅ Display | 6 | 3 | 844 |
 | ✅ Embeddings | 6 | 23 | 2051 |
-| ✅ Evaluation | 1 | 9 | 3537 |
-| ✅ Events | 1 | 2 | 780 |
-| ✅ Execution | 3 | 4 | 1693 |
-| ✅ Failover | 2 | 1 | 480 |
+| ✅ Evaluation | 1 | 9 | 3637 |
+| ✅ Events | 1 | 2 | 782 |
+| ✅ Execution | 3 | 4 | 1694 |
+| ✅ Failover | 2 | 1 | 537 |
 | ✅ Files | 2 | 7 | 2376 |
 | ✅ Flow | 1 | 3 | 811 |
-| ✅ Gateway | 7 | 89 | 34188 |
-| ✅ Guardrails | 5 | 4 | 2529 |
+| ✅ Gateway | 7 | 99 | 38257 |
+| ✅ Guardrails | 5 | 4 | 2642 |
 | ✅ Handoffs | 11 | 6 | 2748 |
-| ✅ Hooks | 2 | 9 | 4990 |
-| ✅ Image | 1 | 11 | 1255 |
-| ✅ Knowledge | 4 | 20 | 6184 |
-| ✅ LLM | 3 | 15 | 5946 |
-| ✅ Loops | 4 | 5 | 1475 |
-| ✅ MCP | 1 | 64 | 15068 |
-| ✅ Memory | 6 | 21 | 8186 |
+| ✅ Hooks | 2 | 9 | 5230 |
+| ✅ Image | 1 | 11 | 1295 |
+| ✅ Knowledge | 4 | 20 | 6260 |
+| ✅ LLM | 3 | 15 | 5958 |
+| ✅ Loops | 4 | 5 | 1478 |
+| ✅ MCP | 1 | 64 | 15116 |
+| ✅ Memory | 6 | 21 | 8198 |
 | ✅ OCR | 2 | 1 | 237 |
 | ✅ Observability | 2 | 23 | 2859 |
 | ✅ Optimizer | 1 | 2 | 792 |
-| ✅ Output | 3 | 5 | 1273 |
+| ✅ Output | 3 | 5 | 1289 |
 | ✅ Parallel Execution | 3 | 2 | 528 |
-| ✅ Planning | 6 | 7 | 1825 |
-| ✅ Plugins | 8 | 5 | 3404 |
+| ✅ Planning | 6 | 7 | 1827 |
+| ✅ Plugins | 8 | 5 | 4112 |
 | ✅ Prompts | 2 | 9 | 2043 |
-| ✅ Providers | 1 | 58 | 8732 |
+| ✅ Providers | 1 | 58 | 8751 |
 | ✅ Query | 1 | 2 | 736 |
 | ✅ RAG | 5 | 15 | 3152 |
 | ✅ Realtime | 2 | 5 | 668 |
 | ✅ Reflection | 3 | 3 | 732 |
 | ✅ Retrieval | 2 | 5 | 1046 |
 | ✅ Routing | 1 | 2 | 430 |
-| ✅ Sandbox | 5 | 7 | 3712 |
-| ✅ Security | 1 | 3 | 3607 |
-| ✅ Sessions | 4 | 21 | 8285 |
-| ✅ Skills | 6 | 16 | 5988 |
+| ✅ Sandbox | 5 | 7 | 3720 |
+| ✅ Security | 1 | 3 | 3642 |
+| ✅ Sessions | 4 | 21 | 8389 |
+| ✅ Skills | 6 | 16 | 6045 |
 | ✅ Tasks | 2 | 8 | 3266 |
 | ✅ Telemetry | 1 | 2 | 660 |
 | ✅ Templates | 1 | 9 | 1859 |
-| ✅ Tools | 12 | 145 | 44594 |
+| ✅ Tools | 12 | 147 | 45414 |
 | ✅ Tracing | 3 | 2 | 139 |
 | ✅ Vector Store | 1 | 12 | 1219 |
 | ✅ Video | 2 | 10 | 1303 |
 | ✅ Vision | 2 | 1 | 329 |
-| ✅ Web | 3 | 10 | 2815 |
-| ✅ Workflows | 5 | 21 | 8862 |
+| ✅ Web | 3 | 10 | 2822 |
+| ✅ Workflows | 5 | 21 | 8933 |
 
 ## Documentation Without Features
 
 These docs exist but don't match any implemented feature category:
 
-- ℹ️ Database (43 docs, 7657 lines)
+- ℹ️ Database (44 docs, 8359 lines)
 - ℹ️ Documents (1 docs, 810 lines)
 - ℹ️ Teams (1 docs, 214 lines)
 

src/praisonai-agents/benchmarks/simple_benchmark.py

@@ -6,8 +6,13 @@ Compares agent instantiation times across popular AI agent frameworks.
 
 Usage:
     python benchmarks/simple_benchmark.py
+    python benchmarks/simple_benchmark.py --skip-crewai --json
+    python benchmarks/simple_benchmark.py --json --fail-over-threshold
 """
 
+import json
+import os
+import sys
 import time
 from typing import Literal
 import argparse
@@ -16,6 +21,14 @@ import argparse
 ITERATIONS = 100
 WARMUP = 10
 
+THRESHOLDS_PATH = os.path.join(os.path.dirname(__file__), 'thresholds.json')
+
+
+def load_thresholds(path: str = THRESHOLDS_PATH) -> dict:
+    """Load benchmark thresholds from thresholds.json."""
+    with open(path) as f:
+        return json.load(f)
+
 
 def sample_tool(city: Literal['nyc', 'sf']):
     """Sample tool for benchmark testing."""
@@ -36,35 +49,44 @@ def measure_instantiation(create_fn, iterations=ITERATIONS, warmup=WARMUP):
     return sum(times) / len(times)
 
 
-def run_benchmark():
+def run_benchmark(skip_crewai: bool = False, quiet: bool = False,
+                  iterations: int = ITERATIONS, warmup: int = WARMUP):
     """Run the benchmark across all available frameworks (without tools)."""
     results = {}
-    
-    print('=' * 60)
-    print('PraisonAI Agents - Performance Benchmark')
-    print('=' * 60)
-    print(f'\nIterations: {ITERATIONS}')
-    print('Metric: Agent instantiation time (microseconds)\n')
+
+    def log(msg: str = ''):
+        if not quiet:
+            print(msg)
+
+    def measure(create_fn):
+        return measure_instantiation(create_fn, iterations=iterations, warmup=warmup)
+
+    log('=' * 60)
+    log('PraisonAI Agents - Performance Benchmark')
+    log('=' * 60)
+    log(f'\nIterations: {iterations}')
+    log(f'Warmup: {warmup}')
+    log('Metric: Agent instantiation time (microseconds)\n')
     
     # PraisonAI (without tools)
-    print("Testing PraisonAI...")
+    log("Testing PraisonAI...")
     from praisonaiagents import Agent as PraisonAgent
     
-    results['PraisonAI'] = measure_instantiation(
+    results['PraisonAI'] = measure(
         lambda: PraisonAgent(name='Test', model='gpt-4o-mini', output="silent")
     )
     
-    results['PraisonAI (LiteLLM)'] = measure_instantiation(
+    results['PraisonAI (LiteLLM)'] = measure(
         lambda: PraisonAgent(name='Test', model='openai/gpt-4o-mini', output="silent")
     )
     
     # Other frameworks for comparison
-    print("Testing other frameworks...")
+    log("Testing other frameworks...")
     
     try:
         from agno.agent import Agent as AgnoAgent
         from agno.models.openai import OpenAIChat
-        results['Agno'] = measure_instantiation(
+        results['Agno'] = measure(
             lambda: AgnoAgent(model=OpenAIChat(id='gpt-4o-mini'))
         )
     except ImportError:
@@ -72,7 +94,7 @@ def run_benchmark():
     
     try:
         from pydantic_ai import Agent as PydanticAgent
-        results['PydanticAI'] = measure_instantiation(
+        results['PydanticAI'] = measure(
             lambda: PydanticAgent('openai:gpt-4o-mini')
         )
     except ImportError:
@@ -80,7 +102,7 @@ def run_benchmark():
     
     try:
         from agents import Agent as OpenAIAgent
-        results['OpenAI Agents SDK'] = measure_instantiation(
+        results['OpenAI Agents SDK'] = measure(
             lambda: OpenAIAgent(name='Test', model='gpt-4o-mini')
         )
     except ImportError:
@@ -96,47 +118,50 @@ def run_benchmark():
             """Get weather info."""
             return 'sunny' if city == 'sf' else 'cloudy'
         
-        results['LangGraph'] = measure_instantiation(
+        results['LangGraph'] = measure(
             lambda: create_react_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[get_weather_lg])
         )
     except ImportError:
         pass
     
-    try:
-        from crewai.agent import Agent as CrewAgent
-        from crewai.tools import tool as crewai_tool
-        
-        @crewai_tool("Weather Tool")
-        def get_weather_crew(city: Literal['nyc', 'sf']):
-            """Get weather info."""
-            return 'sunny' if city == 'sf' else 'cloudy'
-        
-        results['CrewAI'] = measure_instantiation(
-            lambda: CrewAgent(
-                role='Weather Agent',
-                goal='Provide weather info',
-                backstory='A weather expert',
-                tools=[get_weather_crew],
-                verbose=False
+    if skip_crewai:
+        log("Skipping CrewAI (--skip-crewai): optional framework import can hang on some platforms.")
+    else:
+        try:
+            from crewai.agent import Agent as CrewAgent
+            from crewai.tools import tool as crewai_tool
+
+            @crewai_tool("Weather Tool")
+            def get_weather_crew(city: Literal['nyc', 'sf']):
+                """Get weather info."""
+                return 'sunny' if city == 'sf' else 'cloudy'
+
+            results['CrewAI'] = measure(
+                lambda: CrewAgent(
+                    role='Weather Agent',
+                    goal='Provide weather info',
+                    backstory='A weather expert',
+                    tools=[get_weather_crew],
+                    verbose=False
+                )
             )
-        )
-    except ImportError:
-        pass
+        except ImportError:
+            pass
     
     # Print results
-    print('\n' + '=' * 60)
-    print('RESULTS')
-    print('=' * 60)
+    log('\n' + '=' * 60)
+    log('RESULTS')
+    log('=' * 60)
     
     baseline = results.get('PraisonAI', 1)
-    print(f"\n{'Framework':<25} {'Avg Time (μs)':<15} {'Relative':<10}")
-    print('-' * 50)
+    log(f"\n{'Framework':<25} {'Avg Time (μs)':<15} {'Relative':<10}")
+    log('-' * 50)
     
     for name, avg in sorted(results.items(), key=lambda x: x[1]):
         ratio = avg / baseline
-        print(f'{name:<25} {avg:<15.2f} {ratio:.2f}x')
+        log(f'{name:<25} {avg:<15.2f} {ratio:.2f}x')
     
-    print('\n' + '=' * 60)
+    log('\n' + '=' * 60)
     return results
 
 
@@ -163,8 +188,13 @@ def get_package_versions():
     return versions
 
 
-def save_results(results: dict, filename: str = 'BENCHMARK_RESULTS.md'):
-    """Save benchmark results to a markdown file and update README."""
+def save_results(results: dict, filename: str = 'BENCHMARK_RESULTS.md', *,
+                 iterations: int = ITERATIONS, log=print):
+    """Save benchmark results to a markdown file and update README.
+
+    Status messages are emitted via ``log`` (defaults to ``print``). Pass a
+    stderr logger when stdout must stay machine-readable (e.g. ``--json``).
+    """
     import os
     import re
     from datetime import datetime
@@ -191,7 +221,7 @@ def save_results(results: dict, filename: str = 'BENCHMARK_RESULTS.md'):
     with open(filepath, 'w') as f:
         f.write('# PraisonAI Agents - Benchmark Results\n\n')
         f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
-        f.write(f'**Iterations:** {ITERATIONS}\n')
+        f.write(f'**Iterations:** {iterations}\n')
         f.write('**Test:** Agent instantiation (without tools)\n\n')
         f.write('## Results\n\n')
         f.write('| Framework | Avg Time (μs) | Relative |\n')
@@ -208,17 +238,17 @@ def save_results(results: dict, filename: str = 'BENCHMARK_RESULTS.md'):
         f.write('python benchmarks/simple_benchmark.py\n')
         f.write('```\n')
     
-    print(f'\nResults saved to: {filepath}')
+    log(f'\nResults saved to: {filepath}')
     
     # Also update the main README.md
     readme_path = os.path.join(os.path.dirname(__file__), '..', '..', '..', 'README.md')
     if os.path.exists(readme_path):
-        update_readme(readme_path, table_rows)
+        update_readme(readme_path, table_rows, log=log)
     
     return filepath
 
 
-def update_readme(readme_path: str, table_rows: list):
+def update_readme(readme_path: str, table_rows: list, *, log=print):
     """Update the performance section in README.md with latest results."""
     import re
     
@@ -239,17 +269,54 @@ def update_readme(readme_path: str, table_rows: list):
         with open(readme_path, 'w') as f:
             f.write(content)
         
-        print(f'README.md updated: {readme_path}')
+        log(f'README.md updated: {readme_path}')
+
+
+def main():
+    parser = argparse.ArgumentParser(description='PraisonAI Agents - Performance Benchmark')
+    parser.add_argument('--save', action='store_true', help='Save results to file')
+    parser.add_argument('--skip-crewai', action='store_true',
+                        help='Skip the optional CrewAI benchmark (its import can hang on some platforms)')
+    parser.add_argument('--json', action='store_true',
+                        help='Emit results as JSON to stdout (for CI baselines/artifacts)')
+    parser.add_argument('--fail-over-threshold', action='store_true',
+                        help='Exit non-zero if PraisonAI instantiation exceeds the budget in thresholds.json')
+    args = parser.parse_args()
+
+    thresholds = load_thresholds()
+    iterations = int(thresholds.get('iterations', ITERATIONS))
+    warmup = int(thresholds.get('warmup', WARMUP))
+
+    results = run_benchmark(
+        skip_crewai=args.skip_crewai,
+        quiet=args.json,
+        iterations=iterations,
+        warmup=warmup,
+    )
+
+    if args.save:
+        # Keep stdout machine-readable under --json by sending save status to stderr.
+        save_log = (lambda msg='': print(msg, file=sys.stderr)) if args.json else print
+        save_results(results, iterations=iterations, log=save_log)
+    elif not args.json:
+        print('\nResults not saved (use --save flag to save results to file)')
+
+    exit_code = 0
+    if args.fail_over_threshold:
+        max_us = thresholds['praisonai_instantiation_us_max']
+        actual_us = results.get('PraisonAI')
+        if actual_us is not None and actual_us > max_us:
+            print(
+                f"ERROR: PraisonAI instantiation {actual_us:.2f} us > threshold {max_us} us",
+                file=sys.stderr,
+            )
+            exit_code = 1
+
+    if args.json:
+        print(json.dumps({'results': results, 'iterations': iterations, 'warmup': warmup}))
+
+    return exit_code
 
 
 if __name__ == '__main__':
-    parser = argparse.ArgumentParser(description='PraisonAI Agents - Performance Benchmark')
-    parser.add_argument('--save', action='store_true', help='Save results to file')
-    args = parser.parse_args()
-    
-    results = run_benchmark()
-    
-    if args.save:
-        save_results(results)
-    else:
-        print('\nResults not saved (use --save flag to save results to file)')
+    sys.exit(main())

src/praisonai-agents/benchmarks/thresholds.json

@@ -0,0 +1,5 @@
+{
+  "praisonai_instantiation_us_max": 450,
+  "iterations": 100,
+  "warmup": 10
+}

src/praisonai-agents/docs/local-model-layer/05-live-ci-job.md

@@ -183,7 +183,9 @@ banner. Key fragments:
     timeout-minutes: 15
     env:
       # Bump deliberately; the cache key and what we test both follow this.
-      OLLAMA_VERSION: v0.33.2
+      # No leading "v": install.sh passes ?version=$OLLAMA_VERSION and ollama.com
+      # redirects to GitHub tag v$version (vv0.33.2 → 404 if set to v0.33.2).
+      OLLAMA_VERSION: "0.33.2"
       OLLAMA_TEST_MODEL: qwen3:0.6b
     steps:
     - uses: actions/checkout@v4
@@ -204,9 +206,12 @@ banner. Key fragments:
       shell: bash
       run: |
         set -euo pipefail
-        curl -fsSL -o /tmp/ollama.tgz \
-          "https://github.com/ollama/ollama/releases/download/${OLLAMA_VERSION}/ollama-linux-amd64.tgz"
-        sudo tar -C /usr -xzf /tmp/ollama.tgz
+        # The official installer, VERSION-PINNED via OLLAMA_VERSION (which it
+        # honours). Since v0.33 the linux bundle ships as a .tar.zst with GPU
+        # runtimes bundled in, so a direct release-asset download is not a
+        # workable alternative; the installer handles the zst/tgz split.
+        sudo apt-get update -qq && sudo apt-get install -y -qq zstd
+        curl -fsSL https://ollama.com/install.sh | sh
         ollama --version
     - name: Start Ollama and pull model
       shell: bash

src/praisonai-agents/praisonaiagents/__init__.py

@@ -227,6 +227,20 @@ _LAZY_IMPORTS = {
     'aembed': ('praisonaiagents.embedding.embed', 'aembed'),
     'EmbeddingResult': ('praisonaiagents.embedding.result', 'EmbeddingResult'),
     'get_dimensions': ('praisonaiagents.embedding.dimensions', 'get_dimensions'),
+
+    # System One / Jev decision models (Ollama /v1/systemone)
+    'system_one': ('praisonaiagents.decisions', 'system_one'),
+    'asystem_one': ('praisonaiagents.decisions', 'asystem_one'),
+    'SystemOneResult': ('praisonaiagents.decisions', 'SystemOneResult'),
+    'choice_question': ('praisonaiagents.decisions', 'choice_question'),
+    'noul_question': ('praisonaiagents.decisions', 'noul_question'),
+    'score_question': ('praisonaiagents.decisions', 'score_question'),
+    'is_decision_model': ('praisonaiagents.decisions', 'is_decision_model'),
+    'DecisionRoutePlan': ('praisonaiagents.decisions', 'DecisionRoutePlan'),
+    'DecisionTriageRouter': ('praisonaiagents.decisions', 'DecisionTriageRouter'),
+    'triaged_start': ('praisonaiagents.decisions', 'triaged_start'),
+    'triage_decision': ('praisonaiagents.decisions', 'triage_decision'),
+    'default_ticket_triage_questions': ('praisonaiagents.decisions', 'default_ticket_triage_questions'),
     
     # Guardrails
     'GuardrailResult': ('praisonaiagents.guardrails', 'GuardrailResult'),

src/praisonai-agents/praisonaiagents/_lazy.py

@@ -37,6 +37,22 @@ _cache_lock = threading.RLock()
 # Global module cache
 _module_cache: Dict[str, Any] = {}
 
+# Import roots of praisonaiagents' declared *core* runtime dependencies (see
+# pyproject.toml ``[project].dependencies``). A lazy import that fails because
+# one of these is missing/broken indicates a genuinely broken install, which
+# must be surfaced rather than masked as a graceful ``None``. Optional
+# integrations whose own optional SDKs are absent are NOT in this set, so they
+# keep the documented ``None`` fallback.
+_CORE_REQUIRED_DEPENDENCIES = frozenset({
+    "pydantic",
+    "rich",
+    "openai",
+    "posthog",
+    "aiohttp",
+    "yaml",       # PyYAML — core dep; import root differs from the package name
+    "croniter",
+})
+
 
 def lazy_import(
     module_path: str,
@@ -220,9 +236,25 @@ def create_lazy_getattr_with_fallback(
                 _cache[name] = None
                 return None
             except ImportError as exc:
-                # Optional module not available: preserve the graceful None
-                # fallback, but record why so real import failures remain
-                # diagnosable instead of surfacing later as NoneType errors.
+                # Distinguish "optional integration not installed" from
+                # "a required *core* dependency is broken". A missing/broken core
+                # dependency (e.g. ``pydantic``) must be surfaced loudly instead
+                # of being masked as ``None`` (which resurfaces later as a
+                # confusing ``NoneType`` error). But an optional integration whose
+                # own optional SDK is absent must keep the documented graceful
+                # ``None`` fallback — the failing module name there belongs to
+                # that optional SDK, not to us. We therefore only escalate when
+                # the failing module is one of praisonaiagents' declared *core*
+                # runtime dependencies; everything else stays a soft None.
+                failing = getattr(exc, "name", None)
+                failing_root = failing.split(".")[0] if failing else None
+                if failing_root in _CORE_REQUIRED_DEPENDENCIES:
+                    raise ImportError(
+                        f"Failed to load {name!r} because importing {module_path!r} "
+                        f"raised: {exc}. This usually means a required core dependency "
+                        f"({failing_root!r}) is missing or broken, not that {name!r} is "
+                        f"an optional feature you chose not to install."
+                    ) from exc
                 _logger.debug(
                     "Lazy import of %r (%s.%s) failed: %s",
                     name, module_path, attr_name, exc,

src/praisonai-agents/praisonaiagents/_run_lock.py

@@ -0,0 +1,31 @@
+"""Shared lazy run-lock helper for core orchestration modules.
+
+AgentTeam (``agents/agents.py``) and Workflow/AgentFlow (``workflows/workflows.py``)
+both lazily mint a per-instance re-entrancy lock the first time a run begins.
+Creation is serialized through a single module-level guard (double-checked
+locking) so two threads racing into the first ``start()``/``astart()``/``run()``
+observe the *same* lock object rather than each minting and acquiring its own.
+"""
+import threading
+
+# Guards lazy creation of each instance's per-instance ``_run_lock`` so two
+# threads that first reach start()/astart()/run() concurrently observe the same
+# lock object (double-checked locking) rather than each minting its own.
+_RUN_LOCK_INIT_GUARD = threading.Lock()
+
+
+def ensure_run_lock(obj):
+    """Lazily mint ``obj._run_lock`` under a shared module guard.
+
+    ``getattr(obj, "_run_lock", None)`` works for both the dataclass-field case
+    (Workflow) and the plain-attribute case (AgentTeam), so the behaviour is
+    identical to the per-module implementations it replaces.
+    """
+    lock = getattr(obj, "_run_lock", None)
+    if lock is None:
+        with _RUN_LOCK_INIT_GUARD:
+            lock = getattr(obj, "_run_lock", None)
+            if lock is None:
+                lock = threading.Lock()
+                obj._run_lock = lock
+    return lock

src/praisonai-agents/praisonaiagents/agent/agent.py

@@ -7,10 +7,7 @@ from praisonaiagents._logging import get_logger
 import asyncio
 import contextlib
 import threading
-import concurrent.futures
-import random
-import re
-from typing import List, Optional, Any, Dict, Union, Literal, TYPE_CHECKING, Callable, Generator
+from typing import List, Optional, Any, Dict, Union, Literal, TYPE_CHECKING, Callable
 from collections import OrderedDict
 import inspect
 
@@ -1373,12 +1370,17 @@ class Agent(GoalLoopMixin, SteeringMixin, SandboxMixin, SkillReviewMixin, Unifie
                 array_mode=ArrayMode.PASSTHROUGH,
                 default=None,
             )
+        # Per-agent event-hook registry (isolates BEFORE_TOOL/AFTER_AGENT/...
+        # hooks so they never fire on unrelated agents sharing the default
+        # registry). A bare HookRegistry passed as ``hooks=`` still works too.
+        _hooks_registry = None
         if _hooks_config is not None:
             if isinstance(_hooks_config, list):
                 _hooks_list = _hooks_config
             elif isinstance(_hooks_config, HooksConfig):
                 step_callback = _hooks_config.on_step
                 _hooks_list = list(_hooks_config.middleware or [])
+                _hooks_registry = getattr(_hooks_config, "registry", None)
                 # Route on_step / on_tool_call onto the SAME middleware chain
                 # that already powers ``hooks=[...]`` (MiddlewareManager). Both
                 # keys used to be stored-and-never-called, so the TypeError that
@@ -2101,7 +2103,11 @@ class Agent(GoalLoopMixin, SteeringMixin, SandboxMixin, SkillReviewMixin, Unifie
         # Lazy init for HookRunner and StreamEventEmitter (zero overhead when not used)
         self.__hook_runner = None  # Will be initialized on first access
         self.__stream_emitter = None  # Will be initialized on first access
-        self._hooks_registry_param = hooks  # Store for lazy init
+        # Prefer a per-agent registry declared via HooksConfig(registry=...) so
+        # this agent's event hooks stay isolated; else fall back to the raw
+        # ``hooks`` value (a bare HookRegistry still isolates, anything else
+        # resolves to the shared default registry in the _hook_runner property).
+        self._hooks_registry_param = _hooks_registry if _hooks_registry is not None else hooks
         
         # Handle llm= deprecation: model= is the preferred parameter name
         # llm= still works but shows deprecation warning
@@ -2595,6 +2601,11 @@ class Agent(GoalLoopMixin, SteeringMixin, SandboxMixin, SkillReviewMixin, Unifie
         self._on_budget_exceeded = _on_budget_exceeded
         # Thread-safe cost/token tracking (Gap 1a fix)
         self._cost_lock = threading.Lock()
+        # Serializes the read-modify-write of _auto_save_last_index. Auto-save
+        # now runs via asyncio.to_thread, so two concurrent async runs sharing
+        # one Agent could otherwise read the same last-saved index in separate
+        # worker threads and each persist the same turns, duplicating history.
+        self._auto_save_lock = threading.Lock()
         self._total_cost = 0.0
         self._total_tokens_in = 0
         self._total_tokens_out = 0
@@ -2881,7 +2892,10 @@ Your Goal: {self.goal}
                     # Reuse the caller's manager as-is (no disk I/O side-effect).
                     _mgr = self._approval_permissions
                 else:
-                    _mgr = PermissionManager(agent_name=self.name)
+                    _mgr = PermissionManager(
+                        agent_name=self.name,
+                        workspace_root=os.getcwd(),
+                    )
                     if isinstance(self._approval_permissions, dict):
                         _mgr.load_rules_from_config(self._approval_permissions)
                     elif isinstance(self._approval_permissions, (list, tuple)):
@@ -3052,6 +3066,12 @@ Your Goal: {self.goal}
         # Agent-centric feature instances (lazy loaded for zero performance impact)
         self._auto_memory = auto_memory
         self._policy = policy
+        # Route enabled GUARDRAIL/POLICY/SKILL plugins into their typed
+        # subsystems now that guardrail (_setup_guardrail), _skills and _policy
+        # are all set. A plugin declared PluginType.GUARDRAIL/POLICY/SKILL thus
+        # participates in the GuardrailChain / PolicyEngine / SkillManager, not
+        # just a generic lifecycle hook. No-op when no such plugin is enabled.
+        self._merge_plugin_subsystems()
         self._output_style = output_style
         # Backward-compatible: `thinking_budget` property mirrors the legacy int
         # budget when supplied via the alias (Issue #4452); the unified effort is
@@ -3232,7 +3252,7 @@ Your Goal: {self.goal}
         for k, v in self.__dict__.items():
             if k in ("_Agent__cache_lock",):
                 object.__setattr__(result, k, threading.RLock())
-            elif k == "_cost_lock":
+            elif k in ("_cost_lock", "_auto_save_lock"):
                 object.__setattr__(result, k, threading.Lock())
             elif k == "_approvals_lock":
                 object.__setattr__(result, k, asyncio.Lock())
@@ -4156,12 +4176,44 @@ Summary:"""
                     except Exception:
                         pass
 
+        # Gate every code-mode tool call through the SAME PolicyEngine,
+        # permission deny-rules and per-tool ``input_guardrails`` the direct
+        # tool-call loop runs. The approval framework the proxy already honours
+        # does not cover these, so without this hook a model could reach an
+        # allow-listed tool from code and bypass a guardrail that blocks the
+        # same call on the normal path.
+        #
+        # Contract (mirrors the direct path in ``_check_tool_approval_sync``):
+        #   returns ``(denial_reason, arguments)`` where ``denial_reason`` is a
+        #   string to block or ``None`` to allow, and ``arguments`` is the
+        #   possibly-guardrail-rewritten kwargs dict the tool must run with. The
+        #   proxy uses the returned args so a sanitising guardrail's rewrite is
+        #   honoured (not discarded) exactly as on the direct path. BYPASS mode
+        #   skips the permission-deny gate, matching the direct contract, so a
+        #   call permitted directly is not rejected only in code mode. Only runs
+        #   in the tool-capable modes.
+        policy_hook = None
+        if code_tools and code_execution_mode in _tool_capable:
+            def policy_hook(name, arguments):
+                if not self._is_bypass_mode():
+                    deny = self._check_permission_manager_deny(name, arguments)
+                    if isinstance(deny, dict):
+                        return deny.get("error", "denied by permission policy"), arguments
+                verdict = self._check_tool_policy_and_guardrails(name, arguments)
+                if isinstance(verdict, dict):
+                    return verdict.get("error", "denied by policy"), arguments
+                # ``verdict`` is ``(None, rewritten_args)`` when allowed; hand the
+                # possibly-rewritten args back so the proxy dispatches with them.
+                _, rewritten = verdict
+                return None, rewritten
+
         # An unknown code_mode raises out of here rather than silently
         # producing nothing, which is the failure this whole change is about.
         new_tools = build_code_execution_tools(
             code_mode=code_execution_mode,
             allowed_tools=allowed if code_tools else [],
             registry=scoped_registry,
+            policy_hook=policy_hook,
         )
 
         existing = {getattr(t, "__name__", None) for t in (self.tools or [])}
@@ -5339,8 +5391,11 @@ Summary:"""
                         f"response_len={len(response_str)}"
                     )
                 
-                # Auto-save session after each async iteration (memory integration)
-                self._auto_save_session()
+                # Auto-save session after each async iteration (memory integration).
+                # Offloaded to a worker thread so the synchronous FileLock/JSON
+                # write never stalls the shared event loop (mirrors the goal-
+                # completion-judge offload below).
+                await asyncio.to_thread(self._auto_save_session)
 
                 # ─────────────────────────────────────────────────────────────
                 # GOAL COMPLETION JUDGE (opt-in): independent acceptance-criteria
@@ -5480,8 +5535,8 @@ Summary:"""
                 # Yield control to allow other async tasks to run
                 await asyncio.sleep(0)
             
-            # Final auto-save before returning
-            self._auto_save_session()
+            # Final auto-save before returning (offloaded off the event loop).
+            await asyncio.to_thread(self._auto_save_session)
             
             # Max iterations reached
             return AutonomyResult(
@@ -7068,6 +7123,25 @@ Answer:"""
         """The most recent rejection message a guardrail sent back to the model."""
         return getattr(self, "_last_guardrail_error", None)
 
+    def _guardrail_blocked_message(self, error=None):
+        """Fixed, user-facing string returned when an output guardrail blocks a run.
+
+        A blocked response must never surface the rejected (potentially unsafe)
+        model text, but it must also never be ``None`` - callers expect a
+        ``str`` and would otherwise crash on ``.strip()`` (issue #5596, FR-001).
+
+        The returned string is intentionally **fixed** and carries no diagnostic
+        detail: a guardrail's own rejection reason (especially an LLM validator's
+        free-text) can echo the blocked output, so appending it would leak the
+        very content the guardrail rejected. The reason is logged instead, where
+        it stays auditable without reaching the caller. The rejection is also
+        exposed programmatically via :attr:`last_guardrail_error`.
+        """
+        reason = str(error) if error else getattr(self, "_last_guardrail_error", None)
+        if reason:
+            logging.info(f"Agent {self.name}: guardrail block reason: {reason}")
+        return "I'm sorry, but I can't share that response because it did not pass the safety guardrail."
+
     def _process_guardrail(self, task_output):
         """Process the guardrail validation for a task output.
         
@@ -7083,8 +7157,16 @@ Answer:"""
             return GuardrailResult(success=True, result=task_output)
 
         try:
-            # Call the guardrail function
-            result = self._guardrail_fn(task_output)
+            # Call the guardrail function, awaiting it if the caller supplied an
+            # async def. Calling a coroutine function synchronously would only
+            # build a coroutine object (never running its body), which then
+            # crashes GuardrailResult.from_tuple and is misreported as a
+            # validation failure.
+            if asyncio.iscoroutinefunction(self._guardrail_fn):
+                from ..utils.async_bridge import run_coroutine_from_any_context
+                result = run_coroutine_from_any_context(self._guardrail_fn(task_output))
+            else:
+                result = self._guardrail_fn(task_output)
 
             # Convert the result to a GuardrailResult
             return GuardrailResult.from_tuple(result)

src/praisonai-agents/praisonaiagents/agent/async_safety.py

@@ -27,27 +27,21 @@ def run_async_in_sync_context(coro):
     tool loop, otherwise a bare un-awaited coroutine would be handed to the
     model as the tool result and the tool body would never run (silent data
     loss).
+
+    Delegates to the single canonical bridge
+    (:func:`utils.async_bridge.run_coroutine_from_any_context`) so all sync→async
+    call sites share one implementation: the caller's ``contextvars`` (trace /
+    session / approval context) are carried into the coroutine, and the running
+    loop is handled on a dedicated worker thread rather than raising.
+
+    ``timeout=None`` is passed deliberately: this helper backs the sync
+    tool-calling path, where a tool's own timeout policy (or lack of one) already
+    governs how long a tool may run. Imposing the bridge's default 5-minute
+    deadline here would silently cancel long-running tools that previously had no
+    limit — a backward-incompatible regression — so the bridge adds none.
     """
-    try:
-        asyncio.get_running_loop()
-    except RuntimeError:
-        # No event loop - safe to use asyncio.run()
-        return asyncio.run(coro)
-
-    # Event loop exists - avoid deadlock by running in dedicated thread
-    import concurrent.futures
-
-    def run_in_thread():
-        new_loop = asyncio.new_event_loop()
-        asyncio.set_event_loop(new_loop)
-        try:
-            return new_loop.run_until_complete(coro)
-        finally:
-            new_loop.close()
-
-    with concurrent.futures.ThreadPoolExecutor() as executor:
-        future = executor.submit(run_in_thread)
-        return future.result(timeout=300)  # 5 minute timeout
+    from ..utils.async_bridge import run_coroutine_from_any_context
+    return run_coroutine_from_any_context(coro, timeout=None)
 
 
 class DualLock:
@@ -75,9 +69,24 @@ class DualLock:
     """
     
     def __init__(self):
-        """Initialize with separate threading and asyncio locks."""
-        self._thread_lock = threading.RLock()  # Re-entrant lock to handle nested acquisitions
-        self._async_locks = WeakKeyDictionary()  # Per-event-loop async locks
+        """Initialize with a single underlying mutex shared by both call styles.
+
+        A single ``threading.Lock`` backs both ``sync()`` and ``async_lock()``
+        so that holding one genuinely blocks the other — mutual exclusion is
+        real across sync and async callers of the same instance. A plain
+        ``Lock`` (not ``RLock``) is used because the async path acquires on a
+        worker thread but releases on the event-loop thread, which ``RLock``'s
+        thread-ownership semantics forbid. Same-thread ``sync()`` reentrancy is
+        preserved explicitly via an owner/depth guard below. The per-event-loop
+        ``asyncio.Lock`` only serialises coroutines on the same loop so they
+        queue for the underlying mutex cooperatively instead of every coroutine
+        racing straight into the blocking acquire.
+        """
+        self._thread_lock = threading.Lock()  # Shared mutex (cross-thread release ok)
+        self._owner_lock = threading.Lock()    # Guards the reentrancy bookkeeping
+        self._owner_thread: Any = None         # Thread id currently holding for sync reentrancy
+        self._owner_depth = 0
+        self._async_locks = WeakKeyDictionary()  # Per-event-loop async gate
 
     def __deepcopy__(self, memo):
         """Return an unlocked primitive with no event-loop state sharing."""
@@ -87,9 +96,31 @@ class DualLock:
     
     @contextmanager
     def sync(self):
-        """Acquire lock in synchronous context using threading.Lock."""
-        with self._thread_lock:
+        """Acquire lock in synchronous context using the shared mutex.
+
+        Re-entrant for the same thread: a nested ``sync()`` on the thread that
+        already holds the mutex does not re-acquire (and cannot self-deadlock).
+        """
+        me = threading.get_ident()
+        with self._owner_lock:
+            reentrant = self._owner_thread == me and self._owner_depth > 0
+            if reentrant:
+                self._owner_depth += 1
+        if not reentrant:
+            self._thread_lock.acquire()
+            with self._owner_lock:
+                self._owner_thread = me
+                self._owner_depth = 1
+        try:
             yield
+        finally:
+            with self._owner_lock:
+                self._owner_depth -= 1
+                released = self._owner_depth == 0
+                if released:
+                    self._owner_thread = None
+            if released:
+                self._thread_lock.release()
             
     def _get_async_lock(self):
         """Get or create an asyncio.Lock for the current event loop."""
@@ -102,16 +133,64 @@ class DualLock:
             self._async_locks[loop] = asyncio.Lock()
         return self._async_locks[loop]
 
+    async def _acquire_shared_mutex(self, loop):
+        """Acquire the shared mutex off-loop, releasing it if we are cancelled.
+
+        ``run_in_executor`` can complete (mutex acquired) after the awaiting
+        task is cancelled. If we let the ``CancelledError`` propagate without
+        checking, the worker would strand the mutex forever. We therefore wait
+        on the acquisition, and on cancellation ensure the mutex is released
+        once the worker finishes so no caller waits indefinitely.
+        """
+        future = loop.run_in_executor(None, self._thread_lock.acquire)
+        try:
+            await asyncio.shield(future)
+        except asyncio.CancelledError:
+            # The worker may still be mid-acquire; wait for it to settle, then
+            # release if it managed to take the mutex so it is never orphaned.
+            try:
+                acquired = await future
+            except asyncio.CancelledError:
+                acquired = False
+            if acquired:
+                self._thread_lock.release()
+            raise
+
     @asynccontextmanager
     async def async_lock(self):
-        """Acquire lock in asynchronous context using asyncio.Lock.
-        
-        Uses a per-event-loop asyncio.Lock to ensure proper async coordination
-        without blocking the event loop or violating thread ownership semantics.
+        """Acquire lock in asynchronous context using the shared mutex.
+
+        Backs onto the *same* ``threading.Lock`` as ``sync()`` so a concurrent
+        sync caller is genuinely excluded (and vice-versa). The blocking
+        ``acquire`` is offloaded to a worker thread so the event loop is never
+        blocked while waiting; the matching ``release`` (allowed cross-thread on
+        a plain ``Lock``) happens on the loop thread. A per-event-loop
+        ``asyncio.Lock`` serialises coroutines on this loop first so they don't
+        all spin up executor hops at once.
+
+        While the mutex is held, the *event-loop thread* is registered as the
+        sync owner so a nested ``with lock.sync()`` on that thread (used by the
+        async handoff seeding path) is treated as re-entrant instead of
+        self-deadlocking against the mutex we already hold.
         """
-        async_lock = self._get_async_lock()
-        async with async_lock:
-            yield
+        loop = asyncio.get_running_loop()
+        async_gate = self._get_async_lock()
+        async with async_gate:
+            await self._acquire_shared_mutex(loop)
+            me = threading.get_ident()
+            with self._owner_lock:
+                self._owner_thread = me
+                self._owner_depth = 1
+            try:
+                yield
+            finally:
+                with self._owner_lock:
+                    self._owner_depth -= 1
+                    released = self._owner_depth == 0
+                    if released:
+                        self._owner_thread = None
+                if released:
+                    self._thread_lock.release()
             
     def is_async_context(self) -> bool:
         """Check if we're currently in an async context."""
@@ -185,6 +264,11 @@ class AsyncSafeState:
     @asynccontextmanager
     async def async_lock(self):
         """Acquire lock in async context."""
+        # Fail fast BEFORE blocking on the shared mutex: an in-progress
+        # synchronous deepcopy holds the mutex for the whole copy, so waiting on
+        # it here would block until the copy finished instead of surfacing the
+        # documented RuntimeError (and could stall the loop indefinitely).
+        self._raise_if_copying()
         async with self._lock.async_lock():
             self._begin_async_access()
             try:
@@ -192,6 +276,14 @@ class AsyncSafeState:
             finally:
                 self._end_async_access()
 
+    def _raise_if_copying(self) -> None:
+        """Reject entry while a synchronous deepcopy of this state is active."""
+        with self._activity_lock:
+            if self._copying:
+                raise RuntimeError(
+                    "Cannot access AsyncSafeState during synchronous deepcopy"
+                )
+
     def _begin_async_access(self) -> None:
         """Register async access unless a synchronous copy is in progress."""
         with self._activity_lock:
@@ -208,20 +300,32 @@ class AsyncSafeState:
             
     def __enter__(self):
         """Support for synchronous context manager protocol (backward compatibility)."""
-        self._lock._thread_lock.acquire()
+        self._sync_guard = self._lock.sync()
+        self._sync_guard.__enter__()
         return self.value
         
     def __exit__(self, exc_type, exc_val, exc_tb):
         """Support for synchronous context manager protocol (backward compatibility)."""
-        self._lock._thread_lock.release()
-        return None
+        return self._sync_guard.__exit__(exc_type, exc_val, exc_tb)
         
     async def __aenter__(self):
         """Support for asynchronous context manager protocol."""
+        # Fail fast before blocking on the mutex an in-progress deepcopy holds.
+        self._raise_if_copying()
+        loop = asyncio.get_running_loop()
         async_lock = self._lock._get_async_lock()
         await async_lock.acquire()
         try:
-            self._begin_async_access()
+            # Hold the SAME underlying mutex used by the sync path so async and
+            # sync accessors are mutually exclusive (matches DualLock.async_lock).
+            # Cancellation-safe acquire so the mutex is never stranded if the
+            # awaiting task is cancelled mid-acquisition.
+            await self._lock._acquire_shared_mutex(loop)
+            try:
+                self._begin_async_access()
+            except Exception:
+                self._lock._thread_lock.release()
+                raise
         except Exception:
             async_lock.release()
             raise
@@ -233,7 +337,10 @@ class AsyncSafeState:
         try:
             self._end_async_access()
         finally:
-            async_lock.release()
+            try:
+                self._lock._thread_lock.release()
+            finally:
+                async_lock.release()
         return None
             
     def get(self) -> Any:

src/praisonai-agents/praisonaiagents/agent/chat_mixin.py

@@ -751,6 +751,43 @@ Your Goal: {self.goal}"""
         
         return None
 
+    def _coerce_structured_output(self, response_text, schema_model):
+        """Parse a string response into the requested Pydantic model.
+
+        When ``output_pydantic``/``output_json`` is a Pydantic model, callers
+        expect a validated instance rather than a raw string. This tolerates
+        markdown code fences around the JSON payload.
+
+        Returns the parsed model instance on success, or the original
+        ``response_text`` unchanged if parsing is not possible (preserves
+        backward compatibility for non-Pydantic schemas or malformed output).
+        """
+        if not schema_model or not isinstance(response_text, str):
+            return response_text
+        if not hasattr(schema_model, 'model_validate_json'):
+            return response_text
+
+        candidate = response_text.strip()
+        try:
+            return schema_model.model_validate_json(candidate)
+        except Exception:
+            pass
+
+        # Strip a surrounding markdown code fence (```json ... ```) if present
+        fenced = re.search(r"```(?:json)?\s*(.*?)\s*```", candidate, re.DOTALL)
+        if fenced:
+            try:
+                return schema_model.model_validate_json(fenced.group(1).strip())
+            except Exception:
+                pass
+
+        logging.warning(
+            f"Agent {self.name}: output_pydantic={getattr(schema_model, '__name__', schema_model)} "
+            f"was requested but the response could not be parsed into the model; "
+            f"returning raw string."
+        )
+        return response_text
+
     def _supports_native_structured_output(self):
         """Check if current model supports native structured output via response_format.
         
@@ -3562,6 +3599,13 @@ Your Goal: {self.goal}"""
                     # Apply guardrail validation for custom LLM response
                     try:
                         validated_response = self._apply_guardrail_with_retry(response_text, prompt, temperature, tools, task_name, task_description, task_id, cancel_token=cancel_token, messages=processed_history)
+                        # Coerce into the requested Pydantic model so callers
+                        # get a validated instance, not a raw JSON string.
+                        # Only output_pydantic changes the return type; output_json
+                        # must stay a string for backward-compatible downstream
+                        # JSON parsing (task parsing, workflow results).
+                        if output_pydantic:
+                            validated_response = self._coerce_structured_output(validated_response, output_pydantic)
                         # Execute callback and display after validation
                         self._execute_callback_and_display(prompt, validated_response, time.time() - start_time, task_name, task_description, task_id)
                         return self._trigger_after_agent_hook(prompt, validated_response, start_time)
@@ -3569,7 +3613,7 @@ Your Goal: {self.goal}"""
                         logging.error(f"Agent {self.name}: Guardrail validation failed for custom LLM: {e}")
                         # Rollback chat history on guardrail failure
                         self._rollback_chat_history_to(chat_history_length)
-                        return None
+                        return self._guardrail_blocked_message(e)
                 except ToolExecutionError:
                     raise
                 except Exception as e:
@@ -3606,41 +3650,48 @@ Your Goal: {self.goal}"""
             )
             
 
+            # Track messages THIS turn appends so a failure rolls back only our
+            # own messages, never a concurrent turn's (see memory_mixin). The
+            # default (OpenAI) path used to skip this, so its positional rollback
+            # ``del chat_history[n:]`` erased a concurrent turn's interleaved
+            # messages and corrupted the shared transcript.
+            _turn_token = self._begin_turn_tracking()
+
             # Store chat history length for potential rollback
             chat_history_length = len(self.chat_history)
-            
-            # Normalize original_prompt for consistent chat history storage
-            normalized_content = original_prompt
-            if isinstance(original_prompt, list):
-                # Extract text from multimodal prompts
-                normalized_content = next((item["text"] for item in original_prompt if item.get("type") == "text"), "")
-            
-            # Add user message to chat history BEFORE LLM call so handoffs can
-            # access it. Use atomic check-then-act to prevent TOCTOU races: an
-            # unatomic read of chat_history[-1] followed by a separate append
-            # lets a concurrent turn's user message be mistaken for this turn's
-            # own duplicate, silently dropping this user turn from the record.
-            if self._add_to_chat_history_if_not_duplicate("user", normalized_content):
-                # Persist user message to DB (OpenAI path)
-                self._persist_message("user", normalized_content)
 
-            reflection_count = 0
-            start_time = time.time()
-            
-            # Apply context management before LLM call (auto-compaction)
-            # Zero overhead when context=False
-            system_prompt_content = messages[0].get("content", "") if messages and messages[0].get("role") == "system" else ""
-            processed_messages, context_result = self._apply_context_management(
-                messages=messages,
-                system_prompt=system_prompt_content,
-                tools=tools,
-            )
-            # Use processed messages for the LLM call
-            messages = processed_messages
-            
-            
-            # Wrap entire while loop in try-except for rollback on any failure
+            # Wrap entire turn in try/finally so per-turn tracking is always
+            # ended (the finally reads the ownership list during rollback).
             try:
+                # Normalize original_prompt for consistent chat history storage
+                normalized_content = original_prompt
+                if isinstance(original_prompt, list):
+                    # Extract text from multimodal prompts
+                    normalized_content = next((item["text"] for item in original_prompt if item.get("type") == "text"), "")
+
+                # Add user message to chat history BEFORE LLM call so handoffs can
+                # access it. Use atomic check-then-act to prevent TOCTOU races: an
+                # unatomic read of chat_history[-1] followed by a separate append
+                # lets a concurrent turn's user message be mistaken for this turn's
+                # own duplicate, silently dropping this user turn from the record.
+                if self._add_to_chat_history_if_not_duplicate("user", normalized_content):
+                    # Persist user message to DB (OpenAI path)
+                    self._persist_message("user", normalized_content)
+
+                reflection_count = 0
+                start_time = time.time()
+
+                # Apply context management before LLM call (auto-compaction)
+                # Zero overhead when context=False
+                system_prompt_content = messages[0].get("content", "") if messages and messages[0].get("role") == "system" else ""
+                processed_messages, context_result = self._apply_context_management(
+                    messages=messages,
+                    system_prompt=system_prompt_content,
+                    tools=tools,
+                )
+                # Use processed messages for the LLM call
+                messages = processed_messages
+
                 while True:
                     try:
                         if self.verbose:
@@ -3686,6 +3737,11 @@ Your Goal: {self.goal}"""
                             # Apply guardrail validation even for JSON output
                             try:
                                 validated_response = self._apply_guardrail_with_retry(response_text, original_prompt, temperature, tools, task_name, task_description, task_id, cancel_token=cancel_token, messages=messages)
+                                # Coerce into the requested Pydantic model so callers
+                                # get a validated instance, not a raw JSON string.
+                                # output_json intentionally stays a string (backward compat).
+                                if output_pydantic:
+                                    validated_response = self._coerce_structured_output(validated_response, output_pydantic)
                                 # Execute callback after validation
                                 self._execute_callback_and_display(original_prompt, validated_response, time.time() - start_time, task_name, task_description, task_id)
                                 return self._trigger_after_agent_hook(original_prompt, validated_response, start_time)
@@ -3693,7 +3749,7 @@ Your Goal: {self.goal}"""
                                 logging.error(f"Agent {self.name}: Guardrail validation failed for JSON output: {e}")
                                 # Rollback chat history on guardrail failure
                                 self._rollback_chat_history_to(chat_history_length)
-                                return None
+                                return self._guardrail_blocked_message(e)
 
                         if not self.self_reflect:
                             # User message already added before LLM call via _build_messages
@@ -3714,7 +3770,7 @@ Your Goal: {self.goal}"""
                                     logging.error(f"Agent {self.name}: Guardrail validation failed for reasoning content: {e}")
                                     # Rollback chat history on guardrail failure
                                     self._rollback_chat_history_to(chat_history_length)
-                                    return None
+                                    return self._guardrail_blocked_message(e)
                             # Apply guardrail to regular response
                             try:
                                 validated_response = self._apply_guardrail_with_retry(response_text, original_prompt, temperature, tools, task_name, task_description, task_id, cancel_token=cancel_token, messages=messages)
@@ -3725,7 +3781,7 @@ Your Goal: {self.goal}"""
                                 logging.error(f"Agent {self.name}: Guardrail validation failed: {e}")
                                 # Rollback chat history on guardrail failure
                                 self._rollback_chat_history_to(chat_history_length)
-                                return None
+                                return self._guardrail_blocked_message(e)
 
                         reflection_prompt = f"""
 Reflect on your previous response: '{response_text}'.
@@ -3793,7 +3849,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                     # Rollback chat history on guardrail failure
                                     self._rollback_chat_history_to(chat_history_length)
                                     self._end_run(None, "error", {"error": str(e)})
-                                    return None
+                                    return self._guardrail_blocked_message(e)
 
                             # Check if we've hit max reflections
                             if reflection_count >= self.max_reflect - 1:
@@ -3811,7 +3867,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                     logging.error(f"Agent {self.name}: Guardrail validation failed after max reflections: {e}")
                                     # Rollback chat history on guardrail failure
                                     self._rollback_chat_history_to(chat_history_length)
-                                    return None
+                                    return self._guardrail_blocked_message(e)
                             
                             # If not satisfactory and not at max reflections, continue with regeneration
                             logging.debug(f"{self.name} reflection count {reflection_count + 1}, continuing reflection process")
@@ -3858,6 +3914,10 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                 # Rollback chat history
                 self._rollback_chat_history_to(chat_history_length)
                 return None
+            finally:
+                # End per-turn tracking after any rollback has run (rollback
+                # reads the ownership list, so this must come last).
+                self._end_turn_tracking(_turn_token)
 
     def clean_json_output(self, output: str) -> str:
         """Clean JSON output while preserving the legacy agent method."""
@@ -4217,6 +4277,11 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                     # Apply guardrail validation for custom LLM response
                     try:
                         validated_response = await self._aapply_guardrail_with_retry(response_text, prompt, temperature, tools, task_name, task_description, task_id, messages=effective_history)
+                        # Coerce into the requested Pydantic model so callers
+                        # get a validated instance, not a raw JSON string.
+                        # output_json intentionally stays a string (backward compat).
+                        if output_pydantic:
+                            validated_response = self._coerce_structured_output(validated_response, output_pydantic)
                         # Execute callback after validation
                         self._execute_callback_and_display(normalized_content, validated_response, time.time() - start_time, task_name, task_description, task_id)
                         return await self._atrigger_after_agent_hook(prompt, validated_response, start_time)
@@ -4224,7 +4289,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                         logging.error(f"Agent {self.name}: Guardrail validation failed for custom LLM: {e}")
                         # Rollback chat history on guardrail failure
                         self._rollback_chat_history_to(chat_history_length)
-                        return None
+                        return self._guardrail_blocked_message(e)
                 except ToolExecutionError:
                     raise
                 except Exception as e:
@@ -4250,6 +4315,15 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
             if durable_context is not None:
                 messages = await durable_context.arestore_messages(messages)
             
+            # Track messages THIS turn appends so a failure rolls back only our
+            # own messages, never a concurrent turn's (see memory_mixin). The
+            # default (OpenAI) path used to skip this, so its positional rollback
+            # ``del chat_history[n:]`` erased a concurrent turn's interleaved
+            # messages. Each achat() turn runs in its own task/context and
+            # _clear_turn_tracking() ran at the top of this turn, so this list is
+            # cleaned up when the next turn on this task starts.
+            self._begin_turn_tracking()
+
             # Store chat history length for potential rollback
             chat_history_length = len(self.chat_history)
             
@@ -4353,6 +4427,11 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                             # Apply guardrail validation even for JSON output
                             try:
                                 validated_response = await self._aapply_guardrail_with_retry(response_text, original_prompt, temperature, tools, task_name, task_description, task_id, messages=messages)
+                                # Coerce into the requested Pydantic model so callers
+                                # get a validated instance, not a raw JSON string.
+                                # output_json intentionally stays a string (backward compat).
+                                if output_pydantic:
+                                    validated_response = self._coerce_structured_output(validated_response, output_pydantic)
                                 # Execute callback after validation
                                 self._execute_callback_and_display(original_prompt, validated_response, time.time() - start_time, task_name, task_description, task_id)
                                 return await self._atrigger_after_agent_hook(original_prompt, validated_response, start_time)
@@ -4360,7 +4439,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                 logging.error(f"Agent {self.name}: Guardrail validation failed for JSON output: {e}")
                                 # Rollback chat history on guardrail failure
                                 self._rollback_chat_history_to(chat_history_length)
-                                return None
+                                return self._guardrail_blocked_message(e)
 
                         # For regular responses (no self-reflection)
                         if not self.self_reflect:
@@ -4383,7 +4462,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                     logging.error(f"Agent {self.name}: Guardrail validation failed for reasoning content: {e}")
                                     # Rollback chat history on guardrail failure
                                     self._rollback_chat_history_to(chat_history_length)
-                                    return None
+                                    return self._guardrail_blocked_message(e)
                             else:
                                 # Apply guardrail to regular response content
                                 try:
@@ -4395,7 +4474,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                     logging.error(f"Agent {self.name}: Guardrail validation failed: {e}")
                                     # Rollback chat history on guardrail failure
                                     self._rollback_chat_history_to(chat_history_length)
-                                    return None
+                                    return self._guardrail_blocked_message(e)
                         
                         # If self-reflection is enabled, implement reflection logic
                         if self.self_reflect:
@@ -4435,7 +4514,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                             logging.error(f"Agent {self.name}: Guardrail validation failed: {e}")
                                             # Rollback chat history on guardrail failure
                                             self._rollback_chat_history_to(chat_history_length)
-                                            return None
+                                            return self._guardrail_blocked_message(e)
                                     
                                     _areflection_kwargs = {}
                                     if temperature is not None:
@@ -4469,7 +4548,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                             logging.error(f"Agent {self.name}: Guardrail validation failed after reflection: {e}")
                                             # Rollback chat history on guardrail failure
                                             self._rollback_chat_history_to(chat_history_length)
-                                            return None
+                                            return self._guardrail_blocked_message(e)
                                     
                                     # Check if we've hit max reflections
                                     if reflection_count >= self.max_reflect - 1:
@@ -4488,7 +4567,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                             logging.error(f"Agent {self.name}: Guardrail validation failed after max reflections: {e}")
                                             # Rollback chat history on guardrail failure
                                             self._rollback_chat_history_to(chat_history_length)
-                                            return None
+                                            return self._guardrail_blocked_message(e)
                                     
                                     # Regenerate response based on reflection
                                     regenerate_messages = reflection_messages + [
@@ -4538,7 +4617,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                                             logging.error(f"Agent {self.name}: Guardrail validation failed after reflection error: {guard_e}")
                                             # Rollback chat history on guardrail failure
                                             self._rollback_chat_history_to(chat_history_length)
-                                            return None
+                                            return self._guardrail_blocked_message(guard_e)
                                     continue
                         
                         # This should never be reached due to the returns above
@@ -4588,6 +4667,11 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                             if get_logger().getEffectiveLevel() == logging.DEBUG:
                                 total_time = time.time() - start_time
                                 logging.debug(f"Agent.achat completed in {total_time:.2f} seconds")
+                            # Coerce into the requested Pydantic model so callers
+                            # get a validated instance, not a raw JSON string.
+                            # output_json intentionally stays a string (backward compat).
+                            if output_pydantic:
+                                response_text = self._coerce_structured_output(response_text, output_pydantic)
                             # Execute callback after JSON/Pydantic completion
                             self._execute_callback_and_display(original_prompt, response_text, time.time() - start_time, task_name, task_description, task_id)
                             return await self._atrigger_after_agent_hook(original_prompt, response_text, start_time)
@@ -4702,7 +4786,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
                             logging.error(f"Agent {self.name}: Guardrail validation failed for OpenAI client: {e}")
                             # Rollback chat history on guardrail failure
                             self._rollback_chat_history_to(chat_history_length)
-                            return None
+                            return self._guardrail_blocked_message(e)
                 except ToolExecutionError:
                     raise
                 except Exception as e:
@@ -4726,6 +4810,16 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
             # dangling, unanswered user message (mirrors the sync path).
             self._rollback_chat_history_to(chat_history_length)
             return None
+        finally:
+            # Unregister this turn's ownership list from the process-wide
+            # live-turn registry once the turn finishes (success or failure).
+            # The sync path does this in _end_turn_tracking's finally; the async
+            # path previously relied only on the *next* turn's
+            # _clear_turn_tracking(), so a terminal achat/arun/astart left its
+            # completed ownership globally registered and a later ephemeral()
+            # cleanup treated its committed messages as still live. Rollback in
+            # the except blocks above already read ownership before this runs.
+            self._clear_turn_tracking()
 
     async def _achat_completion(self, response, tools, reasoning_steps=False):
         """Async version of _chat_completion method"""
@@ -5698,6 +5792,28 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
         
         return llm_summarize_async
 
+    def _resolve_compaction_max_tokens(self, execution_cfg):
+        """
+        Resolve the compaction trigger budget.
+
+        An explicit ``ExecutionConfig.max_context_tokens`` always wins. When it
+        is unset, size the trigger to the active model's context window (reusing
+        the model-aware ``budgeter`` the in-loop path already uses) instead of a
+        flat default, so large-context models are not compacted prematurely.
+        Falls back to a static default only for the offline / unknown-model case.
+        """
+        _explicit = getattr(execution_cfg, 'max_context_tokens', None)
+        if _explicit:
+            return _explicit
+        try:
+            from ..context.budgeter import get_model_limit, get_output_reserve
+            model_name = self.llm if isinstance(self.llm, str) else "gpt-4o-mini"
+            limit = get_model_limit(model_name)
+            reserve = get_output_reserve(model_name)
+            return max(1, int(limit * 0.8) - reserve)
+        except Exception:
+            return 8000
+
     def _apply_context_compaction(self, messages, hook_event_class):
         """
         Apply context compaction to messages if enabled (sync version).
@@ -5717,7 +5833,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
             from ..compaction import ContextCompactor
             from ..compaction.strategy import CompactionStrategy
             
-            _max_tok = getattr(_execution_cfg, 'max_context_tokens', None) or 8000
+            _max_tok = self._resolve_compaction_max_tokens(_execution_cfg)
             _strategy = getattr(_execution_cfg, 'compaction_strategy', None) or CompactionStrategy.TRUNCATE
             
             # Create LLM summarization function if strategy is LLM_SUMMARIZE
@@ -5749,17 +5865,20 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
             
             # Perform compaction
             if _strategy == CompactionStrategy.LLM_SUMMARIZE and _llm_fn:
-                import asyncio
-                try:
-                    # Run async compaction in event loop
-                    compacted_msgs, _cr = asyncio.run(_compactor.compact_async(messages))
-                except RuntimeError:
-                    # If already in async context, fall back to sync (naive) compaction
-                    logging.warning(
-                        f"[compaction] {self.name}: LLM_SUMMARIZE fell back to naive summarization "
-                        f"(asyncio.run not available in sync context)"
-                    )
-                    compacted_msgs, _cr = _compactor.compact(messages)
+                # Drive the async summariser through the shared bridge so it runs
+                # correctly whether or not a loop is already running (FastAPI /
+                # Jupyter / a bot handler). The previous code caught bare
+                # RuntimeError from ``asyncio.run`` ("event loop already
+                # running") and silently downgraded LLM summarisation to naive
+                # truncation whenever a loop was live; the bridge removes that
+                # spurious fallback so the real summariser always runs. Provider
+                # errors inside the summariser remain best-effort (see
+                # ``_llm_summarize_async``, which falls back to a naive summary on
+                # failure) — compaction never hard-fails the turn.
+                from ..utils.async_bridge import run_coroutine_from_any_context
+                compacted_msgs, _cr = run_coroutine_from_any_context(
+                    _compactor.compact_async(messages)
+                )
             else:
                 compacted_msgs, _cr = _compactor.compact(messages)
             
@@ -5804,7 +5923,7 @@ Output MUST be JSON with 'reflection' and 'satisfactory'.
             from ..compaction import ContextCompactor
             from ..compaction.strategy import CompactionStrategy
             
-            _max_tok = getattr(_execution_cfg, 'max_context_tokens', None) or 8000
+            _max_tok = self._resolve_compaction_max_tokens(_execution_cfg)
             _strategy = getattr(_execution_cfg, 'compaction_strategy', None) or CompactionStrategy.TRUNCATE
             
             # Create LLM summarization function if strategy is LLM_SUMMARIZE

src/praisonai-agents/praisonaiagents/agent/concurrency.py

@@ -31,23 +31,66 @@ from praisonaiagents._logging import get_logger
 logger = get_logger(__name__)
 
 
+class _DynamicLimiter:
+    """Resizable counting limiter: capacity can change with holders outstanding.
+
+    Unlike swapping a fresh threading.Semaphore on every retune (which lets
+    holders of the old primitive and the release() path operate on different
+    objects), this keeps one long-lived object per agent name and mutates its
+    capacity in place. Every acquirer and every release() therefore always
+    operate on the same shared state, so a retune takes effect immediately and
+    the configured cap can never be silently exceeded. Loop-neutral: uses a
+    threading.Condition so it is a true global cap across event loops/threads.
+    """
+
+    def __init__(self, limit: int):
+        self._limit = limit
+        self._held = 0
+        self._cond = threading.Condition()
+
+    def set_limit(self, limit: int) -> None:
+        with self._cond:
+            self._limit = limit
+            self._cond.notify_all()
+
+    def acquire(self, blocking: bool = True) -> bool:
+        with self._cond:
+            while self._limit > 0 and self._held >= self._limit:
+                if not blocking:
+                    return False
+                self._cond.wait(timeout=0.05)
+            self._held += 1
+            return True
+
+    def release(self) -> None:
+        with self._cond:
+            self._held = max(0, self._held - 1)
+            self._cond.notify_all()
+
+    def has_holders(self) -> bool:
+        with self._cond:
+            return self._held > 0
+
+
 class ConcurrencyRegistry:
     """Registry for per-agent concurrency limits.
 
-    Thread-safe. Each agent name maps to a loop-neutral threading.Semaphore, so
-    the limit is a true global cap across every event loop and thread.
+    Thread-safe. Each agent name maps to a loop-neutral, resizable limiter, so
+    the limit is a true global cap across every event loop and thread and a
+    retune takes effect immediately for in-flight holders.
     Limit of 0 means unlimited (no throttling).
     """
 
     def __init__(self, default_limit: int = 0):
         self._default_limit = default_limit
         self._limits: Dict[str, int] = {}
-        # One loop-neutral threading.Semaphore per agent. Unlike asyncio.Semaphore
-        # (which binds to the loop it is first awaited on), a threading.Semaphore
-        # is shared safely across every event loop and thread, so the per-agent
-        # limit is a true global cap under multi-loop / multi-thread deployments.
-        # Async callers acquire it in an executor so they never block their loop.
-        self._semaphores: Dict[str, threading.Semaphore] = {}
+        # One long-lived, loop-neutral _DynamicLimiter per agent. Its capacity is
+        # mutated in place on retune (never swapped for a fresh object), so every
+        # acquirer and every release() always share the same state and the
+        # configured cap can never be silently exceeded. Loop-neutral means the
+        # per-agent limit is a true global cap under multi-loop / multi-thread
+        # deployments; async callers poll without blocking their loop.
+        self._limiters: Dict[str, _DynamicLimiter] = {}
         self._lock = threading.Lock()
 
     def set_limit(self, agent_name: str, max_concurrent: int) -> None:
@@ -59,8 +102,11 @@ class ConcurrencyRegistry:
         """
         with self._lock:
             self._limits[agent_name] = max_concurrent
-            # Reset semaphore so next acquire creates a fresh one at the new limit
-            self._semaphores.pop(agent_name, None)
+            # Mutate the existing limiter in place so the retune takes effect
+            # immediately for in-flight holders without ever swapping the object.
+            limiter = self._limiters.get(agent_name)
+            if limiter is not None:
+                limiter.set_limit(max_concurrent)
 
     def get_limit(self, agent_name: str) -> int:
         """Get concurrency limit for an agent."""
@@ -68,39 +114,57 @@ class ConcurrencyRegistry:
             return self._limits.get(agent_name, self._default_limit)
 
     def remove_limit(self, agent_name: str) -> None:
-        """Remove concurrency limit for an agent (reverts to default)."""
+        """Remove concurrency limit for an agent (reverts to default).
+
+        The limiter object is kept alive while any permit is still held, so
+        in-flight acquire()/release() pairs keep operating on the same shared
+        state; dropping it mid-flight would strand those permits and let a
+        later set_limit() start from a fresh, empty count and admit work beyond
+        the configured cap. Once no permits are outstanding it is safe to drop.
+        """
         with self._lock:
             self._limits.pop(agent_name, None)
-            self._semaphores.pop(agent_name, None)
+            # Retune the surviving limiter to unlimited so it stops blocking
+            # while remaining the single accounting object for its holders.
+            limiter = self._limiters.get(agent_name)
+            if limiter is None:
+                return
+            limiter.set_limit(0)
+            if not limiter.has_holders():
+                self._limiters.pop(agent_name, None)
 
-    def _get_semaphore(self, agent_name: str) -> Optional[threading.Semaphore]:
-        """Get or create the loop-neutral semaphore for an agent.
+    def _get_limiter(self, agent_name: str) -> Optional[_DynamicLimiter]:
+        """Get or create the loop-neutral limiter for an agent.
 
-        Returns None if unlimited. The same threading.Semaphore is shared across
-        every event loop and thread, so the configured limit is a true global cap.
+        Returns None only when the agent has never been given a limit (pure
+        unlimited fast path). Once a limiter exists it is always returned — even
+        while unlimited (limit 0) — so acquire() and release() stay symmetric
+        across a mid-flight retune to/from unlimited and the count can never be
+        corrupted. The same _DynamicLimiter is shared across every event loop
+        and thread, so a configured limit is a true global cap.
         """
         with self._lock:
             limit = self._limits.get(agent_name, self._default_limit)
-            if limit <= 0:
-                return None
-            sem = self._semaphores.get(agent_name)
-            if sem is None:
-                sem = threading.Semaphore(limit)
-                self._semaphores[agent_name] = sem
-            return sem
+            limiter = self._limiters.get(agent_name)
+            if limiter is None:
+                if limit <= 0:
+                    return None
+                limiter = _DynamicLimiter(limit)
+                self._limiters[agent_name] = limiter
+            return limiter
 
     async def acquire(self, agent_name: str) -> None:
         """Acquire concurrency slot for agent. No-op if unlimited.
 
-        Waits on the loop-neutral semaphore in short, cancellable polls so the
+        Waits on the loop-neutral limiter in short, cancellable polls so the
         running event loop is never blocked while other tasks hold permits, and
         a cancelled/timed-out await never leaves a thread blocked on acquire().
         """
-        sem = self._get_semaphore(agent_name)
-        if sem is None:
+        limiter = self._get_limiter(agent_name)
+        if limiter is None:
             return
         while True:
-            if sem.acquire(blocking=False):
+            if limiter.acquire(blocking=False):
                 return
             # Yield to the loop; on cancellation this raises and no permit leaks.
             await asyncio.sleep(0.005)
@@ -110,21 +174,18 @@ class ConcurrencyRegistry:
 
         Prefer async acquire() when possible. Blocks the calling thread until a
         permit is available. Safe to call whether or not a loop is running in the
-        current thread, since the semaphore is loop-neutral.
+        current thread, since the limiter is loop-neutral.
         """
-        sem = self._get_semaphore(agent_name)
-        if sem is not None:
-            sem.acquire()
+        limiter = self._get_limiter(agent_name)
+        if limiter is not None:
+            limiter.acquire()
 
     def release(self, agent_name: str) -> None:
         """Release concurrency slot for agent. No-op if unlimited."""
         with self._lock:
-            sem = self._semaphores.get(agent_name)
-        if sem is not None:
-            try:
-                sem.release()
-            except ValueError:
-                pass  # Already fully released
+            limiter = self._limiters.get(agent_name)
+        if limiter is not None:
+            limiter.release()
 
     @asynccontextmanager
     async def throttle(self, agent_name: str):

src/praisonai-agents/praisonaiagents/agent/cost_persistence.py

@@ -58,8 +58,8 @@ def save_cost_report(
     }
 
     filepath = COST_DIR / f"{safe_name}.json"
-    with open(filepath, "w") as f:
-        json.dump(report, f, indent=2)
+    from ..utils.atomic_io import atomic_write_json
+    atomic_write_json(str(filepath), report, indent=2)
 
     logger.info(f"[costs] Saved cost report to {filepath}")
     return str(filepath)

src/praisonai-agents/praisonaiagents/agent/deep_research_agent.py

@@ -898,14 +898,18 @@ You are a professional research analyst. When conducting research:
         if grounding_file_prompt:
             create_params["grounding_file_prompt"] = grounding_file_prompt
         
-        interaction = self.gemini_client.interactions.create(**create_params)
+        interaction = await asyncio.to_thread(
+            self.gemini_client.interactions.create, **create_params
+        )
         
         if self.verbose:
             self.logger.debug(f"Gemini research started (async): {interaction.id}")
         
         start_time = time.time()
         while True:
-            interaction = self.gemini_client.interactions.get(interaction.id)
+            interaction = await asyncio.to_thread(
+                self.gemini_client.interactions.get, interaction.id
+            )
             
             if interaction.status == "completed":
                 if self.verbose:

src/praisonai-agents/praisonaiagents/agent/execution_mixin.py

@@ -200,8 +200,12 @@ class ExecutionMixin:
                 response = await self.achat(prompt, **kwargs)
                 response_str = str(response) if response else ""
                 
-                # Auto-save session after chat
-                self._auto_save_session()
+                # Auto-save session after chat. Offloaded to a worker thread so
+                # the synchronous FileLock/JSON read-modify-write never stalls
+                # the shared event loop (the session store is a process-wide
+                # singleton, so a blocking write here blocks every concurrent
+                # async agent task on the same loop).
+                await asyncio.to_thread(self._auto_save_session)
                 
                 # Wrap in AutonomyResult for consistent API
                 from .autonomy import AutonomyResult
@@ -472,7 +476,8 @@ class ExecutionMixin:
                             extra={"session_id": sid, "managed_session_id": msid},
                             exc_info=True,
                         )
-            self._auto_save_session()
+            # Offload the blocking session write off the event loop.
+            await asyncio.to_thread(self._auto_save_session)
 
         return result
 
@@ -1733,12 +1738,17 @@ Write the complete compiled report:"""
 
             # Policy/guardrail gate (protocol-driven). Mirrors the sync path in
             # _execute_tool_impl so async callers cannot bypass a PolicyEngine
-            # deny or a tool-call guardrail. The check is pure/sync (no awaits).
+            # deny or a tool-call guardrail. A user-supplied guardrail/policy may
+            # do blocking work (an LLM moderation call, a DB/network lookup), so
+            # offload to a worker thread rather than run it inline and stall the
+            # event loop for every other coroutine sharing it.
             check = getattr(self, "_check_tool_policy_and_guardrails", None)
             if check is not None:
                 # ``tools_override`` is threaded through so a run-scoped tool
                 # list cannot bypass the per-tool guardrails its tools declare.
-                policy_result = check(function_name, arguments, tools_override)
+                policy_result = await asyncio.to_thread(
+                    check, function_name, arguments, tools_override
+                )
                 if isinstance(policy_result, dict):
                     return policy_result  # Error dict
                 _, arguments = policy_result
@@ -1765,9 +1775,12 @@ Write the complete compiled report:"""
                         # Tool-result guardrail gate — the native path applies this
                         # below, but the MCP branch returns early, so gate here too
                         # or successful MCP output would bypass validation entirely.
+                        # Offloaded to a thread: a user guardrail may block on I/O.
                         apply_result_guardrails = getattr(self, "_apply_tool_result_guardrails", None)
                         if apply_result_guardrails is not None:
-                            mcp_result = apply_result_guardrails(function_name, mcp_result)
+                            mcp_result = await asyncio.to_thread(
+                                apply_result_guardrails, function_name, mcp_result
+                            )
                         return mcp_result
 
             # Try to find the function in the override tools list first, then agent's tools list.
@@ -2128,9 +2141,14 @@ Write the complete compiled report:"""
                 # sync path in tool_execution.py. Runs on the raw result before it
                 # re-enters the LLM context so a guardrail can inspect or redact
                 # unsafe tool output. Fail-closed. Zero overhead when unset.
+                # Offloaded to a thread: a user guardrail may block on I/O (an LLM
+                # moderation call, a network lookup), which would otherwise stall
+                # the event loop for every other coroutine sharing it.
                 apply_result_guardrails = getattr(self, "_apply_tool_result_guardrails", None)
                 if apply_result_guardrails is not None:
-                    result = apply_result_guardrails(function_name, result, tools_override)
+                    result = await asyncio.to_thread(
+                        apply_result_guardrails, function_name, result, tools_override
+                    )
 
                 # Loop guard (post-execution) — record the outcome and surface a
                 # block/halt decision back to the model on this same turn, mirroring

src/praisonai-agents/praisonaiagents/agent/handoff.py

@@ -569,17 +569,13 @@ class Handoff:
         handoffs to the same agent serialize on the event loop instead of
         blocking it, keeping each seeded ``achat`` turn atomic.
 
-        ``DualLock`` keeps its sync (threading ``RLock``) and async (per-loop
-        ``asyncio.Lock``) sides independent, so the async lock alone would NOT
-        exclude a concurrent *synchronous* handoff running on another thread —
-        both could seed/restore the same shared ``chat_history`` at once, and a
-        cross-thread sync seed landing *between* our apply and restore would
-        corrupt the awaited turn. To close that gap we ALSO hold the sync
-        thread-lock for the whole seeded turn. The async lock (acquired first)
-        has already serialized every same-loop async handoff to this target, so
-        the thread-lock here only ever contends with a genuine cross-thread sync
-        handoff to the same agent — exactly the case that must be mutually
-        excluded — which is a rare mixed sync+async fan-out to one shared target.
+        ``DualLock`` backs ``sync()`` and ``async_lock()`` onto one shared mutex,
+        so ``async_lock()`` alone already excludes a concurrent *synchronous*
+        handoff on another thread. We still take the inner ``sync()`` so the
+        seed/restore reads as one explicit critical section; because
+        ``async_lock()`` registers the event-loop thread as the mutex owner, that
+        nested ``sync()`` is treated as re-entrant on the same thread instead of
+        self-deadlocking against the mutex the async side already holds.
         """
         lock = _get_handoff_seed_lock(self.agent)
         async with lock.async_lock():

src/praisonai-agents/praisonaiagents/agent/memory_mixin.py

@@ -764,6 +764,50 @@ class MemoryMixin:
         except Exception as e:
             logging.debug(f"Auto-memory extraction failed: {e}")
 
+    def _persist_memory_turn(self, user_message: str, assistant_response: str):
+        """Persist a raw conversation turn to the memory store.
+
+        Called after each response when ``_memory_instance`` is set but
+        ``auto_memory`` extraction is off. Stores the turn to short-term memory
+        so a later Agent sharing the same store (e.g. ``memory={"user_id": uid}``)
+        recalls it via ``get_memory_context()``. Best-effort: never raises.
+
+        Async-safe: a memory store write is blocking file/DB I/O. When this runs
+        inside a live event loop (the async ``achat``/``astart`` after-agent path),
+        the write is offloaded to a worker thread so it never stalls the loop
+        (AGENTS.md §4.5). On the sync path (no running loop) it writes inline.
+        """
+        memory = getattr(self, "_memory_instance", None)
+        if memory is None or not assistant_response:
+            return
+        store = getattr(memory, "store_short_term", None) or getattr(memory, "add_short_term", None)
+        if store is None:
+            return
+        text = f"User: {user_message}\nAssistant: {assistant_response}"
+        metadata = {"agent_id": getattr(self, "agent_id", getattr(self, "name", None))}
+
+        def _do_store():
+            try:
+                store(text, metadata=metadata)
+            except Exception as e:
+                logging.debug(f"Memory turn persistence failed: {e}")
+
+        try:
+            import asyncio
+            asyncio.get_running_loop()
+        except RuntimeError:
+            # No running loop (sync chat/run/start path): write inline.
+            _do_store()
+            return
+        # Inside a live loop: offload the blocking write so it cannot stall the
+        # event loop. Fire-and-forget — persistence is best-effort.
+        try:
+            import asyncio
+            asyncio.get_running_loop().run_in_executor(None, _do_store)
+        except Exception as e:
+            logging.debug(f"Memory turn persistence scheduling failed: {e}")
+            _do_store()
+
     def _process_auto_learning(self):
         """Process auto-learning extraction after agent response.
         
@@ -884,45 +928,56 @@ class MemoryMixin:
         if not self.auto_save:
             return
         
-        try:
-            # Filter out history markers before saving
-            clean_history = [
-                {k: v for k, v in msg.items() if k != "_from_history"}
-                for msg in self.chat_history
-            ]
-            
-            # Issue 1 FIX: Only save NEW messages since last save
-            # Track last saved index to avoid duplicates
-            last_saved = getattr(self, '_auto_save_last_index', 0)
-            new_messages = clean_history[last_saved:]
-            
-            if not new_messages:
-                return  # Nothing new to save
-            
-            # G-1 FIX: Use SessionStore for conversation history persistence
-            # This maintains clean separation: Memory = facts, SessionStore = turns
-            if self._session_store is not None:
-                # Persist only NEW messages to SessionStore
-                for msg in new_messages:
-                    self._session_store.add_message(
-                        self.auto_save,  # Use auto_save name as session_id
-                        role=msg.get("role", "user"),
-                        content=msg.get("content", ""),
+        # Serialize the whole read-modify-write of _auto_save_last_index. This
+        # method now runs via asyncio.to_thread, so two concurrent async runs
+        # sharing one Agent can execute it in separate worker threads. Without
+        # this lock both could read the same last-saved index, then each persist
+        # the same new turns, duplicating conversation history — the store's
+        # file lock serializes the writes but does not protect this index.
+        lock = getattr(self, '_auto_save_lock', None)
+        if lock is None:
+            lock = threading.Lock()
+            self._auto_save_lock = lock
+        with lock:
+            try:
+                # Filter out history markers before saving
+                clean_history = [
+                    {k: v for k, v in msg.items() if k != "_from_history"}
+                    for msg in self.chat_history
+                ]
+                
+                # Issue 1 FIX: Only save NEW messages since last save
+                # Track last saved index to avoid duplicates
+                last_saved = getattr(self, '_auto_save_last_index', 0)
+                new_messages = clean_history[last_saved:]
+                
+                if not new_messages:
+                    return  # Nothing new to save
+                
+                # G-1 FIX: Use SessionStore for conversation history persistence
+                # This maintains clean separation: Memory = facts, SessionStore = turns
+                if self._session_store is not None:
+                    # Persist only NEW messages to SessionStore
+                    for msg in new_messages:
+                        self._session_store.add_message(
+                            self.auto_save,  # Use auto_save name as session_id
+                            role=msg.get("role", "user"),
+                            content=msg.get("content", ""),
+                        )
+                    # Update last saved index
+                    self._auto_save_last_index = len(clean_history)
+                    logging.debug(f"Auto-saved {len(new_messages)} new messages to SessionStore: {self.auto_save}")
+                elif self._memory_instance and hasattr(self._memory_instance, 'save_session'):
+                    # Fallback to Memory.save_session() for backward compatibility
+                    # Memory.save_session() replaces entire history, so no duplicate issue
+                    self._memory_instance.save_session(
+                        name=self.auto_save,
+                        conversation_history=clean_history,
+                        metadata={"agent_name": self.name, "user_id": self.user_id}
                     )
-                # Update last saved index
-                self._auto_save_last_index = len(clean_history)
-                logging.debug(f"Auto-saved {len(new_messages)} new messages to SessionStore: {self.auto_save}")
-            elif self._memory_instance and hasattr(self._memory_instance, 'save_session'):
-                # Fallback to Memory.save_session() for backward compatibility
-                # Memory.save_session() replaces entire history, so no duplicate issue
-                self._memory_instance.save_session(
-                    name=self.auto_save,
-                    conversation_history=clean_history,
-                    metadata={"agent_name": self.name, "user_id": self.user_id}
-                )
-                logging.debug(f"Auto-saved session to Memory: {self.auto_save}")
-        except Exception as e:
-            logging.debug(f"Error auto-saving session: {e}")
+                    logging.debug(f"Auto-saved session to Memory: {self.auto_save}")
+            except Exception as e:
+                logging.debug(f"Error auto-saving session: {e}")
 
     def _save_output_to_file(self, content: str) -> bool:
         """Save agent output to file if output_file is configured.

src/praisonai-agents/praisonaiagents/agent/tool_execution.py

@@ -29,6 +29,11 @@ logger = logging.getLogger(__name__)
 # timeouts so they cannot exhaust process resources.
 _MAX_ORPHANED_TOOL_EXECUTORS = 4
 
+# Worker count for the per-agent tool ThreadPoolExecutor. Used both when the
+# pool is created and to decide (with the per-executor in-flight counter)
+# whether a fresh call would have to queue behind hung workers.
+_TOOL_POOL_MAX_WORKERS = 2
+
 # Sentinel returned as the "arguments" value when a tool call's argument string
 # cannot be parsed (e.g. truncated by max_tokens or a dropped connection).
 # It is distinct from {} ("no arguments"): {} would silently execute the tool
@@ -39,6 +44,57 @@ _MAX_ORPHANED_TOOL_EXECUTORS = 4
 _TOOL_ARGUMENTS_PARSE_FAILED = object()
 
 
+class _CallableGuardrailAdapter:
+    """Expose a plain callable guardrail as a chainable guardrail object.
+
+    A user ``guardrail`` may be a bare callable returning ``(bool, output)``
+    (the Agent's final-output contract). To let it run *inside* a
+    ``GuardrailChain`` alongside a plugin guardrail — so composing the two never
+    silently drops either — this wraps the callable with a ``validate_output``
+    (and passthrough ``validate_input``) the chain understands. Any
+    ``validate_tool_call`` / ``validate_tool_result`` the underlying object
+    already exposes is forwarded so tool-surface behaviour is preserved.
+    """
+
+    __slots__ = ("_fn",)
+
+    def __init__(self, fn):
+        self._fn = fn
+
+    def validate_output(self, content, **kwargs):
+        result = self._fn(content)
+        if isinstance(result, tuple) and len(result) == 2:
+            is_valid, processed = result
+            if is_valid:
+                # Normalise the passthrough value to a string so the chain can
+                # keep threading ``content`` through subsequent guardrails.
+                if isinstance(processed, str):
+                    return True, processed
+                return True, content
+            return False, processed if isinstance(processed, str) else str(processed)
+        # A callable that does not follow the (bool, output) contract is treated
+        # as a no-op passthrough rather than blocking.
+        return True, content
+
+    def validate_input(self, content, **kwargs):
+        inner = getattr(self._fn, "validate_input", None)
+        if callable(inner):
+            return inner(content, **kwargs)
+        return True, content
+
+    def validate_tool_call(self, tool_name, arguments, **kwargs):
+        inner = getattr(self._fn, "validate_tool_call", None)
+        if callable(inner):
+            return inner(tool_name, arguments, **kwargs)
+        return True, arguments
+
+    def validate_tool_result(self, tool_name, result, **kwargs):
+        inner = getattr(self._fn, "validate_tool_result", None)
+        if callable(inner):
+            return inner(tool_name, result, **kwargs)
+        return True, result
+
+
 def tool_arguments_parse_failed(arguments) -> bool:
     """True when tool-call arguments could not be parsed (vs. legitimately empty)."""
     return arguments is _TOOL_ARGUMENTS_PARSE_FAILED
@@ -491,6 +547,15 @@ class ToolExecutionMixin:
         so disabling or unregistering a plugin revokes its tools from this
         agent's discovery and execution paths.
         """
+        # Hot-path fast exit: when no plugins are registered (the common case)
+        # skip constructing/scanning the manager entirely on every Agent init.
+        try:
+            from ..plugins.manager import has_registered_plugins
+
+            if not has_registered_plugins():
+                return
+        except Exception:  # pragma: no cover - defensive plugin boundary
+            pass
         try:
             from ..plugins import get_plugin_manager
 
@@ -597,6 +662,184 @@ class ToolExecutionMixin:
                 display_name = tool.get("type", "spec")
             logging.debug("Added plugin tool to Agent: %s", display_name or tool)
 
+    def _merge_plugin_subsystems(self):
+        """Route enabled GUARDRAIL/POLICY/SKILL plugins into their subsystems.
+
+        Complements ``_merge_plugin_tools``: instead of contributing tools, a
+        plugin declared ``PluginType.GUARDRAIL``/``POLICY``/``SKILL`` now
+        participates in the typed subsystem its category promises —
+
+        * GUARDRAIL: ``as_guardrail()`` objects are folded into ``self.guardrail``
+          so ``_setup_guardrail()`` (which runs after this) wires them into the
+          same ``GuardrailChain`` the Agent already runs — including
+          ``validate_tool_call`` / ``validate_tool_result``.
+        * POLICY: ``get_policies()`` rules are added to ``self._policy``
+          (a ``PolicyEngine``) so they gate tool calls.
+        * SKILL: ``get_skills()`` entries are appended to ``self._skills`` so the
+          lazy ``SkillManager`` discovers them as real, loadable skills.
+
+        Best-effort and fail-open on plugin errors: a broken plugin must never
+        prevent the Agent from constructing. Zero overhead when no such plugin
+        is enabled (each collector returns an empty list).
+        """
+        # Hot-path fast exit: when no plugins are registered (the common case)
+        # skip constructing/scanning the manager entirely on every Agent init.
+        try:
+            from ..plugins.manager import has_registered_plugins
+
+            if not has_registered_plugins():
+                return
+        except Exception:  # pragma: no cover - defensive plugin boundary
+            pass
+        try:
+            from ..plugins import get_plugin_manager
+
+            manager = get_plugin_manager()
+        except Exception as exc:  # pragma: no cover - defensive plugin boundary
+            logging.warning("Failed to access plugin manager for subsystems: %s", exc)
+            return
+
+        # --- GUARDRAIL plugins -> self.guardrail (GuardrailChain) -------------
+        get_guardrails = getattr(manager, "get_all_guardrails", None)
+        if callable(get_guardrails):
+            try:
+                plugin_guardrails = get_guardrails() or []
+            except Exception as exc:  # pragma: no cover - defensive boundary
+                logging.warning("Failed to collect plugin guardrails: %s", exc)
+                plugin_guardrails = []
+            if plugin_guardrails:
+                try:
+                    from ..guardrails import GuardrailChain
+                    from ..guardrails.protocols import is_guardrail_object
+
+                    chain_items = [
+                        g for g in plugin_guardrails if is_guardrail_object(g)
+                    ]
+                    if chain_items:
+                        chain = GuardrailChain(chain_items)
+                        # Register the plugin guardrails' tool-call/result
+                        # surface on the runtime lists ``_check_tool_policy_and_
+                        # guardrails`` / ``_apply_tool_result_guardrails`` read,
+                        # so a guardrail plugin finally sees tool calls and raw
+                        # tool results — the blind spot ``after_llm`` never
+                        # covered. Runs after ``_setup_guardrail()`` so we append
+                        # rather than replace any user-supplied guardrail surface.
+                        calls = getattr(self, "_tool_call_guardrails", None)
+                        if isinstance(calls, list):
+                            calls.append(chain)
+                        else:
+                            self._tool_call_guardrails = [chain]
+                        results = getattr(self, "_tool_result_guardrails", None)
+                        if isinstance(results, list):
+                            results.append(chain)
+                        else:
+                            self._tool_result_guardrails = [chain]
+                        # Also make the chain participate in the Agent's
+                        # input/output validation (not just the tool surface).
+                        existing_fn = getattr(self, "_guardrail_fn", None)
+                        if existing_fn is None:
+                            # No user guardrail: the plugin chain becomes the
+                            # Agent's output/input validator directly.
+                            self.guardrail = chain
+                            self._guardrail_fn = chain
+                        elif existing_fn is not chain:
+                            # User already supplied a guardrail. Compose rather
+                            # than replace so BOTH run on the final response:
+                            # wrap the user's callable in an adapter exposing
+                            # validate_output/validate_input, then chain it with
+                            # the plugin chain. The user's guardrail runs first;
+                            # a plugin guardrail can no longer be silently
+                            # skipped just because a user guardrail exists.
+                            adapter = _CallableGuardrailAdapter(existing_fn)
+                            combined = GuardrailChain([adapter, chain])
+                            self.guardrail = combined
+                            self._guardrail_fn = combined
+                except Exception as exc:  # pragma: no cover - defensive boundary
+                    logging.warning("Failed to wire plugin guardrails: %s", exc)
+
+        # --- POLICY plugins -> self._policy (PolicyEngine) -------------------
+        get_policies = getattr(manager, "get_all_policies", None)
+        if callable(get_policies):
+            try:
+                plugin_policies = get_policies() or []
+            except Exception as exc:  # pragma: no cover - defensive boundary
+                logging.warning("Failed to collect plugin policies: %s", exc)
+                plugin_policies = []
+            if plugin_policies:
+                try:
+                    policy = getattr(self, "_policy", None)
+                    if policy is None:
+                        from ..policy import PolicyEngine
+
+                        policy = PolicyEngine()
+                        self._policy = policy
+                    add_policy = getattr(policy, "add_policy", None)
+                    if callable(add_policy):
+                        # A user-supplied policy always wins: never let a plugin
+                        # rule silently replace an existing policy of the same
+                        # name (PolicyEngine.add_policy overwrites by name), which
+                        # could otherwise loosen a user-configured restriction.
+                        existing = set()
+                        get_policy = getattr(policy, "get_policy", None)
+                        for rule in plugin_policies:
+                            rule_name = getattr(rule, "name", None)
+                            if rule_name is not None and callable(get_policy):
+                                try:
+                                    if get_policy(rule_name) is not None:
+                                        logging.warning(
+                                            "Skipping plugin policy %r: a policy "
+                                            "with that name already exists.",
+                                            rule_name,
+                                        )
+                                        continue
+                                except Exception:  # pragma: no cover
+                                    pass
+                            if rule_name is not None and rule_name in existing:
+                                logging.warning(
+                                    "Skipping duplicate plugin policy %r.",
+                                    rule_name,
+                                )
+                                continue
+                            try:
+                                add_policy(rule)
+                                if rule_name is not None:
+                                    existing.add(rule_name)
+                            except Exception as exc:  # pragma: no cover
+                                logging.warning(
+                                    "Failed to add plugin policy %r: %s", rule, exc
+                                )
+                except Exception as exc:  # pragma: no cover - defensive boundary
+                    logging.warning("Failed to wire plugin policies: %s", exc)
+
+        # --- SKILL plugins -> self._skills (SkillManager discovers them) -----
+        get_skills = getattr(manager, "get_all_skills", None)
+        if callable(get_skills):
+            try:
+                plugin_skills = get_skills() or []
+            except Exception as exc:  # pragma: no cover - defensive boundary
+                logging.warning("Failed to collect plugin skills: %s", exc)
+                plugin_skills = []
+            if plugin_skills:
+                current = getattr(self, "_skills", None)
+                if not isinstance(current, list):
+                    current = [current] if current else []
+                for skill in plugin_skills:
+                    # ``self._skills`` feeds ``SkillManager.add_skill(path)``,
+                    # which resolves each entry as a filesystem path. Only
+                    # string path selectors are loadable; a non-string (e.g. a
+                    # source object) would raise inside ``Path()``, so skip it
+                    # with a clear warning rather than corrupt discovery.
+                    if not isinstance(skill, str):
+                        logging.warning(
+                            "Skipping plugin skill %r: get_skills() must return "
+                            "filesystem path strings to skill directories.",
+                            skill,
+                        )
+                        continue
+                    if skill not in current:
+                        current.append(skill)
+                self._skills = current
+
     def _cast_arguments(self, func, arguments):
         """Cast arguments to their expected types based on function signature."""
         if not callable(func) or not arguments:
@@ -1081,13 +1324,72 @@ class ToolExecutionMixin:
                                         return self._execute_tool_with_circuit_breaker(function_name, arguments)
 
                                 # Use reusable executor to prevent resource leaks
+                                pool_exhausted = False
                                 with self._get_tool_executor_lock().sync():
                                     if not hasattr(self, '_tool_executor') or self._tool_executor is None:
                                         self._tool_executor = concurrent.futures.ThreadPoolExecutor(
-                                            max_workers=2, thread_name_prefix=f"tool-{self.name}"
+                                            max_workers=_TOOL_POOL_MAX_WORKERS,
+                                            thread_name_prefix=f"tool-{self.name}"
                                         )
+                                        # Per-executor in-flight counter so a
+                                        # retired pool's late-finishing future can
+                                        # never corrupt the *new* pool's count.
+                                        self._tool_executor._inflight = 0
                                     executor = self._tool_executor
-                                    future = executor.submit(ctx.run, execute_with_context)
+                                    orphaned = getattr(self, '_tool_executor_orphaned', 0)
+                                    inflight = getattr(executor, '_inflight', 0)
+                                    # Once the orphan cap is hit we can no longer
+                                    # retire a stuck pool, so we would otherwise
+                                    # queue behind hung workers and time out a
+                                    # healthy tool that never got to start. Refuse
+                                    # fast and visibly instead when no worker is
+                                    # free, and let the pool self-heal (the orphan
+                                    # count drops as hung threads finish).
+                                    if orphaned >= _MAX_ORPHANED_TOOL_EXECUTORS and inflight >= _TOOL_POOL_MAX_WORKERS:
+                                        pool_exhausted = True
+                                    else:
+                                        executor._inflight = inflight + 1
+                                        future = executor.submit(ctx.run, execute_with_context)
+
+                                        def _on_tool_done(_f, _agent=self, _executor=executor):  # noqa: ANN001
+                                            with _agent._get_tool_executor_lock().sync():
+                                                _executor._inflight = max(0, getattr(_executor, '_inflight', 1) - 1)
+                                                # Orphan accounting is per-POOL, not
+                                                # per-future: retiring a pool adds
+                                                # exactly one to the orphan count, so
+                                                # a pool may return exactly one slot.
+                                                # Recover it only when this retired
+                                                # pool's LAST in-flight worker exits
+                                                # (``_inflight`` == 0), and guard with
+                                                # a one-shot flag so a multi-worker
+                                                # pool can never decrement more than
+                                                # the single unit it contributed and
+                                                # steal another orphaned pool's slot.
+                                                if (
+                                                    getattr(_agent, '_tool_executor', None) is not _executor
+                                                    and _executor._inflight == 0
+                                                    and not getattr(_executor, '_orphan_recovered', False)
+                                                ):
+                                                    _executor._orphan_recovered = True
+                                                    _agent._tool_executor_orphaned = max(
+                                                        0, getattr(_agent, '_tool_executor_orphaned', 0) - 1
+                                                    )
+
+                                        future.add_done_callback(_on_tool_done)
+                                if pool_exhausted:
+                                    logging.warning(
+                                        f"Tool worker pool exhausted; refusing to queue '{function_name}' "
+                                        f"behind {orphaned} hung tool call(s)."
+                                    )
+                                    result = {
+                                        "error": (
+                                            f"Tool worker pool exhausted: {orphaned} earlier tool call(s) "
+                                            f"are still running past their timeout. Refusing to queue "
+                                            f"'{function_name}'."
+                                        ),
+                                        "tool_pool_exhausted": True,
+                                    }
+                                    break
                                 try:
                                     result = future.result(timeout=tool_timeout)
                                 except concurrent.futures.TimeoutError:
@@ -1620,6 +1922,11 @@ class ToolExecutionMixin:
         ``_atrigger_after_agent_hook`` without any async-safety change.
         """
         from ..hooks import HookEvent, AfterAgentInput
+        # AfterAgentInput.response is typed/serialized as a string (its
+        # to_dict truncates via ``response[:500]``). Structured-output turns
+        # (output_pydantic) return a Pydantic instance, so coerce to str here
+        # to keep string-only hooks (e.g. command hooks) from crashing while
+        # the public chat() return value stays the model.
         return AfterAgentInput(
             session_id=getattr(self, '_session_id', 'default'),
             cwd=os.getcwd(),
@@ -1627,7 +1934,7 @@ class ToolExecutionMixin:
             timestamp=str(time.time()),
             agent_name=self.name,
             prompt=prompt if isinstance(prompt, str) else str(prompt),
-            response=response or "",
+            response=response if isinstance(response, str) else (str(response) if response else ""),
             tools_used=tools_used or [],
             total_tokens=0,
             execution_time_ms=(time.time() - start_time) * 1000
@@ -1644,7 +1951,16 @@ class ToolExecutionMixin:
         # Auto-memory extraction (opt-in via MemoryConfig(auto_memory=True))
         if response:
             prompt_str = prompt if isinstance(prompt, str) else str(prompt)
-            self._process_auto_memory(prompt_str, str(response))
+            if getattr(self, "_auto_memory", False):
+                self._process_auto_memory(prompt_str, str(response))
+            else:
+                # When memory is enabled but auto_memory extraction is off, still
+                # persist the raw turn so a later Agent sharing the same memory
+                # (e.g. memory={"user_id": uid}) can recall it. Without this the
+                # sync chat/run/start path wrote nothing and cross-session recall
+                # silently returned empty (issue #5595). Mirrors the write already
+                # done on the async unified path.
+                self._persist_memory_turn(prompt_str, str(response))
 
         # Auto-learning extraction (opt-in via LearnConfig(mode=LearnMode.AGENTIC))
         self._process_auto_learning()
@@ -1898,6 +2214,30 @@ class ToolExecutionMixin:
         tail = text[-tail_limit:] if tail_limit > 0 else ""
         return f"{head}\n...[{len(text):,} chars, showing first/last portions]...\n{tail}"
 
+    @staticmethod
+    def _approval_backend_accepts_timeout(backend) -> bool:
+        """Return True if ``backend.request_approval_sync`` accepts a ``timeout`` kwarg.
+
+        Determined by signature introspection so the caller can route the
+        per-call timeout without a call-and-catch that would re-invoke a
+        backend whose *own* body raised ``TypeError``. Backends that accept
+        ``**kwargs`` are treated as accepting ``timeout``. On any introspection
+        failure we conservatively return False (call without ``timeout``).
+        """
+        method = getattr(backend, 'request_approval_sync', None)
+        if method is None:
+            return False
+        try:
+            sig = inspect.signature(method)
+        except (TypeError, ValueError):
+            return False
+        params = sig.parameters
+        if 'timeout' in params:
+            return True
+        return any(
+            p.kind == inspect.Parameter.VAR_KEYWORD for p in params.values()
+        )
+
     def _resolve_approval_decision(self, tool_name: str, tool_args: dict, is_async: bool = False):
         """Shared approval logic for both sync and async paths.
         
@@ -2071,41 +2411,51 @@ class ToolExecutionMixin:
 
                     return _ask_backend_async()
                 else:
-                    # Sync path - handle timeout and sync/async backend compatibility
+                    # Sync path - handle timeout and sync/async backend compatibility.
+                    #
+                    # The per-call timeout is passed explicitly rather than
+                    # mutated onto the (often SHARED) backend object. Mutating
+                    # ``backend._timeout`` here leaked between agents that pass
+                    # one backend instance: an overlapping call captured the
+                    # first agent's temporary value as its "original" and
+                    # restored it, leaving the backend permanently wrong.
                     cfg_timeout = getattr(self, '_approval_timeout', 0)
-                    orig_timeout = None
+
+                    # Compute effective timeout from agent configuration.
+                    #   None  -> indefinite wait
+                    #   > 0   -> explicit per-call timeout
+                    #   == 0  -> backend default (or 60s fallback)
                     if cfg_timeout is None:
-                        orig_timeout = getattr(backend, '_timeout', None)
-                        if orig_timeout is not None:
-                            backend._timeout = 86400 * 365
+                        effective_timeout = None
                     elif cfg_timeout > 0:
-                        orig_timeout = getattr(backend, '_timeout', None)
-                        if orig_timeout is not None:
-                            backend._timeout = cfg_timeout
-                    
-                    try:
-                        if hasattr(backend, 'request_approval_sync'):
-                            return _remember(backend.request_approval_sync(request))
-                        else:
-                            # Use the shared utility to avoid code duplication and handle timeout correctly
-                            from ..approval.utils import run_coroutine_safely
-                            
-                            # Compute effective timeout from agent configuration
-                            if cfg_timeout is None:
-                                effective_timeout = None  # indefinite wait
-                            elif cfg_timeout > 0:
-                                effective_timeout = cfg_timeout
-                            else:
-                                # cfg_timeout == 0: use backend default or fallback
-                                effective_timeout = getattr(backend, '_timeout', 60)
-                            
-                            return _remember(run_coroutine_safely(
-                                backend.request_approval(request),
-                                timeout=effective_timeout
+                        effective_timeout = cfg_timeout
+                    else:
+                        effective_timeout = getattr(backend, '_timeout', 60)
+
+                    if hasattr(backend, 'request_approval_sync'):
+                        # Decide *up front* whether the backend accepts a
+                        # ``timeout`` kwarg by inspecting its signature, rather
+                        # than calling and catching ``TypeError``. A try/except
+                        # would re-invoke the backend when the *user callback*
+                        # inside it raised its own ``TypeError`` — duplicating
+                        # side effects and masking the real error. Signature
+                        # introspection is side-effect free.
+                        if self._approval_backend_accepts_timeout(backend):
+                            return _remember(backend.request_approval_sync(
+                                request, timeout=effective_timeout
                             ))
-                    finally:
-                        if orig_timeout is not None and hasattr(backend, '_timeout'):
-                            backend._timeout = orig_timeout
+                        # Legacy backend whose request_approval_sync has no
+                        # timeout kwarg: honour its own default, no mutation of
+                        # (often shared) backend state.
+                        return _remember(backend.request_approval_sync(request))
+                    else:
+                        # Use the shared utility to avoid code duplication and handle timeout correctly
+                        from ..approval.utils import run_coroutine_safely
+
+                        return _remember(run_coroutine_safely(
+                            backend.request_approval(request),
+                            timeout=effective_timeout
+                        ))
             else:
                 if is_async:
                     # For async, wrap the decision in a coroutine
@@ -2514,6 +2864,42 @@ class ToolExecutionMixin:
                 logging.debug("permission manager is_denied failed for %s: %s", target, e)
         return None
 
+    def _check_permission_manager_doom_loop(self, function_name, arguments=None):
+        """Return an error dict if the PermissionManager detects a doom loop.
+
+        Deliver the doom-loop detection the ``PermissionManager`` advertises:
+        record this call in its detector and deny on a genuine loop. Previously
+        ``check_doom_loop()`` was unreachable dead code so a configured manager
+        gave zero doom-loop protection.
+
+        This is intentionally **separate** from :meth:`_check_permission_manager_deny`
+        and invoked exactly **once per approved tool call**, with the final
+        (post-approval, possibly rewritten) arguments. Folding it into the deny
+        gate would record the call on every gate pass — counting a rejected
+        attempt, or counting an approved call twice when its args are rewritten —
+        which could trip the threshold early and deny legitimate repeats.
+        """
+        manager = getattr(self, "_permission_manager", None)
+        if manager is None:
+            return None
+        check_loop = getattr(manager, "check_doom_loop", None)
+        if not callable(check_loop):
+            return None
+        try:
+            loop_result = check_loop(function_name, arguments)
+            if loop_result is not None and getattr(loop_result, "is_loop", False):
+                return {
+                    "error": (
+                        f"Tool '{function_name}' blocked: "
+                        f"{getattr(loop_result, 'reason', 'doom loop detected')}"
+                    ),
+                    "permission_denied": True,
+                    "loop_blocked": True,
+                }
+        except Exception as e:  # noqa: BLE001
+            logging.debug("permission manager doom-loop check failed for %s: %s", function_name, e)
+        return None
+
     def _is_bypass_mode(self) -> bool:
         """Return ``True`` when ``PermissionMode.BYPASS`` is active.
 
@@ -2575,6 +2961,13 @@ class ToolExecutionMixin:
                 if manager_denial is not None:
                     return manager_denial
 
+        # Record the (approved, final-args) call in the doom-loop detector once,
+        # after approval, so rejected attempts and arg rewrites are not counted.
+        if not self._is_bypass_mode():
+            loop_denial = self._check_permission_manager_doom_loop(function_name, arguments)
+            if loop_denial is not None:
+                return loop_denial
+
         from ..approval import get_approval_registry
         get_approval_registry().mark_approved(
             function_name, arguments, agent_name=getattr(self, "name", None),
@@ -2621,6 +3014,13 @@ class ToolExecutionMixin:
                 if manager_denial is not None:
                     return manager_denial
 
+        # Record the (approved, final-args) call in the doom-loop detector once,
+        # after approval, so rejected attempts and arg rewrites are not counted.
+        if not self._is_bypass_mode():
+            loop_denial = self._check_permission_manager_doom_loop(function_name, arguments)
+            if loop_denial is not None:
+                return loop_denial
+
         from ..approval import get_approval_registry
         get_approval_registry().mark_approved(
             function_name, arguments, agent_name=getattr(self, "name", None),

src/praisonai-agents/praisonaiagents/agents/agents.py

@@ -24,10 +24,7 @@ try:
 except ImportError:
     get_token_collector = None
 
-# Guards lazy creation of each AgentTeam's per-instance _run_lock so two threads
-# that first reach start()/astart() concurrently observe the same lock object
-# (double-checked locking) rather than each minting and acquiring its own.
-_RUN_LOCK_INIT_GUARD = threading.Lock()
+from .._run_lock import ensure_run_lock
 
 # Task status constants
 class TaskStatus(Enum):
@@ -454,7 +451,10 @@ def _build_execution_context(agents_instance, task_id, skip_memory_init=False):
     if llm is None:
         llm = getattr(executor_agent, 'llm', None)
     if llm and hasattr(llm, 'set_current_agent'):
-        llm.set_current_agent(executor_agent.display_name)
+        # Pass the agent's stable per-instance id so token usage is aggregated
+        # by identity, not by (collidable) display name.
+        agent_scope_id = getattr(executor_agent, '_approval_scope_id', None)
+        llm.set_current_agent(executor_agent.display_name, agent_id=agent_scope_id)
 
     # Place the agent's tools BEFORE snapshotting them. This snapshot is passed
     # to chat() as an explicit `tools=`, which wins over `agent.tools` -- so if
@@ -2085,14 +2085,7 @@ class AgentTeam(SpawnAnnounceProtocol):
         guard (double-checked locking) so two threads racing into start()/astart()
         observe the same lock object.
         """
-        lock = getattr(self, '_run_lock', None)
-        if lock is None:
-            with _RUN_LOCK_INIT_GUARD:
-                lock = getattr(self, '_run_lock', None)
-                if lock is None:
-                    lock = threading.Lock()
-                    self._run_lock = lock
-        return lock
+        return ensure_run_lock(self)
 
     def _execution_identity(self):
         """Identity of the current execution for re-entrancy checks.
@@ -2379,6 +2372,20 @@ class AgentTeam(SpawnAnnounceProtocol):
 
     def run_all_tasks(self):
         """Synchronous version of run_all_tasks method"""
+        # The sync entry point (.start()/.run()) drives tasks one at a time and
+        # cannot batch async_execution=True tasks via asyncio.gather the way
+        # arun_all_tasks does. Warn once so the parallel fan-out feature the
+        # constructor docstring/ValueError advertise doesn't silently degrade to
+        # sequential execution -- point the caller at the async entry point.
+        if self.process in ("workflow", "sequential") and any(
+            getattr(t, 'async_execution', False) for t in self.tasks.values()
+        ):
+            logger.warning(
+                "One or more tasks have async_execution=True, but the synchronous "
+                "entry point (.start()/.run()) runs them sequentially -- use "
+                "`await team.astart()` for parallel fan-out."
+            )
+
         process = Process(
             tasks=self.tasks,
             agents=self.agents,
@@ -3258,21 +3265,45 @@ class AgentTeam(SpawnAnnounceProtocol):
             if getattr(agent, "name", None)
         }
 
-    def _scoped_recent_interactions(self, own_names: set) -> List[Dict[str, Any]]:
+    def _own_agent_ids(self) -> set:
+        """Stable per-instance identities of this instance's own agents.
+
+        Used as the collision-free key for token-report scoping so two agents
+        that share a display name (e.g. the default ``"Agent"``) are never
+        merged. Falls back to names elsewhere when ids are unavailable.
+        """
+        return {
+            getattr(agent, "_approval_scope_id", None)
+            for agent in (self.agents or [])
+            if getattr(agent, "_approval_scope_id", None)
+        }
+
+    def _scoped_recent_interactions(
+        self, own_names: set, own_ids: Optional[set] = None
+    ) -> List[Dict[str, Any]]:
         """Return this instance's own recorded interactions, most-recent last.
 
         The process-wide collector keeps a bounded window of every agent's
-        interactions; filter it to this instance's agent names so per-model
-        totals, interaction counts, and the detailed report never leak another
-        concurrent instance's records.
+        interactions; filter it to this instance's agents so per-model totals,
+        interaction counts, and the detailed report never leak another
+        concurrent instance's records. Filtering prefers the collision-free
+        ``agent_id`` when present, falling back to the display name.
         """
+        own_ids = own_ids or set()
         try:
             interactions = get_token_collector().get_recent_interactions(
                 limit=get_token_collector()._max_recent
             )
         except Exception:
             interactions = get_token_collector().get_recent_interactions(limit=100)
-        return [i for i in interactions if i.get("agent") in own_names]
+
+        def _matches(i: Dict[str, Any]) -> bool:
+            agent_id = i.get("agent_id")
+            if agent_id is not None and own_ids:
+                return agent_id in own_ids
+            return i.get("agent") in own_names
+
+        return [i for i in interactions if _matches(i)]
 
     def _scoped_token_summary(self) -> Dict[str, Any]:
         """Return the global token summary filtered to this instance's agents.
@@ -3280,25 +3311,54 @@ class AgentTeam(SpawnAnnounceProtocol):
         The process-wide TokenCollector aggregates every agent in the process,
         so two concurrent PraisonAIAgents instances would otherwise read each
         other's totals. Scope every reported field — ``by_agent``, ``by_model``,
-        ``total_interactions`` and the totals — to this instance's own agent
-        names so per-instance/per-tenant cost accounting reports only its own
-        spend. ``by_agent`` is taken from the collector's aggregate breakdown;
-        ``by_model`` and ``total_interactions`` are re-derived from this
-        instance's own interaction records (the aggregate cannot be split by
-        model per agent). Falls back to the unfiltered summary when this
-        instance has no named agents to scope by.
+        ``total_interactions`` and the totals — to this instance's own agents so
+        per-instance/per-tenant cost accounting reports only its own spend.
+        Scoping prefers the collector's collision-free ``by_agent_id``
+        breakdown; ``by_model`` and ``total_interactions`` are re-derived from
+        this instance's own interaction records (the aggregate cannot be split
+        by model per agent). Falls back to the unfiltered summary when this
+        instance has no agents to scope by.
         """
         summary = get_token_collector().get_session_summary()
         own_names = self._own_agent_names()
+        own_ids = self._own_agent_ids()
         by_agent = summary.get("by_agent", {})
-        # No named agents to scope by (or nothing tracked yet): fall back to the
+        by_agent_id = summary.get("by_agent_id", {})
+        # No agents to scope by (or nothing tracked yet): fall back to the
         # unfiltered summary rather than silently reporting zeros.
-        if not own_names or not by_agent:
+        if (not own_names and not own_ids) or (not by_agent and not by_agent_id):
             return summary
 
-        scoped_by_agent = {
-            name: metrics for name, metrics in by_agent.items() if name in own_names
-        }
+        # Prefer the collision-free per-identity breakdown; fall back to
+        # name-based scoping when ids are unavailable (backward compatible).
+        scoped_by_agent: Dict[str, Dict[str, int]] = {}
+        if own_ids and by_agent_id:
+            for key, entry in by_agent_id.items():
+                name = entry.get("name") or key
+                # Match by stable id (collision-free) OR by display name for
+                # id-less buckets — e.g. the native OpenAI-client path records
+                # ``agent`` without an ``agent_id``, so the collector keys that
+                # bucket by name (``key == name``). Dropping those would zero out
+                # real OpenAI usage even though the interaction and per-model
+                # rows still show it (Greptile P1). We treat a bucket as id-less
+                # when its key equals its own display name.
+                is_id_match = key in own_ids
+                is_idless_name_match = key == name and name in own_names
+                if not (is_id_match or is_idless_name_match):
+                    continue
+                metrics = entry.get("metrics", {})
+                if name in scoped_by_agent:
+                    scoped_by_agent[name] = {
+                        k: scoped_by_agent[name].get(k, 0) + v
+                        for k, v in metrics.items()
+                        if isinstance(v, (int, float))
+                    }
+                else:
+                    scoped_by_agent[name] = dict(metrics)
+        else:
+            scoped_by_agent = {
+                name: metrics for name, metrics in by_agent.items() if name in own_names
+            }
         totals = {
             "input_tokens": 0, "output_tokens": 0, "cached_tokens": 0,
             "reasoning_tokens": 0, "audio_input_tokens": 0, "audio_output_tokens": 0,
@@ -3314,7 +3374,7 @@ class AgentTeam(SpawnAnnounceProtocol):
         # process-global summary (which mixes in other instances' models and
         # counts). The collector's ``by_agent`` aggregate cannot be split by
         # model, so the per-interaction window is the correct source.
-        own_interactions = self._scoped_recent_interactions(own_names)
+        own_interactions = self._scoped_recent_interactions(own_names, own_ids)
         by_model: Dict[str, Dict[str, int]] = {}
         for interaction in own_interactions:
             model = interaction.get("model")
@@ -3348,10 +3408,11 @@ class AgentTeam(SpawnAnnounceProtocol):
         
         summary = self._scoped_token_summary()
         own_names = self._own_agent_names()
-        if own_names:
+        own_ids = self._own_agent_ids()
+        if own_names or own_ids:
             # Only surface this instance's own interactions so the detailed
             # report never exposes another concurrent instance's records.
-            recent = self._scoped_recent_interactions(own_names)[-20:]
+            recent = self._scoped_recent_interactions(own_names, own_ids)[-20:]
         else:
             recent = get_token_collector().get_recent_interactions(limit=20)
         

src/praisonai-agents/praisonaiagents/approval/backends.py

@@ -13,7 +13,7 @@ from __future__ import annotations
 import asyncio
 import re
 from praisonaiagents._logging import get_logger
-from typing import Any
+from typing import Any, Optional
 
 from .protocols import ApprovalDecision, ApprovalRequest
 
@@ -175,7 +175,7 @@ class AutoApproveBackend:
     async def request_approval(self, request: ApprovalRequest) -> ApprovalDecision:
         return ApprovalDecision(approved=True, reason="auto-approved", approver="system")
 
-    def request_approval_sync(self, request: ApprovalRequest) -> ApprovalDecision:
+    def request_approval_sync(self, request: ApprovalRequest, timeout: Optional[float] = None) -> ApprovalDecision:
         return ApprovalDecision(approved=True, reason="auto-approved", approver="system")
 
 class ConsoleBackend:
@@ -293,7 +293,7 @@ class ConsoleBackend:
             return (True, "always", suggested or None, None)
         return (True, "once", None, None)
 
-    def request_approval_sync(self, request: ApprovalRequest) -> ApprovalDecision:
+    def request_approval_sync(self, request: ApprovalRequest, timeout: Optional[float] = None) -> ApprovalDecision:
         """Synchronous approval via Rich console prompt."""
         try:
             approved, scope, scope_pattern, feedback = self._prompt_user(request)
@@ -474,10 +474,12 @@ class AgentApproval:
                 reason=f"Agent approval error: {e}",
             )
 
-    def request_approval_sync(self, request: ApprovalRequest) -> ApprovalDecision:
+    def request_approval_sync(self, request: ApprovalRequest, timeout: Optional[float] = None) -> ApprovalDecision:
         """Synchronous wrapper."""
         from .utils import run_coroutine_safely
-        return run_coroutine_safely(self.request_approval(request), timeout=60)
+        return run_coroutine_safely(
+            self.request_approval(request), timeout=60 if timeout is None else timeout
+        )
 
 class CallbackBackend:
     """Wraps a legacy ``(function_name, arguments, risk_level) -> ApprovalDecision`` callback
@@ -489,7 +491,7 @@ class CallbackBackend:
     def __init__(self, callback):
         self._callback = callback
 
-    def request_approval_sync(self, request: ApprovalRequest) -> ApprovalDecision:
+    def request_approval_sync(self, request: ApprovalRequest, timeout: Optional[float] = None) -> ApprovalDecision:
         result = self._callback(request.tool_name, request.arguments, request.risk_level)
         if isinstance(result, ApprovalDecision):
             return result

src/praisonai-agents/praisonaiagents/approval/protocols.py

@@ -189,6 +189,18 @@ class ApprovalProtocol(Protocol):
         """
         ...
 
+    # Backends MAY also implement an optional synchronous entry point:
+    #
+    #     def request_approval_sync(self, request, timeout=None) -> ApprovalDecision
+    #
+    # It is deliberately NOT declared on this runtime-checkable Protocol so an
+    # async-only backend still satisfies ``isinstance(b, ApprovalProtocol)``.
+    # When present, the per-call ``timeout`` (seconds; ``None`` = wait forever)
+    # is passed explicitly by the caller so shared backend state is never
+    # mutated — mutating a shared backend's timeout leaks between agents.
+    # Legacy sync backends without the ``timeout`` parameter are still
+    # supported; the caller falls back to a no-argument call on ``TypeError``.
+
 
 @runtime_checkable
 class ApprovalStoreProtocol(Protocol):

src/praisonai-agents/praisonaiagents/background/runner.py

@@ -69,6 +69,13 @@ class BackgroundRunner:
         """
         self.config = config or BackgroundConfig()
         self._tasks: Dict[str, BackgroundTask] = {}
+        # Guards every read/write of ``_tasks``. This runner is a process-wide
+        # singleton (``get_background_runner``) whose sync wrappers insert on a
+        # separate background-loop thread via ``run_coroutine_threadsafe`` while
+        # other surfaces read/mutate the dict on their own thread; without this
+        # a concurrent insert + iterate/delete can raise
+        # ``RuntimeError: dictionary changed size during iteration``.
+        self._tasks_lock: threading.Lock = threading.Lock()
         # Lazily create asyncio.Semaphore() to avoid Python 3.9 event loop issues
         self._semaphore: Optional[asyncio.Semaphore] = None
         self._semaphore_lock: threading.Lock = threading.Lock()  # Thread-safe initialization
@@ -86,17 +93,20 @@ class BackgroundRunner:
     @property
     def tasks(self) -> List[BackgroundTask]:
         """Get all tasks."""
-        return list(self._tasks.values())
+        with self._tasks_lock:
+            return list(self._tasks.values())
     
     @property
     def running_tasks(self) -> List[BackgroundTask]:
         """Get currently running tasks."""
-        return [t for t in self._tasks.values() if t.is_running]
+        with self._tasks_lock:
+            return [t for t in self._tasks.values() if t.is_running]
     
     @property
     def pending_tasks(self) -> List[BackgroundTask]:
         """Get pending tasks."""
-        return [t for t in self._tasks.values() if t.status == TaskStatus.PENDING]
+        with self._tasks_lock:
+            return [t for t in self._tasks.values() if t.status == TaskStatus.PENDING]
     
     async def submit(
         self,
@@ -133,7 +143,8 @@ class BackgroundRunner:
             }
         )
         
-        self._tasks[task.id] = task
+        with self._tasks_lock:
+            self._tasks[task.id] = task
         
         # Create the execution coroutine
         async def execute():
@@ -235,7 +246,8 @@ class BackgroundRunner:
     
     def get_task(self, task_id: str) -> Optional[BackgroundTask]:
         """Get a task by ID."""
-        return self._tasks.get(task_id)
+        with self._tasks_lock:
+            return self._tasks.get(task_id)
     
     async def cancel_task(self, task_id: str) -> bool:
         """
@@ -247,7 +259,8 @@ class BackgroundRunner:
         Returns:
             True if cancelled, False if not found or already completed
         """
-        task = self._tasks.get(task_id)
+        with self._tasks_lock:
+            task = self._tasks.get(task_id)
         if task is None:
             return False
         
@@ -276,10 +289,12 @@ class BackgroundRunner:
         Returns:
             List of completed tasks
         """
-        futures = [t._future for t in self._tasks.values() if t._future]
+        with self._tasks_lock:
+            futures = [t._future for t in self._tasks.values() if t._future]
         
         if not futures:
-            return list(self._tasks.values())
+            with self._tasks_lock:
+                return list(self._tasks.values())
         
         try:
             await asyncio.wait_for(
@@ -289,16 +304,18 @@ class BackgroundRunner:
         except asyncio.TimeoutError:
             pass
         
-        return list(self._tasks.values())
+        with self._tasks_lock:
+            return list(self._tasks.values())
     
     def clear_completed(self):
         """Remove completed tasks from tracking."""
-        completed_ids = [
-            task_id for task_id, task in self._tasks.items()
-            if task.is_completed
-        ]
-        for task_id in completed_ids:
-            del self._tasks[task_id]
+        with self._tasks_lock:
+            completed_ids = [
+                task_id for task_id, task in self._tasks.items()
+                if task.is_completed
+            ]
+            for task_id in completed_ids:
+                del self._tasks[task_id]
     
     def list_tasks(self, status: Optional[TaskStatus] = None) -> List[Dict[str, Any]]:
         """
@@ -310,7 +327,8 @@ class BackgroundRunner:
         Returns:
             List of task dictionaries
         """
-        tasks = self._tasks.values()
+        with self._tasks_lock:
+            tasks = list(self._tasks.values())
         
         if status:
             tasks = [t for t in tasks if t.status == status]

src/praisonai-agents/praisonaiagents/bots/config.py

@@ -125,11 +125,29 @@ class BotConfig:
     
     # Edit interval for streaming responses in milliseconds (default: 700ms)
     stream_edit_interval_ms: int = 700
+
+    # Opt-in privacy-safe runtime footer on the final reply (default: False).
+    # When True, a short trailing line (e.g. ``— gpt-4o · 4.2s``) reporting the
+    # model and turn latency is appended to the final reply only. Privacy-safe:
+    # never surfaces tool args, URLs, paths or message content. Unavailable
+    # fields are omitted rather than shown as placeholders.
+    footer: bool = False
     
     # Intentional silence support
     allow_silence: bool = False
     silence_token: Optional[str] = None
 
+    # Bot-to-bot inbound (#5062). Off by default: a bot-authored inbound
+    # message is dropped as before. When ``allow_bots=True`` the channel accepts
+    # bot-authored messages and the runaway-loop guard (``BotLoopGuard``) is
+    # auto-active, tracking each participant pair and suppressing further replies
+    # once ``bot_loop_protection``'s sliding-window budget is exceeded.
+    # ``bot_loop_protection`` is a plain dict (enabled / max_events_per_window /
+    # window_seconds / cooldown_seconds) coerced into a ``BotLoopPolicy``; ``None``
+    # uses the default budget.
+    allow_bots: bool = False
+    bot_loop_protection: Optional[Dict[str, Any]] = None
+
     def __post_init__(self) -> None:
         if self.unknown_user_policy not in {"deny", "pair", "allow"}:
             raise ValueError(
@@ -174,8 +192,11 @@ class BotConfig:
             "owner_user_id": "***" if self.owner_user_id else None,
             "streaming": self.streaming,
             "stream_edit_interval_ms": self.stream_edit_interval_ms,
+            "footer": self.footer,
             "allow_silence": self.allow_silence,
             "silence_token": self.silence_token,
+            "allow_bots": self.allow_bots,
+            "bot_loop_protection": self.bot_loop_protection,
             "metadata": self.metadata,
         }
     

src/praisonai-agents/praisonaiagents/bots/silence.py

@@ -138,15 +138,31 @@ class BotLoopPolicy:
         """Build a policy from a config mapping (YAML/kwargs), tolerating None.
 
         Unknown keys are ignored so a forward-compatible config never breaks
-        construction. ``None`` yields the enabled default policy.
+        construction. ``None`` yields the enabled default policy. Malformed
+        numeric values (e.g. ``max_events_per_window: "abc"``) fall back to the
+        field default rather than raising, so a config typo cannot crash the
+        inbound path on the first bot message — the guard stays fail-safe.
         """
         if not data:
             return cls()
+
+        def _as_int(value: object, default: int) -> int:
+            try:
+                return int(value)  # type: ignore[arg-type]
+            except (TypeError, ValueError):
+                return default
+
+        def _as_float(value: object, default: float) -> float:
+            try:
+                return float(value)  # type: ignore[arg-type]
+            except (TypeError, ValueError):
+                return default
+
         return cls(
             enabled=bool(data.get("enabled", True)),
-            max_events_per_window=int(data.get("max_events_per_window", 20)),
-            window_seconds=float(data.get("window_seconds", 60.0)),
-            cooldown_seconds=float(data.get("cooldown_seconds", 60.0)),
+            max_events_per_window=_as_int(data.get("max_events_per_window", 20), 20),
+            window_seconds=_as_float(data.get("window_seconds", 60.0), 60.0),
+            cooldown_seconds=_as_float(data.get("cooldown_seconds", 60.0), 60.0),
         )
 
 

src/praisonai-agents/praisonaiagents/bus/bus.py

@@ -267,17 +267,13 @@ class EventBus:
             The published Event object
         """
         type_str = event_type.value if isinstance(event_type, EventType) else event_type
-        
-        # Fast path: if no subscribers and no durable sinks, return a minimal
-        # event without expensive operations.
-        if not self._subscribers and not self._sinks:
-            return Event(
-                type=type_str,
-                data=data or {},
-                source=source,
-                metadata=metadata or {},
-            )
-        
+
+        # No pre-history no-subscriber shortcut here, matching publish_event's
+        # already-fixed shape: an event that was published happened, whether or
+        # not anyone was listening, and skipping the history append below would
+        # make it vanish from get_history(). The append is O(1) under a lock and
+        # the list is capped, so the only work worth skipping is subscriber
+        # matching/dispatch -- which the guard after the append still skips.
         event = Event(
             type=type_str,
             data=data or {},
@@ -336,24 +332,20 @@ class EventBus:
         """
         Get recent event history.
 
-        Only events published while at least one subscriber was registered are
-        recorded. #2066 added a fast path that skips history (and the lock, and
-        Event construction) when ``has_subscribers`` is False, because memory
-        and sub-agent lifecycle publishes were paying uuid4 + lock + append on
-        every call with nothing listening.
-
-        The consequence is worth stating plainly, because it is surprising in
-        the case you would most want history -- debugging after the fact, with
-        nothing subscribed: publishing three events to a bus with no
-        subscribers and then calling this returns an empty list, not three
-        events. Subscribe (even a no-op) before publishing if you need a record.
+        Every event passed to ``publish``/``publish_event``/``publish_async`` is
+        recorded here (capped at ``_max_history``), regardless of whether any
+        subscriber was registered at publish time. History is a debugging record
+        of what the system did; it does not silently depend on who was watching.
+        Both the sync and async publish paths append before any no-subscriber
+        fast path runs, so publishing to a bus with nothing subscribed still
+        shows up here.
 
         Args:
             event_type: Optional filter by event type
             limit: Maximum number of events to return
 
         Returns:
-            List of recent events published while subscribers existed
+            List of recent events published (subscribed or not)
         """
         with self._lock:
             events = self._event_history.copy()

src/praisonai-agents/praisonaiagents/checkpoints/service.py

@@ -83,7 +83,8 @@ class CheckpointService:
         auto_checkpoint: bool = True,
         max_checkpoints: int = 100,
         user_name: str = "PraisonAI Checkpoints",
-        user_email: str = "[email protected]"
+        user_email: str = "[email protected]",
+        max_file_size: int = 2 * 1024 * 1024,
     ):
         """
         Initialize the checkpoint service.
@@ -96,6 +97,8 @@ class CheckpointService:
             max_checkpoints: Maximum checkpoints to keep
             user_name: Git user.name for commits (default: "PraisonAI Checkpoints")
             user_email: Git user.email for commits (default: "[email protected]")
+            max_file_size: Skip staging files larger than this many bytes
+                (default 2 MB); 0 disables the cap.
         """
         self.config = CheckpointConfig(
             workspace_dir=workspace_dir,
@@ -104,10 +107,14 @@ class CheckpointService:
             auto_checkpoint=auto_checkpoint,
             max_checkpoints=max_checkpoints,
             user_name=user_name,
-            user_email=user_email
+            user_email=user_email,
+            max_file_size=max_file_size,
         )
         
         self._initialized = False
+        # Save counter used to run `git gc --auto` on a bounded cadence so the
+        # shadow object store is periodically pruned during long sessions.
+        self._saves_since_gc = 0
         self._checkpoints: List[Checkpoint] = []
         self._event_handlers: Dict[CheckpointEvent, List[Callable]] = {
             event: [] for event in CheckpointEvent
@@ -184,6 +191,78 @@ class CheckpointService:
         
         return True
     
+    async def _get_project_objects_dir(self) -> Optional[str]:
+        """
+        Return the object database directory of the project's real git repo if
+        the workspace is inside one, else None.
+
+        Uses ``--git-common-dir`` so worktrees resolve to the shared object
+        store. Best-effort: any failure (no git, not a repo) returns None so the
+        shadow repo falls back to the standalone path unchanged.
+        """
+        try:
+            env = self._get_sanitized_env()
+            for flag in ("--git-common-dir", "--git-dir"):
+                process = await asyncio.create_subprocess_exec(
+                    "git", "rev-parse", flag,
+                    stdout=asyncio.subprocess.PIPE,
+                    stderr=asyncio.subprocess.PIPE,
+                    env=env,
+                    cwd=self.config.workspace_dir,
+                )
+                stdout, _ = await asyncio.wait_for(process.communicate(), timeout=30)
+                if process.returncode != 0:
+                    continue
+                git_dir = stdout.decode().strip()
+                if not git_dir:
+                    continue
+                if not os.path.isabs(git_dir):
+                    git_dir = os.path.join(self.config.workspace_dir, git_dir)
+                objects_dir = os.path.join(git_dir, "objects")
+                if os.path.isdir(objects_dir):
+                    return os.path.realpath(objects_dir)
+        except Exception as e:
+            logger.debug(f"Could not resolve project git objects dir: {e}")
+        return None
+
+    async def _apply_large_repo_config(self) -> None:
+        """
+        Apply large-repo git settings so index writes and untracked scans stay
+        fast. Each setting is best-effort: older git versions may not support
+        every key, and a failure must never break checkpointing.
+        """
+        settings = [
+            ("feature.manyFiles", "true"),
+            ("index.version", "4"),
+            ("core.untrackedCache", "true"),
+            ("core.fsmonitor", "true"),
+        ]
+        for key, value in settings:
+            try:
+                await self._run_git("config", key, value)
+            except Exception as e:
+                logger.debug(f"Skipping large-repo config {key}={value}: {e}")
+
+    async def _seed_from_project_objects(self, objects_dir: str) -> bool:
+        """
+        Point the shadow object store at the project's objects via a file-based
+        ``objects/info/alternates`` so new blobs resolve against existing
+        objects instead of being re-hashed and re-stored.
+
+        A file is used (not GIT_ALTERNATE_OBJECT_DIRECTORIES) because
+        ``_get_sanitized_env`` strips that env var. Returns True on success.
+        """
+        try:
+            info_dir = os.path.join(self.git_dir, "objects", "info")
+            os.makedirs(info_dir, exist_ok=True)
+            with open(os.path.join(info_dir, "alternates"), "w") as f:
+                f.write(f"{objects_dir}\n")
+            logger.debug(f"Seeded shadow git objects from {objects_dir}")
+            return True
+        except Exception as e:
+            logger.debug(f"Could not seed shadow git from project objects: {e}")
+            return False
+
     async def _init_shadow_git(self) -> bool:
         """Initialize the shadow git repository."""
         try:
@@ -196,7 +275,17 @@ class CheckpointService:
             
             # Set worktree to workspace
             await self._run_git("config", "core.worktree", self.config.workspace_dir)
-            
+
+            # Seed objects from the project's real repo (if any) so the first
+            # checkpoint only stores changed files, and tune for large repos.
+            # Both are best-effort: no real repo → unchanged standalone path.
+            objects_dir = await self._get_project_objects_dir()
+            if objects_dir and objects_dir != os.path.realpath(
+                os.path.join(self.git_dir, "objects")
+            ):
+                await self._seed_from_project_objects(objects_dir)
+            await self._apply_large_repo_config()
+
             # Create exclude file
             exclude_dir = os.path.join(self.git_dir, "info")
             os.makedirs(exclude_dir, exist_ok=True)
@@ -259,6 +348,64 @@ class CheckpointService:
             env.pop(var, None)
         
         return env
+
+    async def _unstage_oversized_files(self) -> None:
+        """
+        Remove files larger than ``config.max_file_size`` from the index so they
+        are not committed into the shadow store. Best-effort: any failure leaves
+        staging exactly as ``git add -A`` produced it and never breaks the turn.
+        """
+        max_size = self.config.max_file_size
+        if not max_size or max_size <= 0:
+            return
+        try:
+            # NUL-delimited so paths with spaces/newlines are handled safely.
+            staged = await self._run_git("diff", "--cached", "--name-only", "-z")
+            for path in staged.split("\0"):
+                if not path:
+                    continue
+                abs_path = os.path.join(self.config.workspace_dir, path)
+                try:
+                    # Use lstat so symlinks are measured by the link itself, not
+                    # their target: a small link to a large file must stay in the
+                    # checkpoint. Skip anything that isn't a regular file.
+                    if os.path.islink(abs_path) or not os.path.isfile(abs_path):
+                        continue
+                    if os.lstat(abs_path).st_size > max_size:
+                        # Keep the file on disk; only drop it from the index.
+                        await self._run_git("rm", "--cached", "--quiet", "--", path)
+                        logger.debug(f"Skipped oversized file from checkpoint: {path}")
+                except Exception as e:
+                    logger.debug(f"Could not evaluate size of {path}: {e}")
+        except Exception as e:
+            logger.debug(f"Oversized-file skip pass failed (continuing): {e}")
+
+    async def _maybe_gc(self, cadence: int = 20) -> None:
+        """
+        Every ``cadence`` saves, pack the shadow repo and make it self-contained.
+
+        We seed the shadow store from the project's objects via ``alternates``
+        for a fast first checkpoint, but a borrowed object can disappear if the
+        project repo later prunes or repacks it — which would make checkpoint
+        history unreadable. To keep ``/undo``/``/revert`` durable we periodically
+        run ``git repack -a -d`` (without ``--local``) so every reachable object,
+        including borrowed ones, is copied into a local pack the shadow store
+        owns. ``git gc --auto`` still handles pruning/packing of loose objects.
+
+        Best-effort: failures are swallowed so a gc issue never breaks a turn.
+        """
+        self._saves_since_gc += 1
+        if self._saves_since_gc < cadence:
+            return
+        self._saves_since_gc = 0
+        try:
+            # -a: pack all reachable objects; -d: drop redundant packs. Omitting
+            # --local pulls borrowed (alternate) objects into the local pack so
+            # the shadow store no longer depends on the project object database.
+            await self._run_git("repack", "-a", "-d", "--quiet")
+            await self._run_git("gc", "--auto", "--quiet")
+        except Exception as e:
+            logger.debug(f"Shadow-store gc skipped: {e}")
     
     async def save(
         self,
@@ -295,7 +442,13 @@ class CheckpointService:
         try:
             # Stage all changes
             await self._run_git("add", "-A")
-            
+
+            # Drop oversized files from the index so the shadow store never
+            # copies giant blobs (build outputs, model weights, media). This
+            # mirrors the existing "never fail the turn" contract: any error
+            # here is swallowed and staging proceeds as before.
+            await self._unstage_oversized_files()
+
             # Check if there are changes
             try:
                 await self._run_git("diff", "--cached", "--quiet")
@@ -329,7 +482,12 @@ class CheckpointService:
             
             # Prune old checkpoints if needed
             await self._prune_checkpoints()
-            
+
+            # Periodically let git pack loose objects and drop unreachable ones
+            # so the shadow store doesn't leak disk over a long session. Bounded
+            # cadence keeps the per-turn cost near zero; best-effort only.
+            await self._maybe_gc()
+
             self._emit(CheckpointEvent.CHECKPOINT_CREATED, checkpoint)
             logger.info(f"Created checkpoint: {checkpoint.short_id} - {message}")
             

src/praisonai-agents/praisonaiagents/checkpoints/types.py

@@ -40,6 +40,11 @@ class CheckpointConfig:
     max_checkpoints: int = 100  # Maximum checkpoints to keep
     user_name: str = field(default_factory=lambda: os.getenv("PRAISONAI_GIT_USER_NAME", "PraisonAI Checkpoints"))
     user_email: str = field(default_factory=lambda: os.getenv("PRAISONAI_GIT_USER_EMAIL", "[email protected]"))
+    # Skip staging files larger than this (bytes); 0 disables the cap. Kept
+    # after the identity fields so existing positional construction of
+    # CheckpointConfig(workspace, storage, enabled, auto, max_checkpoints,
+    # user_name, user_email) is unaffected.
+    max_file_size: int = 2 * 1024 * 1024
     exclude_patterns: List[str] = field(default_factory=lambda: [
         ".git",
         ".praisonai",

src/praisonai-agents/praisonaiagents/compaction/compactor.py

@@ -6,6 +6,7 @@ Manages context window by compacting messages when needed.
 
 import re
 import threading
+import logging
 from typing import List, Dict, Any, Optional, Callable, Awaitable, Tuple
 import asyncio
 
@@ -14,6 +15,8 @@ from .strategy import CompactionStrategy
 from .result import CompactionResult
 from .protocols import ToolResultPrunerProtocol, MessageFormatterProtocol, SummaryBuilderProtocol
 
+logger = logging.getLogger(__name__)
+
 
 # Cache for the one-time offline-safe tiktoken probe (see estimate_tokens).
 # None = not yet probed; True/False = probe result. Guarded by a lock so a
@@ -337,18 +340,28 @@ class ContextCompactor:
             compacted = self._prune(processed_messages)
         elif self.strategy == CompactionStrategy.LLM_SUMMARIZE:
             if self.llm_summarize_fn:
-                # For sync calls with LLM function, we need to run async
+                # Route through the canonical sync-to-async bridge so the
+                # user-configured LLM summariser runs identically whether or not
+                # a loop is already running (e.g. an agent hosted in FastAPI,
+                # aiohttp, Jupyter or a bot handler). The naive fallback is kept
+                # only for genuine failures, and is now logged so a silent
+                # degradation in context quality is visible.
+                #
+                # Keep the bridge's bounded default timeout rather than
+                # ``timeout=None`` so a summariser that never completes (a hung
+                # network call that ignores cancellation) falls back to the
+                # naive summary below instead of pinning the caller's thread
+                # indefinitely. ``TimeoutError`` is caught by the ``except`` here.
+                from ..utils.async_bridge import run_coroutine_from_any_context
                 try:
-                    # Check if we're already in an async context
-                    try:
-                        loop = asyncio.get_running_loop()
-                        # If in async context, fallback to naive summarization
-                        compacted = self._summarize(processed_messages)
-                    except RuntimeError:
-                        # No running loop, safe to create one
-                        compacted = asyncio.run(self._llm_summarize_async(processed_messages, focus_topic))
+                    compacted = run_coroutine_from_any_context(
+                        self._llm_summarize_async(processed_messages, focus_topic),
+                    )
                 except Exception:
-                    # Fallback to naive summarization if async fails
+                    logger.warning(
+                        "LLM summarisation failed; using naive summary",
+                        exc_info=True,
+                    )
                     compacted = self._summarize(processed_messages)
             else:
                 compacted = self._llm_summarize(processed_messages, focus_topic)

src/praisonai-agents/praisonaiagents/config/feature_configs.py

@@ -1395,13 +1395,21 @@ class HooksConfig:
     
     # Middleware list
     middleware: List[Any] = field(default_factory=list)
-    
+
+    # Optional per-agent HookRegistry for event-based hooks (BEFORE_TOOL,
+    # AFTER_AGENT, ...). When None the agent uses the process-wide default
+    # registry (so plugins bridged into it still fire). Set this to a private
+    # ``HookRegistry()`` to keep event hooks registered for THIS agent from
+    # firing on unrelated agents that share the default registry.
+    registry: Optional[Any] = None
+
     def to_dict(self) -> Dict[str, Any]:
         """Convert to dictionary."""
         return {
             "on_step": str(self.on_step) if self.on_step else None,
             "on_tool_call": str(self.on_tool_call) if self.on_tool_call else None,
             "middleware": [str(m) for m in self.middleware],
+            "registry": type(self.registry).__name__ if self.registry is not None else None,
         }
 
 

src/praisonai-agents/praisonaiagents/decisions/__init__.py

@@ -0,0 +1,39 @@
+"""Jev / System One typed decision models (Ollama ``/v1/systemone``, TypeSafe API)."""
+
+from .routing import (
+    DecisionRoutePlan,
+    DecisionTriageRouter,
+    default_ticket_triage_questions,
+    triage_decision,
+    triaged_start,
+)
+from .system_one import (
+    KNOWN_DECISION_MODELS,
+    SystemOneResult,
+    asystem_one,
+    choice_question,
+    is_decision_model,
+    noul_question,
+    resolve_system_one_base_url,
+    score_question,
+    system_one,
+    system_one_via_typesafe_sdk,
+)
+
+__all__ = [
+    "DecisionRoutePlan",
+    "DecisionTriageRouter",
+    "default_ticket_triage_questions",
+    "triage_decision",
+    "triaged_start",
+    "KNOWN_DECISION_MODELS",
+    "SystemOneResult",
+    "asystem_one",
+    "choice_question",
+    "is_decision_model",
+    "noul_question",
+    "resolve_system_one_base_url",
+    "score_question",
+    "system_one",
+    "system_one_via_typesafe_sdk",
+]

src/praisonai-agents/praisonaiagents/decisions/routing.py

@@ -0,0 +1,168 @@
+"""Route user input through System One decision models before chat agents."""
+
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass, field
+from typing import Any, Callable, Mapping, Optional, Protocol, Union
+
+from .system_one import (
+    SystemOneResult,
+    choice_question,
+    system_one,
+)
+
+
+class AgentLike(Protocol):
+    def start(self, prompt: str, **kwargs: Any) -> Any: ...
+
+
+RouteTarget = Union[AgentLike, str, Callable[[SystemOneResult], Any]]
+
+
+@dataclass
+class DecisionRoutePlan:
+    """Configuration for decision-first routing."""
+
+    route_question: str = "team"
+    decision_model: str = field(
+        default_factory=lambda: os.environ.get("OLLAMA_DECISION_MODEL", "nimble")
+    )
+    api_base: Optional[str] = None
+    state_key: str = "message"
+    min_confidence: Optional[float] = None
+    questions: Mapping[str, Mapping[str, Any]] = field(default_factory=dict)
+    routes: Mapping[str, RouteTarget] = field(default_factory=dict)
+    fallback_route: Optional[str] = None
+
+    def __post_init__(self) -> None:
+        if not self.questions:
+            raise ValueError("DecisionRoutePlan requires a non-empty questions mapping")
+        if self.route_question not in self.questions:
+            raise ValueError(
+                f"route_question {self.route_question!r} missing from questions keys "
+                f"{list(self.questions.keys())}"
+            )
+        route_q = self.questions[self.route_question]
+        if route_q.get("type") != "choice":
+            raise ValueError(
+                f"route_question {self.route_question!r} must be a choice question, "
+                f"got type {route_q.get('type')!r}"
+            )
+
+
+def default_ticket_triage_questions() -> dict[str, dict[str, Any]]:
+    """Preset for support ticket routing (Ollama blog example)."""
+    return {
+        "team": choice_question(
+            "Which team should handle this ticket?",
+            {
+                "billing": "Payments and refunds",
+                "technical": "Bugs and integrations",
+                "other": "None of the above",
+            },
+        ),
+        "refund": {
+            "type": "noul",
+            "instructions": "Does the customer explicitly ask for a refund?",
+        },
+        "urgency": {
+            "type": "score",
+            "instructions": "How urgent is this ticket?",
+            "criteria": ["Routine", "Soon", "Urgent"],
+        },
+    }
+
+
+def triage_decision(
+    message: str,
+    *,
+    questions: Mapping[str, Mapping[str, Any]],
+    decision_model: str = "nimble",
+    api_base: Optional[str] = None,
+    state_key: str = "message",
+    extra_state: Optional[Mapping[str, Any]] = None,
+) -> SystemOneResult:
+    """Run System One on *message* without invoking a chat LLM."""
+    state = dict(extra_state or {})
+    state[state_key] = message
+    return system_one(
+        state=state,
+        questions=questions,
+        model=decision_model,
+        api_base=api_base,
+    )
+
+
+def resolve_route_key(result: SystemOneResult, route_question: str) -> Optional[str]:
+    return result.choice(route_question)
+
+
+def _invoke_target(target: RouteTarget, prompt: str, decision: SystemOneResult) -> Any:
+    if isinstance(target, str):
+        from praisonaiagents import Agent
+
+        return Agent(instructions="You are a helpful assistant.", llm=target).start(prompt)
+    if callable(target):
+        return target(decision)
+    return target.start(prompt)
+
+
+def triaged_start(
+    prompt: str,
+    plan: DecisionRoutePlan,
+    *,
+    extra_state: Optional[Mapping[str, Any]] = None,
+) -> dict[str, Any]:
+    """
+    Triage with a decision model, then delegate to the mapped chat agent/model.
+
+    Returns a dict with ``decision``, ``route``, ``confidence``, and ``response``.
+    """
+    decision = triage_decision(
+        prompt,
+        questions=plan.questions,
+        decision_model=plan.decision_model,
+        api_base=plan.api_base,
+        state_key=plan.state_key,
+        extra_state=extra_state,
+    )
+    route_key = resolve_route_key(decision, plan.route_question)
+    conf = decision.confidence(plan.route_question)
+    if plan.min_confidence is not None and (conf is None or conf < plan.min_confidence):
+        route_key = plan.fallback_route or route_key
+    if not route_key or route_key not in plan.routes:
+        route_key = plan.fallback_route or next(iter(plan.routes), None)
+    if not route_key or route_key not in plan.routes:
+        raise RuntimeError(
+            f"No route for decision choice {route_key!r}; "
+            f"available routes: {list(plan.routes.keys())}"
+        )
+    target = plan.routes[route_key]
+    response = _invoke_target(target, prompt, decision)
+    return {
+        "decision": decision,
+        "route": route_key,
+        "confidence": conf,
+        "response": response,
+    }
+
+
+class DecisionTriageRouter:
+    """Object-oriented wrapper for repeated triage + chat routing."""
+
+    def __init__(self, plan: DecisionRoutePlan):
+        self.plan = plan
+
+    def decide(self, message: str, **extra_state: Any) -> SystemOneResult:
+        return triage_decision(
+            message,
+            questions=self.plan.questions,
+            decision_model=self.plan.decision_model,
+            api_base=self.plan.api_base,
+            state_key=self.plan.state_key,
+            extra_state=extra_state or None,
+        )
+
+    def start(self, prompt: str, **extra_state: Any) -> dict[str, Any]:
+        return triaged_start(prompt, self.plan, extra_state=extra_state or None)

src/praisonai-agents/praisonaiagents/decisions/system_one.py

@@ -0,0 +1,289 @@
+"""Call Ollama / TypeSafe System One decision endpoint (``POST /v1/systemone``).
+
+See: https://ollama.com/blog/ollama-now-supports-jev-style-decision-models
+"""
+
+from __future__ import annotations
+
+import ipaddress
+import json
+import os
+import urllib.error
+import urllib.request
+from dataclasses import dataclass, field
+from typing import Any, Mapping, MutableMapping, Optional
+from urllib.parse import urlparse
+
+KNOWN_DECISION_MODELS = frozenset({"nimble", "tev1", "tev1:0.8b"})
+
+
+def is_decision_model(model: str) -> bool:
+    """Return True if *model* looks like a known Ollama decision model id."""
+    name = (model or "").strip().split("/")[-1]
+    if name in KNOWN_DECISION_MODELS:
+        return True
+    return name.startswith("tev1")
+
+
+def _base_source(api_base: Optional[str] = None) -> tuple[str, Optional[str]]:
+    """Return ``(base_url, source)`` where *source* names the env var selected.
+
+    *source* is ``None`` when ``api_base`` is explicit, ``"TYPESAFE_BASE_URL"`` or
+    ``"OLLAMA_HOST"`` when resolved from the environment, or ``"default"`` for the
+    loopback fallback. The source lets callers scope credentials to the host that
+    was actually chosen (so a TypeSafe key is never sent to an Ollama host).
+    """
+    if api_base:
+        return api_base.rstrip("/"), None
+    for key in ("TYPESAFE_BASE_URL", "OLLAMA_HOST"):
+        raw = (os.environ.get(key) or "").strip()
+        if raw:
+            from praisonaiagents.local.target import parse_ollama_host
+
+            parsed = parse_ollama_host(raw) if not raw.startswith("http") else raw.rstrip("/")
+            return (parsed or raw).rstrip("/"), key
+    return "http://127.0.0.1:11434", "default"
+
+
+def resolve_system_one_base_url(api_base: Optional[str] = None) -> str:
+    """Resolve base URL for ``/v1/systemone`` (Ollama 0.35+ or TypeSafe cloud)."""
+    return _base_source(api_base)[0]
+
+
+def _is_loopback_host(host: Optional[str]) -> bool:
+    if not host:
+        return False
+    if host.lower() == "localhost":
+        return True
+    try:
+        return ipaddress.ip_address(host).is_loopback
+    except ValueError:
+        return False
+
+
+def _validate_base_url(base: str) -> None:
+    """Reject non-HTTP(S) schemes and plain HTTP to non-loopback hosts."""
+    parsed = urlparse(base)
+    if parsed.scheme not in {"http", "https"}:
+        raise ValueError(f"system_one requires an http(s) base URL, got {base!r}")
+    if parsed.scheme == "http" and not _is_loopback_host(parsed.hostname):
+        raise ValueError(
+            f"system_one refuses plain HTTP for non-loopback host {parsed.hostname!r}; use https"
+        )
+
+
+def choice_question(instructions: str, criteria: Mapping[str, str]) -> dict[str, Any]:
+    return {"type": "choice", "instructions": instructions, "criteria": dict(criteria)}
+
+
+def noul_question(instructions: str) -> dict[str, Any]:
+    return {"type": "noul", "instructions": instructions}
+
+
+def score_question(instructions: str, criteria: list[str]) -> dict[str, Any]:
+    return {"type": "score", "instructions": instructions, "criteria": list(criteria)}
+
+
+@dataclass
+class SystemOneResult:
+    """Parsed ``/v1/systemone`` response."""
+
+    model: str
+    answers: dict[str, Any]
+    usage: dict[str, Any] = field(default_factory=dict)
+    raw: dict[str, Any] = field(default_factory=dict)
+
+    def choice(self, name: str) -> Optional[str]:
+        entry = self.answers.get(name) or {}
+        if entry.get("type") == "choice":
+            return entry.get("choice")
+        return None
+
+    def noul(self, name: str) -> Optional[float]:
+        entry = self.answers.get(name) or {}
+        if entry.get("type") == "noul":
+            val = entry.get("noul")
+            return float(val) if val is not None else None
+        return None
+
+    def score(self, name: str) -> Optional[float]:
+        entry = self.answers.get(name) or {}
+        if entry.get("type") == "score":
+            val = entry.get("score")
+            return float(val) if val is not None else None
+        return None
+
+    def confidence(self, name: str) -> Optional[float]:
+        entry = self.answers.get(name) or {}
+        val = entry.get("confidence")
+        return float(val) if val is not None else None
+
+
+def _default_timeout(explicit: Optional[float]) -> float:
+    if explicit is not None:
+        return explicit
+    raw = (os.environ.get("SYSTEM_ONE_TIMEOUT") or os.environ.get("OLLAMA_SYSTEM_ONE_TIMEOUT") or "").strip()
+    if raw:
+        try:
+            return float(raw)
+        except ValueError:
+            pass
+    return 120.0
+
+
+def _default_api_key(api_key: Optional[str], *, allow_typesafe_key: bool) -> Optional[str]:
+    """Resolve the bearer key, scoping ``TYPESAFE_API_KEY`` to TypeSafe hosts only."""
+    if api_key is not None:
+        return api_key
+    if allow_typesafe_key and os.environ.get("TYPESAFE_API_KEY"):
+        return os.environ["TYPESAFE_API_KEY"]
+    return os.environ.get("OLLAMA_API_KEY") or "ollama"
+
+
+def _build_payload(
+    *,
+    model: str,
+    state: Mapping[str, Any],
+    questions: Mapping[str, Mapping[str, Any]],
+) -> dict[str, Any]:
+    return {
+        "model": model,
+        "state": dict(state),
+        "questions": {k: dict(v) for k, v in questions.items()},
+    }
+
+
+def _parse_response(data: dict[str, Any]) -> SystemOneResult:
+    return SystemOneResult(
+        model=str(data.get("model") or ""),
+        answers=dict(data.get("answers") or {}),
+        usage=dict(data.get("usage") or {}),
+        raw=data,
+    )
+
+
+def system_one(
+    *,
+    state: Mapping[str, Any],
+    questions: Mapping[str, Mapping[str, Any]],
+    model: Optional[str] = None,
+    api_base: Optional[str] = None,
+    api_key: Optional[str] = None,
+    timeout: Optional[float] = None,
+) -> SystemOneResult:
+    """
+    Run a System One decision request against Ollama or TypeSafe.
+
+    Example::
+
+        from praisonaiagents.decisions import system_one, choice_question, noul_question
+
+        result = system_one(
+            state={"ticket": "Please refund my duplicate charge."},
+            questions={
+                "team": choice_question(
+                    "Which team?",
+                    {"billing": "Payments", "technical": "Bugs", "other": "Other"},
+                ),
+                "refund": noul_question("Does the user ask for a refund?"),
+            },
+            model="nimble",
+        )
+        assert result.choice("team") == "billing"
+    """
+    resolved_model = (
+        model
+        or os.environ.get("TYPESAFE_DEFAULT_MODEL")
+        or os.environ.get("OLLAMA_DECISION_MODEL")
+        or "nimble"
+    )
+    base, source = _base_source(api_base)
+    _validate_base_url(base)
+    url = f"{base}/v1/systemone"
+    body = json.dumps(_build_payload(model=resolved_model, state=state, questions=questions)).encode("utf-8")
+    headers = {"Content-Type": "application/json", "Accept": "application/json"}
+    key = _default_api_key(api_key, allow_typesafe_key=source == "TYPESAFE_BASE_URL")
+    if key:
+        headers["Authorization"] = f"Bearer {key}"
+    timeout = _default_timeout(timeout)
+    req = urllib.request.Request(url, data=body, headers=headers, method="POST")
+    try:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            data = json.loads(resp.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        detail = exc.read().decode("utf-8", errors="replace")[:2000]
+        raise RuntimeError(f"system_one HTTP {exc.code}: {detail}") from exc
+    except urllib.error.URLError as exc:
+        raise RuntimeError(
+            f"system_one failed to reach {url!r}. Is Ollama 0.35+ running with a decision model pulled?"
+        ) from exc
+    if not isinstance(data, dict):
+        raise RuntimeError(f"system_one expected JSON object, got {type(data).__name__}")
+    return _parse_response(data)
+
+
+async def asystem_one(
+    *,
+    state: Mapping[str, Any],
+    questions: Mapping[str, Mapping[str, Any]],
+    model: Optional[str] = None,
+    api_base: Optional[str] = None,
+    api_key: Optional[str] = None,
+    timeout: Optional[float] = None,
+) -> SystemOneResult:
+    """Async ``system_one`` using aiohttp (already a core dependency)."""
+    import aiohttp
+
+    resolved_model = (
+        model
+        or os.environ.get("TYPESAFE_DEFAULT_MODEL")
+        or os.environ.get("OLLAMA_DECISION_MODEL")
+        or "nimble"
+    )
+    base, source = _base_source(api_base)
+    _validate_base_url(base)
+    url = f"{base}/v1/systemone"
+    payload = _build_payload(model=resolved_model, state=state, questions=questions)
+    headers: MutableMapping[str, str] = {"Content-Type": "application/json", "Accept": "application/json"}
+    key = _default_api_key(api_key, allow_typesafe_key=source == "TYPESAFE_BASE_URL")
+    if key:
+        headers["Authorization"] = f"Bearer {key}"
+    timeout_obj = aiohttp.ClientTimeout(total=_default_timeout(timeout))
+    async with aiohttp.ClientSession(timeout=timeout_obj) as session:
+        async with session.post(url, json=payload, headers=headers) as resp:
+            text = await resp.text()
+            if resp.status >= 400:
+                raise RuntimeError(f"system_one HTTP {resp.status}: {text[:2000]}")
+            data = json.loads(text)
+    if not isinstance(data, dict):
+        raise RuntimeError(f"system_one expected JSON object, got {type(data).__name__}")
+    return _parse_response(data)
+
+
+def system_one_via_typesafe_sdk(
+    *,
+    state: Mapping[str, Any],
+    questions: Mapping[str, Any],
+    model: Optional[str] = None,
+    timeout: float = 120.0,
+) -> SystemOneResult:
+    """Optional path through the official ``typesafe-sdk`` (``pip install typesafe-sdk``)."""
+    try:
+        from typesafe_sdk import TypeSafeClient
+    except ImportError as exc:
+        raise ImportError(
+            "typesafe-sdk is not installed. Use: pip install typesafe-sdk "
+            "or call system_one() which uses HTTP directly."
+        ) from exc
+
+    with TypeSafeClient(timeout=timeout) as client:
+        result = client.system_one(state=dict(state), questions=dict(questions), model=model)
+    answers: dict[str, Any] = {}
+    for key, val in getattr(result, "answers", {}).items():
+        answers[key] = val if isinstance(val, dict) else getattr(val, "__dict__", {"value": val})
+    return SystemOneResult(
+        model=str(getattr(result, "model", model or "")),
+        answers=answers,
+        usage=dict(getattr(result, "usage", {}) or {}),
+        raw={"via": "typesafe-sdk"},
+    )

src/praisonai-agents/praisonaiagents/eval/evalport.py

@@ -7,9 +7,16 @@ another tool's harness, and results can be diffed/aggregated across tools.
 
 Mapping (1:1 with EvalPort's ``Suite`` / ``ResultSet`` concepts):
 
-    EvalPackage <-> EvalPort Suite      (name/description/version + cases + thresholds)
-    EvalCase    <-> EvalPort test case  (input / expected_output / grader refs)
-    EvalReport   -> EvalPort ResultSet  (aggregate + per-case pass/score/latency)
+    EvalPackage <-> EvalPort Suite      (id/name/description + test_cases + graders)
+    EvalCase    <-> EvalPort TestCase   (id / input / expected_output / grader refs)
+    EvalReport   -> EvalPort ResultSet  (summary + per-case grader_results)
+
+Per-criterion scores map to native EvalPort ``grader_results`` (one per criterion,
+matching the criterion graders referenced by the Suite). PraisonAI-only fields
+with no EvalPort slot (package version, thresholds, seed, trajectory records, a
+nonpositive ``timeout_seconds`` that the spec's ``timeout_ms`` cannot represent,
+...) are kept under the ``praisonai`` key of the relevant ``metadata`` object so
+round trips stay lossless.
 
 Design: this module is intentionally **dependency-free**. It emits and consumes
 plain spec-shaped dicts (JSON), so the core SDK does not take on ``evalport-sdk``
@@ -27,53 +34,90 @@ Example:
     >>> pkg2.name == pkg.name
     True
 """
-from typing import Any, Dict, List
+import uuid
+from datetime import datetime, timezone
+from typing import Any, Dict, List, Optional
 
 from .package import EvalCase, EvalPackage, EvalReport
 
-EVALPORT_SPEC_VERSION = "1.0"
+# EvalPort spec version the emitted documents conform to (semver, per spec).
+EVALPORT_SPEC_VERSION = "1.0.0-rc.5"
+
+# Grader that carries PraisonAI's own per-case verdict (EvalResult.passed/score).
+# EvalPort requires every test case to reference at least one grader.
+VERDICT_GRADER_ID = "praisonai_verdict"
+_HANDLER = "praisonaiagents.eval"
+_META_KEY = "praisonai"
+
+
+def _grader_def(grader_id: str, description: str) -> Dict[str, Any]:
+    return {
+        "id": grader_id,
+        "type": "custom",
+        "params": {"handler": _HANDLER},
+        "description": description,
+    }
 
 
 def _case_to_evalport(case: EvalCase) -> Dict[str, Any]:
-    """Map an ``EvalCase`` to an EvalPort test-case dict.
+    """Map an ``EvalCase`` to an EvalPort TestCase dict.
 
-    The native ``timeout_seconds`` is emitted as a dedicated top-level field so
-    it never collides with a user-supplied ``metadata["timeout_seconds"]`` key,
-    keeping round trips lossless for both.
+    ``criteria`` become grader references (defined at suite level); a case
+    without criteria references the verdict grader. ``timeout_seconds`` maps to
+    the spec's ``timeout_ms``; user ``metadata`` is passed through untouched.
     """
+    # A case's own criteria are its graders; the verdict grader is always
+    # referenced too so PraisonAI's pass/score verdict has a home in the
+    # ResultSet (spec requires >= 1 grader per case).
+    graders = [VERDICT_GRADER_ID] + [c for c in case.criteria if c != VERDICT_GRADER_ID]
     item: Dict[str, Any] = {
         "id": case.name,
         "input": case.input,
+        "graders": list(dict.fromkeys(graders)),
     }
     if case.expected is not None:
         item["expected_output"] = case.expected
-    if case.criteria:
-        item["graders"] = list(case.criteria)
-    if case.timeout_seconds is not None:
-        item["timeout_seconds"] = case.timeout_seconds
-    metadata = dict(case.metadata or {})
-    if metadata:
-        item["metadata"] = metadata
+    if case.metadata:
+        item["metadata"] = dict(case.metadata)
+    if case.timeout_seconds is not None and case.timeout_seconds > 0:
+        item["timeout_ms"] = max(1, int(round(case.timeout_seconds * 1000)))
+    elif case.timeout_seconds is not None:
+        # Nonpositive timeout is not spec-representable (timeout_ms >= 1); keep
+        # the raw value under the reserved namespace so import restores it
+        # instead of silently applying the 30s default.
+        meta = item.setdefault("metadata", {})
+        native = dict(meta.get(_META_KEY) or {})
+        native["timeout_seconds"] = case.timeout_seconds
+        meta[_META_KEY] = native
     return item
 
 
 def _case_from_evalport(item: Dict[str, Any]) -> EvalCase:
-    """Map an EvalPort test-case dict to an ``EvalCase``.
+    """Map an EvalPort TestCase dict to an ``EvalCase``.
 
-    Prefers the dedicated top-level ``timeout_seconds`` field; falls back to a
-    ``metadata["timeout_seconds"]`` value for suites emitted by other tools,
-    without mutating the caller's metadata.
+    Reads ``timeout_ms`` (spec); falls back to a legacy top-level or
+    ``metadata["timeout_seconds"]`` value, without mutating caller metadata.
     """
     metadata = dict(item.get("metadata") or {})
-    if "timeout_seconds" in item:
+    native = metadata.get(_META_KEY) or {}
+    if "timeout_ms" in item:
+        timeout = item["timeout_ms"] / 1000.0
+    elif "timeout_seconds" in item:
         timeout = item["timeout_seconds"]
+    elif "timeout_seconds" in native:
+        timeout = native["timeout_seconds"]  # nonpositive timeout preserved on export
     else:
         timeout = metadata.get("timeout_seconds", 30.0)
+    criteria = []
+    for g in item.get("graders") or item.get("criteria") or []:
+        gid = g.get("id") if isinstance(g, dict) else g
+        if gid and gid != VERDICT_GRADER_ID:
+            criteria.append(gid)
     return EvalCase(
         name=item.get("id") or item.get("name") or "case",
         input=item.get("input", ""),
         expected=item.get("expected_output", item.get("expected")),
-        criteria=list(item.get("graders") or item.get("criteria") or []),
+        criteria=criteria,
         metadata=metadata,
         timeout_seconds=timeout,
     )
@@ -88,16 +132,28 @@ def to_evalport(package: EvalPackage) -> Dict[str, Any]:
     Returns:
         A spec-shaped dict suitable for ``openeval.validate.validate_suite()``.
     """
-    return {
-        "evalport_version": EVALPORT_SPEC_VERSION,
-        "kind": "suite",
+    criteria = dict.fromkeys(c for case in package.cases for c in case.criteria)
+    criteria.pop(VERDICT_GRADER_ID, None)
+    graders = [_grader_def(VERDICT_GRADER_ID, "PraisonAI per-case verdict (EvalResult.passed/score)")]
+    graders += [_grader_def(c, f"PraisonAI criterion: {c}") for c in criteria]
+
+    suite: Dict[str, Any] = {
+        "version": EVALPORT_SPEC_VERSION,
+        "id": package.name,
         "name": package.name,
-        "description": package.description,
-        "version": package.version,
-        "cases": [_case_to_evalport(c) for c in package.cases],
-        "thresholds": dict(package.thresholds or {}),
-        "seed": package.seed,
+        "graders": graders,
+        "test_cases": [_case_to_evalport(c) for c in package.cases],
+        "metadata": {
+            _META_KEY: {
+                "version": package.version,
+                "thresholds": dict(package.thresholds or {}),
+                "seed": package.seed,
+            }
+        },
     }
+    if package.description:
+        suite["description"] = package.description
+    return suite
 
 
 def from_evalport(suite: Dict[str, Any]) -> EvalPackage:
@@ -110,22 +166,68 @@ def from_evalport(suite: Dict[str, Any]) -> EvalPackage:
         An ``EvalPackage`` whose cases can be run via ``EvalSuite`` /
         ``HarnessEvaluator`` as native ``EvalCase`` objects.
     """
-    cases: List[EvalCase] = [_case_from_evalport(c) for c in suite.get("cases", [])]
+    items = suite.get("test_cases", suite.get("cases", []))
+    cases: List[EvalCase] = [_case_from_evalport(c) for c in items]
+    meta = (suite.get("metadata") or {}).get(_META_KEY) or {}
+    legacy = "evalport_version" in suite  # documents emitted before spec alignment
     return EvalPackage(
-        name=suite.get("name", "suite"),
+        name=suite.get("name") or suite.get("id") or "suite",
         description=suite.get("description", ""),
-        version=suite.get("version", "1.0.0"),
+        version=meta.get("version") or (suite.get("version") if legacy else None) or "1.0.0",
         cases=cases,
-        thresholds=dict(suite.get("thresholds") or {}),
-        seed=suite.get("seed"),
+        thresholds=dict(meta.get("thresholds") or suite.get("thresholds") or {}),
+        seed=meta.get("seed", suite.get("seed")),
     )
 
 
-def report_to_evalport(report: EvalReport) -> Dict[str, Any]:
+def _unit_score(value: Any) -> Optional[float]:
+    if isinstance(value, bool) or not isinstance(value, (int, float)) or value != value:
+        return None
+    return min(1.0, max(0.0, float(value)))
+
+
+def _grader_result(grader_id: str, raw_score: Any, passed: bool) -> Dict[str, Any]:
+    """Build one spec-shaped ``grader_result`` entry.
+
+    Scores are clamped to the spec's [0, 1] range; the raw value is preserved
+    under ``metadata`` when clamping changes it. A null score means "not
+    verified", so ``passed`` is forced ``False`` in that case.
+    """
+    score = _unit_score(raw_score)
+    grader: Dict[str, Any] = {
+        "grader_id": grader_id,
+        "type": "custom",
+        "score": score,
+        "passed": bool(passed) if score is not None else False,
+    }
+    if score is not None and score != raw_score:
+        grader["metadata"] = {"openeval.raw_score": raw_score}
+    return grader
+
+
+def _runner_info() -> Dict[str, str]:
+    runner = {"name": "praisonaiagents"}
+    try:
+        from importlib.metadata import version
+
+        runner["version"] = version("praisonaiagents")
+    except Exception:
+        pass
+    return runner
+
+
+def report_to_evalport(
+    report: EvalReport,
+    *,
+    run_id: Optional[str] = None,
+    started_at: Optional[str] = None,
+) -> Dict[str, Any]:
     """Convert an ``EvalReport`` to an EvalPort ``ResultSet`` dict.
 
     Args:
         report: The aggregated report from running an eval package.
+        run_id: Unique run identifier (defaults to a random UUID).
+        started_at: RFC 3339 run start time (defaults to now, UTC).
 
     Returns:
         A spec-shaped dict suitable for
@@ -133,33 +235,52 @@ def report_to_evalport(report: EvalReport) -> Dict[str, Any]:
     """
     results: List[Dict[str, Any]] = []
     for r in report.results:
+        verdict = _grader_result(VERDICT_GRADER_ID, r.score, r.passed)
+        grader_results = [verdict]
+        # Surface each per-criterion score as its own grader_result so EvalPort
+        # consumers can diff criteria across tools (they match the criterion
+        # grader ids referenced by the exported Suite). ``passed`` is unknown at
+        # the criterion level, so mirror the case verdict.
+        for name, score in (r.criteria_scores or {}).items():
+            if name == VERDICT_GRADER_ID:
+                continue
+            grader_results.append(_grader_result(name, score, r.passed))
+
         item: Dict[str, Any] = {
-            "case_id": r.case_name,
-            "passed": r.passed,
-            "score": r.score,
-            "latency_ms": r.latency_ms,
+            "test_case_id": r.case_name,
+            "passed": verdict["passed"],
+            "grader_results": grader_results,
         }
         if r.actual_output is not None:
-            item["actual_output"] = r.actual_output
+            item["actual_output"] = str(r.actual_output)
+        if r.latency_ms is not None and r.latency_ms >= 0:
+            item["duration_ms"] = int(round(r.latency_ms))
         if r.error is not None:
-            item["error"] = r.error
-        if r.criteria_scores:
-            item["grader_scores"] = dict(r.criteria_scores)
+            item["error"] = {"type": "runner_error", "message": str(r.error)}
         if r.record is not None:
-            item["record"] = r.record
+            item["metadata"] = {_META_KEY: {"record": r.record}}
         results.append(item)
 
+    summary: Dict[str, Any] = {
+        "total": report.total_cases,
+        "passed": report.passed_cases,
+        "failed": report.failed_cases,
+        "pass_rate": report.pass_rate,
+    }
+    meta: Dict[str, Any] = {"thresholds_met": dict(report.thresholds_met or {})}
+    avg = _unit_score(report.average_score)
+    if avg is not None:
+        summary["avg_score"] = avg
+        if avg != report.average_score:
+            meta["average_score"] = report.average_score  # clamped
+
     return {
-        "evalport_version": EVALPORT_SPEC_VERSION,
-        "kind": "result_set",
-        "suite_name": report.package_name,
-        "summary": {
-            "total": report.total_cases,
-            "passed": report.passed_cases,
-            "failed": report.failed_cases,
-            "pass_rate": report.pass_rate,
-            "average_score": report.average_score,
-            "thresholds_met": dict(report.thresholds_met or {}),
-        },
+        "version": EVALPORT_SPEC_VERSION,
+        "suite_id": report.package_name,
+        "run_id": run_id or uuid.uuid4().hex,
+        "started_at": started_at or datetime.now(timezone.utc).isoformat(),
+        "runner": _runner_info(),
         "results": results,
+        "summary": summary,
+        "metadata": {_META_KEY: meta},
     }

src/praisonai-agents/praisonaiagents/gateway/__init__.py

@@ -31,6 +31,18 @@ _LAZY_IMPORTS = {
     "SessionVisibility": ("praisonaiagents.gateway.protocols", "SessionVisibility"),
     "SessionSharingRole": ("praisonaiagents.gateway.protocols", "SessionSharingRole"),
     "SessionObserverProtocol": ("praisonaiagents.gateway.protocols", "SessionObserverProtocol"),
+    # Session-projection reducer (Issue #5324)
+    "SessionProjection": ("praisonaiagents.gateway.session_projection", "SessionProjection"),
+    "SessionProjectionState": ("praisonaiagents.gateway.session_projection", "SessionProjectionState"),
+    "RunView": ("praisonaiagents.gateway.session_projection", "RunView"),
+    # Canonical session-address grammar (Issue #5383)
+    "SessionRef": ("praisonaiagents.gateway.addressing", "SessionRef"),
+    "build_session_path": ("praisonaiagents.gateway.addressing", "build_session_path"),
+    "parse_session_path": ("praisonaiagents.gateway.addressing", "parse_session_path"),
+    "derive_short_id": ("praisonaiagents.gateway.addressing", "derive_short_id"),
+    "slugify": ("praisonaiagents.gateway.addressing", "slugify"),
+    "RESERVED_NAMES": ("praisonaiagents.gateway.addressing", "RESERVED_NAMES"),
+    "SHORT_ID_LEN": ("praisonaiagents.gateway.addressing", "SHORT_ID_LEN"),
     "GatewayCloseCode": ("praisonaiagents.gateway.protocols", "GatewayCloseCode"),
     # Declarative method -> required-scope registry (Issue #3206)
     "GatewayMethodDescriptor": ("praisonaiagents.gateway.protocols", "GatewayMethodDescriptor"),
@@ -45,6 +57,11 @@ _LAZY_IMPORTS = {
     # Saturation / back-pressure observability (Issue #4265)
     "HealthPressure": ("praisonaiagents.gateway.protocols", "HealthPressure"),
     "evaluate_pressure": ("praisonaiagents.gateway.protocols", "evaluate_pressure"),
+    # Trusted-proxy ingress attribution (Issue #5312)
+    "IngressTrust": ("praisonaiagents.gateway.protocols", "IngressTrust"),
+    "IngressAttribution": ("praisonaiagents.gateway.protocols", "IngressAttribution"),
+    "resolve_ingress_attribution": ("praisonaiagents.gateway.protocols", "resolve_ingress_attribution"),
+    "parse_forwarded_for": ("praisonaiagents.gateway.protocols", "parse_forwarded_for"),
     # Push protocols and dataclasses
     "PushChannelProtocol": ("praisonaiagents.gateway.protocols", "PushChannelProtocol"),
     "PresenceProtocol": ("praisonaiagents.gateway.protocols", "PresenceProtocol"),
@@ -73,10 +90,18 @@ _LAZY_IMPORTS = {
     # Agent/gateway-callable thread creation (Issue #3987)
     "ThreadResult": ("praisonaiagents.gateway.protocols", "ThreadResult"),
     "ThreadStatus": ("praisonaiagents.gateway.protocols", "ThreadStatus"),
+    # Agent-callable message mutation: edit/delete (Issue #5054)
+    "MessageActionResult": ("praisonaiagents.gateway.protocols", "MessageActionResult"),
+    "MessageActionStatus": ("praisonaiagents.gateway.protocols", "MessageActionStatus"),
     # Agent-callable cross-conversation request/reply (Issue #3689)
     "ConversationReply": ("praisonaiagents.gateway.protocols", "ConversationReply"),
     "ConversationReplyStatus": ("praisonaiagents.gateway.protocols", "ConversationReplyStatus"),
     "ConversationRequestProtocol": ("praisonaiagents.gateway.protocols", "ConversationRequestProtocol"),
+    # Server→client interactive request/reply over the transport (Issue #5351)
+    "GatewayRequestKind": ("praisonaiagents.gateway.protocols", "GatewayRequestKind"),
+    "GatewayServerRequest": ("praisonaiagents.gateway.protocols", "GatewayServerRequest"),
+    "GatewayServerReply": ("praisonaiagents.gateway.protocols", "GatewayServerReply"),
+    "GatewayRequestChannelProtocol": ("praisonaiagents.gateway.protocols", "GatewayRequestChannelProtocol"),
     # Agent-facing live status/health (Issue #3688)
     "GatewayStatusProtocol": ("praisonaiagents.gateway.protocols", "GatewayStatusProtocol"),
     "GatewayStatus": ("praisonaiagents.gateway.protocols", "GatewayStatus"),
@@ -227,6 +252,8 @@ _LAZY_IMPORTS = {
     # liveness.py — Event-loop liveness watchdog (Issue #3385)
     "LoopWatchdogPolicy": ("praisonaiagents.gateway.liveness", "LoopWatchdogPolicy"),
     "LoopWatchdog": ("praisonaiagents.gateway.liveness", "LoopWatchdog"),
+    # liveness.py — Startup-phase watchdog (Issue #5265)
+    "StartupWatchdog": ("praisonaiagents.gateway.liveness", "StartupWatchdog"),
     # degraded_state.py — Unified degraded-capability registry (Issue #3518)
     "DegradedOwner": ("praisonaiagents.gateway.degraded_state", "DegradedOwner"),
     "DegradedCapabilityProtocol": ("praisonaiagents.gateway.degraded_state", "DegradedCapabilityProtocol"),
@@ -369,6 +396,18 @@ __all__ = [
     "SessionVisibility",
     "SessionSharingRole",
     "SessionObserverProtocol",
+    # Session-projection reducer (Issue #5324)
+    "SessionProjection",
+    "SessionProjectionState",
+    "RunView",
+    # Canonical session-address grammar (Issue #5383)
+    "SessionRef",
+    "build_session_path",
+    "parse_session_path",
+    "derive_short_id",
+    "slugify",
+    "RESERVED_NAMES",
+    "SHORT_ID_LEN",
     "GatewayCloseCode",
     # Declarative method -> required-scope registry (Issue #3206)
     "GatewayMethodDescriptor",
@@ -382,6 +421,11 @@ __all__ = [
     "compute_config_revision",
     "HealthPressure",
     "evaluate_pressure",
+    # Trusted-proxy ingress attribution (Issue #5312)
+    "IngressTrust",
+    "IngressAttribution",
+    "resolve_ingress_attribution",
+    "parse_forwarded_for",
     # Push protocols (always available)
     "PushChannelProtocol",
     "PresenceProtocol",
@@ -405,11 +449,18 @@ __all__ = [
     "ReactionResult",
     "ReactionStatus",
     "ThreadResult",
+    "MessageActionResult",
+    "MessageActionStatus",
     "ThreadStatus",
     # Agent-callable cross-conversation request/reply (Issue #3689)
     "ConversationReply",
     "ConversationReplyStatus",
     "ConversationRequestProtocol",
+    # Server→client interactive request/reply over the transport (Issue #5351)
+    "GatewayRequestKind",
+    "GatewayServerRequest",
+    "GatewayServerReply",
+    "GatewayRequestChannelProtocol",
     # Agent-facing live status/health (Issue #3688)
     "GatewayStatusProtocol",
     "GatewayStatus",
@@ -557,6 +608,8 @@ __all__ = [
     # Event-loop liveness watchdog (Issue #3385)
     "LoopWatchdogPolicy",
     "LoopWatchdog",
+    # Startup-phase watchdog (Issue #5265)
+    "StartupWatchdog",
     # Unified degraded-capability registry (Issue #3518)
     "DegradedOwner",
     "DegradedCapabilityProtocol",

src/praisonai-agents/praisonaiagents/gateway/addressing.py

@@ -0,0 +1,227 @@
+"""Canonical session-address grammar for the Gateway (Issue #5383).
+
+A pure, dependency-free grammar that maps between an opaque runtime
+``session_key`` (a UUID, or an ``agent:<id>:<uuid>``-style key) and a stable,
+human-readable, reconnect-survivable path — e.g. ``chat/<agentId>/<slug>-<shortId>``.
+It gives every surface (dashboard deep-links, bot channels, the CLI, the
+``praisonai-ts`` mirror, reconnecting clients) a single addressing contract to
+agree on, instead of each one carrying the raw UUID or re-inventing a resolver.
+
+This module lives in **core** because the grammar is a set of pure functions
+that define a protocol/contract every surface must share. It has **no**
+third-party dependencies and does no I/O — the concrete store-backed lookup
+(``session.resolve``) is implemented in the wrapper behind this contract.
+
+The grammar is deliberately lossy in one direction: ``build_session_path``
+derives a short id from the trailing UUID so the whole opaque instance id never
+appears in a URL, and an optional slug from the session label. ``parse_session_path``
+recovers a :class:`SessionRef` (agent id + short id + slug hint) that a resolver
+maps back to the concrete ``session_id``; it never assumes it can reconstruct the
+key byte-for-byte, except via the exact-key escape hatch (a ``!``-prefixed final
+segment) for sessions that must keep their literal key.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Optional
+from urllib.parse import quote, unquote
+
+__all__ = [
+    "SessionRef",
+    "SHORT_ID_LEN",
+    "RESERVED_NAMES",
+    "derive_short_id",
+    "slugify",
+    "build_session_path",
+    "parse_session_path",
+]
+
+# Length of the short id derived from the trailing hex of a session key. Eight
+# hex chars (32 bits) keeps URLs terse while staying collision-resistant enough
+# for a per-agent namespace, where a resolver disambiguates the rare clash.
+SHORT_ID_LEN = 8
+
+# Reserved final-segment names that denote a well-known / sentinel session
+# (e.g. the agent's ``main`` conversation) rather than a ``slug-shortId`` pair.
+# A resolver treats these as a friendly sentinel, not a short-id lookup.
+RESERVED_NAMES = frozenset({"main", "global", "default", "root"})
+
+# Trailing run of hex digits that terminates the key — the opaque UUID tail we
+# shorten from. Anchored at end-of-string so a *hex run followed by non-hex*
+# (e.g. ``report-deadbeef.js``) is NOT shortened and instead round-trips via the
+# literal-key escape hatch; only a genuine hex terminus (a UUID tail) is lossy.
+_HEX_TAIL_RE = re.compile(r"([0-9a-fA-F]{%d,})$" % SHORT_ID_LEN)
+
+# A ``slug-shortId`` final segment: optional slug, then a hex short id.
+_SLUG_SHORT_RE = re.compile(r"^(?:(?P<slug>.+)-)?(?P<short>[0-9a-fA-F]{%d})$" % SHORT_ID_LEN)
+
+# Characters allowed verbatim in a slug; everything else collapses to ``-``.
+_SLUG_STRIP_RE = re.compile(r"[^a-z0-9]+")
+
+# Exact-key escape hatch: a final segment prefixed with ``!`` carries the
+# percent-encoded literal key verbatim (for sessions that must not be shortened).
+_LITERAL_PREFIX = "!"
+
+
+@dataclass(frozen=True)
+class SessionRef:
+    """A parsed friendly reference to a session, for a resolver to map to an id.
+
+    Exactly one of the resolution hints is authoritative depending on how the
+    path was written:
+
+    - ``literal_key`` set  -> resolve by exact key (escape hatch).
+    - ``short_id`` set     -> resolve by short id (optionally aided by ``slug_hint``).
+    - ``slug_hint`` only, matching a reserved name -> resolve the sentinel session.
+
+    ``agent_id`` scopes the lookup so short ids only need to be unique per agent.
+    """
+
+    agent_id: str
+    short_id: Optional[str] = None
+    slug_hint: Optional[str] = None
+    literal_key: Optional[str] = None
+
+
+def derive_short_id(session_key: str) -> Optional[str]:
+    """Derive a stable short id from the terminating hex run of ``session_key``.
+
+    Returns the last :data:`SHORT_ID_LEN` hex chars of the key's *terminating*
+    hex run (e.g. a UUID tail), lower-cased. The run must end the key: a key
+    whose hex is followed by non-hex characters (e.g. ``report-deadbeef.js``)
+    is deliberately not shortened, so it round-trips via the literal-key escape
+    hatch instead of silently losing its suffix. Returns ``None`` when the key
+    has no terminating hex run long enough to shorten — the caller then falls
+    back to the literal key or reserved-name form.
+    """
+    if not session_key:
+        return None
+    match = _HEX_TAIL_RE.search(session_key)
+    if not match:
+        return None
+    hex_tail = match.group(1)
+    return hex_tail[-SHORT_ID_LEN:].lower()
+
+
+def slugify(display_name: Optional[str], max_len: int = 48) -> Optional[str]:
+    """Turn a human label into a lowercase, dash-separated URL slug.
+
+    Non-alphanumeric runs collapse to a single ``-``; leading/trailing dashes are
+    trimmed. Returns ``None`` for an empty/whitespace-only or unusable name so
+    the path omits the slug rather than emitting an empty segment.
+    """
+    if not display_name:
+        return None
+    slug = _SLUG_STRIP_RE.sub("-", display_name.strip().lower()).strip("-")
+    if not slug:
+        return None
+    if len(slug) > max_len:
+        slug = slug[:max_len].rstrip("-")
+    return slug or None
+
+
+def build_session_path(
+    agent_id: str,
+    session_key: str,
+    display_name: Optional[str] = None,
+    namespace: str = "chat",
+) -> Optional[str]:
+    """Build a stable, shareable path for a session.
+
+    Produces ``<namespace>/<agentId>/<slug>-<shortId>`` when a short id can be
+    derived (with the slug omitted when there is no usable ``display_name``), so
+    the whole opaque UUID never appears in the URL. When the key has no hex tail
+    to shorten, falls back to an exact-key segment (``!<percent-encoded-key>``)
+    so the address round-trips losslessly. Returns ``None`` if ``agent_id`` or
+    ``session_key`` is empty.
+
+    Path segments are percent-escaped, so a key that looks like a file
+    (e.g. ``report.js``) is never mistaken for a static asset.
+    """
+    if not agent_id or not session_key:
+        return None
+    ns = _seg(namespace) or "chat"
+    agent_seg = _seg(agent_id)
+    if not agent_seg:
+        return None
+
+    short_id = derive_short_id(session_key)
+    if short_id:
+        slug = slugify(display_name)
+        tail = f"{slug}-{short_id}" if slug else short_id
+    else:
+        # No hex tail to shorten: keep the literal key behind the escape hatch.
+        tail = _LITERAL_PREFIX + quote(session_key, safe="")
+    return f"{ns}/{agent_seg}/{tail}"
+
+
+def parse_session_path(path: str) -> Optional[SessionRef]:
+    """Parse a session path back into a :class:`SessionRef`.
+
+    Accepts the forms produced by :func:`build_session_path`, with or without a
+    leading namespace segment:
+
+    - ``chat/<agent>/<slug>-<shortId>`` / ``<agent>/<slug>-<shortId>``
+    - ``chat/<agent>/<shortId>`` / ``<agent>/<shortId>``
+    - ``chat/<agent>/main`` (reserved sentinel) / ``<agent>/main``
+    - ``chat/<agent>/!<percent-encoded-key>`` (exact-key escape hatch)
+
+    Returns ``None`` when the path has no agent segment or is otherwise
+    unparseable. A resolver combines the returned hints with its store to reach
+    the concrete ``session_id``.
+    """
+    if not path:
+        return None
+    parts = [p for p in path.strip().strip("/").split("/") if p]
+    # ``build_session_path`` percent-escapes the namespace and agent into single
+    # segments, so a canonical address is exactly ``<ns>/<agent>/<ref>`` (3) or
+    # ``<agent>/<ref>`` (2). Reject anything longer so a malformed link like
+    # ``chat/x/y/z`` never silently acquires a different meaning by using only
+    # its last two segments.
+    if len(parts) == 3:
+        # Drop the leading namespace segment; agent + final ref remain.
+        agent_id = unquote(parts[-2])
+        final = parts[-1]
+    elif len(parts) == 2:
+        agent_id = unquote(parts[0])
+        final = parts[1]
+    else:
+        return None
+
+    agent_id = agent_id.strip()
+    if not agent_id or not final:
+        return None
+
+    # Exact-key escape hatch.
+    if final.startswith(_LITERAL_PREFIX):
+        literal = unquote(final[len(_LITERAL_PREFIX):])
+        if not literal:
+            return None
+        return SessionRef(agent_id=agent_id, literal_key=literal)
+
+    final = unquote(final)
+
+    # Reserved sentinel (main/global/...): a friendly named session.
+    if final.lower() in RESERVED_NAMES:
+        return SessionRef(agent_id=agent_id, slug_hint=final.lower())
+
+    match = _SLUG_SHORT_RE.match(final)
+    if match:
+        return SessionRef(
+            agent_id=agent_id,
+            short_id=match.group("short").lower(),
+            slug_hint=match.group("slug") or None,
+        )
+
+    # No short id present: treat the whole segment as a slug hint the resolver
+    # can match against session labels.
+    return SessionRef(agent_id=agent_id, slug_hint=final)
+
+
+def _seg(value: Optional[str]) -> str:
+    """Percent-escape a single path segment (``/`` and dots are encoded)."""
+    if not value:
+        return ""
+    return quote(value.strip(), safe="")

src/praisonai-agents/praisonaiagents/gateway/config.py

@@ -753,10 +753,17 @@ class ApiConfig:
             backed by the gateway's live agents and sessions.
         mcp: Serve an MCP JSON-RPC endpoint (``/mcp``) exposing the gateway's
             registered agents as callable tools.
+        stream: Emit true token-level deltas on the OpenAI-compatible
+            ``/v1/chat/completions`` SSE surface (``stream:true``) by wiring the
+            agent's existing ``stream_emitter`` through the gateway hot path.
+            Off by default: streaming responses stay byte-for-byte the buffered
+            single-chunk path, so enabling it never regresses non-streaming
+            latency or correctness.
     """
 
     openai: bool = False
     mcp: bool = False
+    stream: bool = False
 
     @property
     def enabled(self) -> bool:
@@ -765,7 +772,7 @@ class ApiConfig:
 
     def to_dict(self) -> Dict[str, Any]:
         """Convert to dictionary."""
-        return {"openai": self.openai, "mcp": self.mcp}
+        return {"openai": self.openai, "mcp": self.mcp, "stream": self.stream}
 
     @classmethod
     def from_dict(cls, data: Optional[Dict[str, Any]]) -> "ApiConfig":
@@ -775,6 +782,7 @@ class ApiConfig:
         return cls(
             openai=bool(data.get("openai", False)),
             mcp=bool(data.get("mcp", False)),
+            stream=bool(data.get("stream", False)),
         )
 
 
@@ -933,6 +941,73 @@ class AttachmentConfig:
         )
 
 
+def _parse_drain_timeout(value: Any) -> Optional[float]:
+    """Coerce a ``gateway.drain_timeout`` YAML value to a float (or None).
+
+    Tolerant of env-substituted strings (e.g. ``"20"``); an unparseable /
+    negative value falls back to None (no drain) rather than raising, matching
+    the wrapper's fail-open handling of a malformed drain window (#5362).
+    """
+    if value is None:
+        return None
+    try:
+        parsed = float(value)
+    except (TypeError, ValueError):
+        return None
+    return parsed if parsed >= 0 else None
+
+
+def _parse_watchdog_enabled(value: Any) -> bool:
+    """Whether a ``gateway.watchdog`` YAML value enables the loop watchdog.
+
+    Accepts both a bare truthy scalar and the documented block form
+    (``watchdog: { enabled: true, ... }``), mirroring the wrapper's watchdog
+    wiring so the public config reflects the same on/off decision (#5362).
+    """
+    if isinstance(value, dict):
+        flag = value.get("enabled", False)
+    else:
+        flag = value
+    if isinstance(flag, str):
+        return flag.strip().lower() in ("1", "true", "yes", "on")
+    return bool(flag)
+
+
+def _parse_watchdog_timeout(value: Any) -> Optional[float]:
+    """Derive the loop-stall budget (seconds) from a ``gateway.watchdog`` block.
+
+    Reads ``liveness_interval * liveness_strikes`` from the block form when
+    present so the public config exposes the effective wedge budget the CLI
+    ``--watchdog-timeout`` flag sets; returns None when unset/unparseable.
+    """
+    if not isinstance(value, dict):
+        return None
+    try:
+        interval = float(value.get("liveness_interval", 5.0))
+        strikes = int(value.get("liveness_strikes", 3)) or 3
+    except (TypeError, ValueError):
+        return None
+    budget = interval * strikes
+    return budget if budget > 0 else None
+
+
+def _parse_watchdog_timeout_scalar(value: Any) -> Optional[float]:
+    """Coerce an explicit ``gateway.watchdog_timeout`` scalar to a float.
+
+    Mirrors the ``watchdog_timeout > 0`` invariant (see ``GatewayConfig``):
+    tolerant of env-substituted strings (e.g. ``"15"``); an unparseable or
+    non-positive value falls back to None so a malformed timeout disables the
+    budget rather than raising (#5362).
+    """
+    if value is None:
+        return None
+    try:
+        parsed = float(value)
+    except (TypeError, ValueError):
+        return None
+    return parsed if parsed > 0 else None
+
+
 @dataclass
 class GatewayConfig:
     """Configuration for the gateway server.
@@ -965,6 +1040,13 @@ class GatewayConfig:
     host: str = "127.0.0.1"
     port: int = 8765
     bind_host: Optional[str] = None
+    # Issue #5312: CIDRs/IPs of upstream proxies/tunnels the operator declares
+    # trusted. Empty (default) means the gateway is directly exposed: forwarded
+    # headers are never believed and per-IP policy keys on the socket peer.
+    # When set, the real client IP is resolved by walking the forwarded chain
+    # across trusted hops (see ``resolve_ingress_attribution``); anything
+    # proxy-shaped but unattributable fails closed to the socket peer.
+    trusted_proxies: List[str] = field(default_factory=list)
     cors_origins: List[str] = field(default_factory=lambda: [])
     allowed_origins: List[str] = field(default_factory=lambda: [])
     auth_token: Optional[str] = None
@@ -988,6 +1070,14 @@ class GatewayConfig:
     max_concurrent_runs: int = 0  # Aggregate concurrency ceiling (0 = unlimited)
     queue_depth: int = 0  # Bounded wait queue when at the ceiling
     overflow_policy: str = "reject"  # reject | queue | shed_oldest
+    # Issue #5362: complete the back-pressure/lifecycle posture on the public
+    # config so a Python embedder / gateway.yaml can pin it declaratively —
+    # parity with the CLI ``--drain-timeout`` / ``--watchdog`` /
+    # ``--watchdog-timeout`` flags. ``None``/False keep today's behaviour
+    # (no drain, watchdog off); a CLI flag still overrides these at start.
+    drain_timeout: Optional[float] = None  # graceful-drain window on shutdown (s)
+    watchdog: bool = False  # event-loop liveness watchdog (opt-in)
+    watchdog_timeout: Optional[float] = None  # loop-stall budget before restart (s)
     # Issue #5168: per-tenant/per-scope concurrency sub-limit within the global
     # ceiling so one tenant cannot occupy every run slot. 0 disables the
     # sub-limit (today's global-only behaviour).
@@ -1067,6 +1157,14 @@ class GatewayConfig:
             raise ValueError(
                 "overflow_policy must be one of 'reject', 'queue', 'shed_oldest'"
             )
+        if self.drain_timeout is not None and self.drain_timeout < 0:
+            raise ValueError(
+                "drain_timeout must be >= 0 (use 0/None to disable the drain window)"
+            )
+        if self.watchdog_timeout is not None and self.watchdog_timeout <= 0:
+            raise ValueError(
+                "watchdog_timeout must be > 0 (leave unset to use the default budget)"
+            )
         if self.max_concurrent_runs_per_scope < 0:
             raise ValueError(
                 "max_concurrent_runs_per_scope must be >= 0 "
@@ -1139,6 +1237,7 @@ class GatewayConfig:
         return {
             "host": self.host,
             "port": self.port,
+            "trusted_proxies": list(self.trusted_proxies),
             "cors_origins": self.cors_origins,
             "allowed_origins": self.allowed_origins,
             "auth_token": "***" if self.auth_token else None,
@@ -1154,6 +1253,9 @@ class GatewayConfig:
             "max_concurrent_runs": self.max_concurrent_runs,
             "queue_depth": self.queue_depth,
             "overflow_policy": self.overflow_policy,
+            "drain_timeout": self.drain_timeout,
+            "watchdog": self.watchdog,
+            "watchdog_timeout": self.watchdog_timeout,
             "max_concurrent_runs_per_scope": self.max_concurrent_runs_per_scope,
             "preauth_max_connections_per_ip": self.preauth_max_connections_per_ip,
             "max_unauthorized_frames": self.max_unauthorized_frames,
@@ -1333,9 +1435,17 @@ class MultiChannelGatewayConfig:
                 for tok, scopes in gw_data["auth_scopes"].items()
             }
 
+        raw_trusted = gw_data.get("trusted_proxies", [])
+        trusted_proxies = (
+            [str(p).strip() for p in raw_trusted if str(p).strip()]
+            if isinstance(raw_trusted, (list, tuple))
+            else []
+        )
+
         gateway_config = GatewayConfig(
             host=gw_data.get("host", "127.0.0.1"),
             port=gw_data.get("port", 8765),
+            trusted_proxies=trusted_proxies,
             cors_origins=gw_data.get("cors_origins", []),
             allowed_origins=gw_data.get("allowed_origins", []),
             auth_token=gw_data.get("auth_token"),
@@ -1352,6 +1462,17 @@ class MultiChannelGatewayConfig:
             max_concurrent_runs=int(gw_data.get("max_concurrent_runs", 0) or 0),
             queue_depth=int(gw_data.get("queue_depth", 0) or 0),
             overflow_policy=str(gw_data.get("overflow_policy", "reject") or "reject"),
+            drain_timeout=_parse_drain_timeout(gw_data.get("drain_timeout")),
+            watchdog=_parse_watchdog_enabled(gw_data.get("watchdog")),
+            # Prefer an explicit ``gateway.watchdog_timeout`` scalar (as emitted
+            # by ``to_dict`` and settable directly, matching the CLI
+            # ``--watchdog-timeout`` flag); otherwise derive the budget from the
+            # ``watchdog`` block so both round-trip and block forms work (#5362).
+            watchdog_timeout=(
+                _parse_watchdog_timeout_scalar(gw_data.get("watchdog_timeout"))
+                if gw_data.get("watchdog_timeout") is not None
+                else _parse_watchdog_timeout(gw_data.get("watchdog"))
+            ),
             max_concurrent_runs_per_scope=int(
                 gw_data.get("max_concurrent_runs_per_scope", 0) or 0
             ),

src/praisonai-agents/praisonaiagents/gateway/liveness.py

@@ -52,6 +52,7 @@ from .protocols import GATEWAY_RESTART_EXIT_CODE
 __all__ = [
     "LoopWatchdogPolicy",
     "LoopWatchdog",
+    "StartupWatchdog",
 ]
 
 
@@ -437,3 +438,271 @@ class LoopWatchdog:
                     faulthandler.dump_traceback(file=fh, all_threads=True)
             except Exception:  # pragma: no cover - fail open
                 pass
+
+
+class StartupWatchdog:
+    """Front-rung liveness guard for the pre-loop startup window (Issue #5265).
+
+    :class:`LoopWatchdog` only becomes meaningful once the asyncio event loop
+    is running — it probes the loop via ``loop.call_soon_threadsafe``, so it can
+    do nothing before a loop exists. That leaves the window between process
+    entry and "loop is live" (heavy imports, config/secret loading, the initial
+    adapter ``connect()``\\ s) uncovered. A wedge there — a hanging SDK import, a
+    blocking synchronous DNS/network call while minting credentials, a deadlock
+    inside ``connect()`` — leaves the process *alive but making no progress*: it
+    holds no serving loop, answers no probes, and never exits, so an external
+    supervisor never restarts it. It becomes a silent zombie with no diagnostics.
+
+    This primitive closes that gap. It arms a wall-clock **deadline** on a
+    dedicated daemon OS thread *before* the heavy work begins, so it keeps
+    working precisely when the main thread is wedged. If the deadline elapses
+    without :meth:`confirm_loop_live`, it dumps all-thread stacks via
+    :mod:`faulthandler` (so the hang is diagnosable) and hands the process back
+    to the supervisor via ``os._exit`` with a restart-intent code. A legitimately
+    slow-but-alive boot extends the deadline via :meth:`report_startup_progress`
+    (bounded, so a truly wedged phase still trips). :meth:`confirm_loop_live`
+    disarms it the instant the serving loop begins — at which point
+    :class:`LoopWatchdog` takes over.
+
+    Design mirrors :class:`LoopWatchdog`: **opt-in** (nothing arms it unless the
+    embedder calls :meth:`arm`), **fail-open** (any error inside the watchdog
+    must never kill a healthy startup), and **stdlib-only** (it must run when the
+    rest of the process cannot, so its correctness cannot depend on the code it
+    guards). Typical use at the gateway entry point::
+
+        sw = StartupWatchdog()
+        sw.arm(deadline_s=config.startup_watchdog_timeout)
+        load_config(config);   sw.report_startup_progress("config loaded")
+        connect_adapters();    sw.report_startup_progress("adapters connected")
+        loop = asyncio.new_event_loop()
+        LoopWatchdog().arm(loop)
+        sw.confirm_loop_live()  # disarm; loop watchdog now owns liveness
+        loop.run_forever()
+    """
+
+    def __init__(
+        self,
+        *,
+        on_expire: Literal["dump_and_exit", "dump_only"] = "dump_and_exit",
+        exit_code: int = GATEWAY_RESTART_EXIT_CODE,
+        dump_file: Optional[str] = None,
+        max_extensions: int = 10,
+    ) -> None:
+        if on_expire not in ("dump_and_exit", "dump_only"):
+            raise ValueError(
+                "on_expire must be 'dump_and_exit' or 'dump_only'"
+            )
+        if max_extensions < 0:
+            raise ValueError("max_extensions must be >= 0")
+        self.on_expire = on_expire
+        self.exit_code = exit_code
+        self.dump_file = dump_file
+        self.max_extensions = max_extensions
+        self._thread: Optional[threading.Thread] = None
+        # Each armed generation gets its own stop event handed straight to the
+        # thread, so a confirm/re-arm race can never let a stale generation
+        # observe a fresh generation's cleared flag and self-exit (mirrors the
+        # per-generation discipline of LoopWatchdog's deadline mode). The stop
+        # event, once set, stays set: a healthy confirm/disarm is monotonic.
+        self._stop = threading.Event()
+        # A separate wake event carries progress reports: it is briefly set to
+        # nudge the waiting thread to re-read the (extended) deadline, then
+        # cleared. Keeping it distinct from _stop means a progress nudge can
+        # never be mistaken for a disarm (and vice versa).
+        self._wake = threading.Event()
+        self._deadline_s = 0.0
+        self._extensions_used = 0
+        # Progress reports enqueue here; the watchdog thread drains this under
+        # the lock so budget is spent exactly once per report even when several
+        # reports batch between two wakes.
+        self._pending_progress = 0
+        self._expired = False
+        self._confirmed = False
+        self._lock = threading.Lock()
+
+    @property
+    def armed(self) -> bool:
+        return self._thread is not None and self._thread.is_alive()
+
+    @property
+    def expired(self) -> bool:
+        """True once the startup deadline expired (mainly for ``dump_only``)."""
+        return self._expired
+
+    @property
+    def confirmed(self) -> bool:
+        """True once :meth:`confirm_loop_live` has disarmed the watchdog."""
+        return self._confirmed
+
+    def arm(self, *, deadline_s: float, dump_stacks: bool = True) -> None:
+        """Arm a startup deadline on a dedicated daemon thread. Fail-open.
+
+        Idempotent: arming while already armed is a no-op. A non-finite or
+        non-positive ``deadline_s`` is ignored (nothing to guard). ``dump_stacks``
+        toggles the all-thread stack dump on expiry.
+        """
+        if not math.isfinite(deadline_s) or deadline_s <= 0:
+            return
+        with self._lock:
+            if self.armed:
+                return
+            stop = threading.Event()
+            wake = threading.Event()
+            self._stop = stop
+            self._wake = wake
+            self._deadline_s = deadline_s
+            self._extensions_used = 0
+            self._pending_progress = 0
+            self._expired = False
+            self._confirmed = False
+            self._dump_stacks_on_expire = dump_stacks
+            try:
+                thread = threading.Thread(
+                    target=self._run,
+                    args=(stop, wake),
+                    name="praisonai-startup-watchdog",
+                    daemon=True,
+                )
+                thread.start()
+            except Exception:  # pragma: no cover - fail open
+                self._thread = None
+                return
+            self._thread = thread
+
+    def report_startup_progress(self, phase: str) -> None:
+        """Signal forward progress, extending the deadline by one interval.
+
+        A legitimately slow boot (e.g. many adapters, cold DNS) calls this at
+        each phase so it is not killed. Extensions are bounded by
+        ``max_extensions`` so a phase that itself wedges still trips the
+        deadline. Fail-open and a no-op when not armed.
+
+        Budget accounting is done by the watchdog thread, not here: each report
+        merely enqueues a pending nudge. If several reports land before the
+        thread wakes, each still costs exactly one extension *and* each still
+        earns one fresh deadline window when the thread drains them — so a later
+        slow-but-progressing phase can never find the budget spent on resets it
+        never received (Issue #5265 review: "progress reports lose extensions").
+        """
+        if not self.armed:
+            return
+        with self._lock:
+            self._pending_progress += 1
+            wake = self._wake
+        # Wake the waiting thread so it drains the pending nudges, spends budget,
+        # and resets its window once per still-affordable report. The thread
+        # keeps waiting against its own (still-unset) stop event.
+        wake.set()
+
+    def confirm_loop_live(self) -> None:
+        """Disarm the startup watchdog; the event loop is now live.
+
+        Called immediately before :class:`LoopWatchdog` is armed. Safe to call
+        multiple times / when not armed. Sets the *current* generation's stop
+        event so that exact thread suppresses its self-exit even if it is
+        mid-dump, then drops the reference.
+        """
+        with self._lock:
+            self._confirmed = True
+            self._stop.set()
+            # Nudge the thread out of its wait immediately so the join is prompt.
+            self._wake.set()
+            thread = self._thread
+            self._thread = None
+        if thread is not None and thread is not threading.current_thread():
+            try:
+                thread.join(timeout=1.0)
+            except Exception:  # pragma: no cover - fail open
+                pass
+
+    def _run(self, stop: "threading.Event", wake: "threading.Event") -> None:
+        # Wait out the deadline against *this generation's* events. ``stop`` is
+        # set (and stays set) by a confirm/disarm — that must break us out and
+        # suppress the self-exit. ``wake`` is a transient progress nudge — on
+        # each wake we drain the queued progress reports, spending one extension
+        # of budget per report and resetting the window each time we can still
+        # afford it. A plain timeout with neither event set (and no affordable
+        # pending progress) means the startup genuinely wedged.
+        start = time.monotonic()
+        while not stop.is_set():
+            remaining = (start + self._deadline_s) - time.monotonic()
+            if remaining <= 0:
+                break
+            if wake.wait(remaining):
+                wake.clear()
+                if stop.is_set():
+                    return
+                # Drain queued progress reports. Each report costs one extension
+                # and, while budget remains, grants one fresh deadline window —
+                # so batched reports never lose the resets they paid for.
+                extended = False
+                with self._lock:
+                    pending = self._pending_progress
+                    self._pending_progress = 0
+                    for _ in range(pending):
+                        if self._extensions_used >= self.max_extensions:
+                            break
+                        self._extensions_used += 1
+                        extended = True
+                if extended:
+                    start = time.monotonic()
+                # Whether or not budget remained, loop back and re-check the
+                # (possibly unchanged) deadline against this generation's stop.
+                continue
+        self._on_expire(stop)
+
+    def _on_expire(self, stop: "threading.Event") -> None:
+        # Only fire if this generation was not disarmed. A confirm_loop_live()
+        # that raced in has set this exact event, so a stale generation stays
+        # inert and never self-exits a healthy startup.
+        if stop.is_set():
+            return
+        self._expired = True
+        if getattr(self, "_dump_stacks_on_expire", True):
+            try:
+                self._dump_stacks()
+            except Exception:  # pragma: no cover - fail open
+                pass
+        if self.on_expire == "dump_and_exit":
+            # If confirm_loop_live() raced in while we were dumping, honour it.
+            if stop.is_set():
+                return
+            try:
+                sys.stderr.flush()
+            except Exception:  # pragma: no cover - fail open
+                pass
+            # Final re-check the instant before the irreversible os._exit: the
+            # flush above can block (e.g. a full pipe) long enough for a
+            # concurrent confirm_loop_live() to land. A startup that has just
+            # confirmed a live loop must never be terminated (Issue #5265
+            # review: "confirmation cannot prevent late exit").
+            if stop.is_set():
+                return
+            os._exit(self.exit_code)
+
+    def _dump_stacks(self) -> None:
+        reason = (
+            f"gateway startup did not confirm a live event loop within "
+            f"~{self._deadline_s:.0f}s"
+        )
+        header = (
+            f"praisonai-startup-watchdog: {reason}; "
+            f"dumping all-thread stacks\n"
+        )
+        try:
+            sys.stderr.write(header)
+        except Exception:  # pragma: no cover - fail open
+            pass
+        try:
+            faulthandler.dump_traceback(file=sys.stderr, all_threads=True)
+        except Exception:  # pragma: no cover - fail open
+            pass
+        dump_file = self.dump_file
+        if dump_file:
+            try:
+                with open(dump_file, "a", encoding="utf-8") as fh:
+                    fh.write(header)
+                    fh.flush()
+                    faulthandler.dump_traceback(file=fh, all_threads=True)
+            except Exception:  # pragma: no cover - fail open
+                pass

src/praisonai-agents/praisonaiagents/gateway/protocols.py

@@ -2555,6 +2555,225 @@ def resolve_auth_mode(bind_host: str, configured: Optional[AuthMode] = None) ->
     return "local" if is_loopback(bind_host) else "token"
 
 
+# ---------------------------------------------------------------------------
+# Ingress attribution / trusted-proxy real-client-IP resolution (Issue #5312)
+# ---------------------------------------------------------------------------
+#
+# The gateway keys rate-limiting, the pre-auth connection budget, and the
+# operator identity on the *raw socket peer*. Behind a reverse proxy or tunnel
+# (nginx, Caddy, Traefik, a cloud LB, Cloudflare, Tailscale Serve/Funnel) that
+# peer is the *proxy* for every request, so every per-IP bucket collapses to
+# one and the operator id becomes identical for all callers. Trusting
+# ``X-Forwarded-For`` unconditionally is worse — a directly-reachable gateway
+# would accept a spoofed client IP any attacker can set.
+#
+# This pure, import-free contract resolves the real client address *only* by
+# walking the forwarded chain across hops the operator has explicitly declared
+# trusted, and **fails closed** for anything proxy-shaped but unattributable
+# (rate-limited as one bucket keyed on the socket peer; never trusts a header).
+# It lives in core so every seam consults one unit-testable decision, exactly
+# like the sibling ``resolve_route`` / ``evaluate_pressure`` contracts.
+
+IngressTrust = Literal[
+    "direct-local",
+    "direct-remote",
+    "trusted-proxy",
+    "tunnel",
+    "unattributable-proxy",
+]
+"""Classification of where an inbound request actually came from.
+
+- ``direct-local``: loopback peer, no proxy headers (local CLI / same host).
+- ``direct-remote``: a remote peer talking to us directly, no proxy headers.
+- ``trusted-proxy``: arrived via one or more hops all in ``trusted_proxies``;
+  the real client IP was resolved from the forwarded chain.
+- ``tunnel``: a trusted loopback/private tunnel hop (e.g. Tailscale/Cloudflare
+  sidecar on loopback) presented a forwarded client — resolved like a trusted
+  proxy.
+- ``unattributable-proxy``: proxy-shaped (forwarded headers present) but the
+  immediate hop is not declared trusted — fails closed, header not trusted.
+"""
+
+
+@dataclass(frozen=True)
+class IngressAttribution:
+    """Resolved answer to "who/where is this request from", consulted before
+    any per-IP policy.
+
+    Attributes:
+        trust: The :data:`IngressTrust` classification.
+        client_ip: The subject to key per-IP policy on. The *real* client only
+            when it was resolved across trusted hops; otherwise the raw socket
+            peer (never a spoofable header value).
+        via_proxy: Whether forwarded headers were present at all.
+        fail_closed: True when the request was proxy-shaped but unattributable —
+            the caller must rate-limit it as a single bucket keyed on the socket
+            peer and must never trust a client IP from a header.
+    """
+
+    trust: IngressTrust
+    client_ip: str
+    via_proxy: bool
+    fail_closed: bool
+
+
+def parse_forwarded_for(header: Optional[str]) -> "List[str]":
+    """Parse an ``X-Forwarded-For`` header into an ordered list of hops.
+
+    The chain is left→right, client-first: ``client, proxy1, proxy2``. Empty
+    tokens and surrounding whitespace are dropped; ``None``/empty yields an
+    empty list. Pure string handling — no name resolution, no validation of
+    whether each token is a real IP (that is the caller's trust decision).
+    """
+    if not header:
+        return []
+    return [tok.strip() for tok in header.split(",") if tok.strip()]
+
+
+def _is_valid_ip(host: Optional[str]) -> bool:
+    """Return whether ``host`` parses as a literal IPv4/IPv6 address.
+
+    Forwarded-header tokens are attacker-influenced strings; a value such as
+    ``unknown`` or an arbitrary label must never become a per-IP policy subject
+    (it would group unrelated callers or split one caller across buckets). The
+    caller uses this to fail closed to the socket peer for a non-IP hop.
+    """
+    if not host:
+        return False
+    import ipaddress
+
+    try:
+        ipaddress.ip_address(str(host).strip())
+        return True
+    except ValueError:
+        return False
+
+
+def _is_trusted_hop(host: str, trusted_proxies: "Sequence[str]") -> bool:
+    """Return whether ``host`` is a member of the declared trusted set.
+
+    A trusted entry may be a bare IP or a CIDR (``10.0.0.0/8``). Loopback is
+    *not* implicitly trusted here — the caller decides how to treat a loopback
+    tunnel — so this stays a pure membership test.
+    """
+    import ipaddress
+
+    try:
+        ip = ipaddress.ip_address(host)
+    except ValueError:
+        return False
+    for entry in trusted_proxies:
+        entry = str(entry).strip()
+        if not entry:
+            continue
+        try:
+            if "/" in entry:
+                if ip in ipaddress.ip_network(entry, strict=False):
+                    return True
+            elif ip == ipaddress.ip_address(entry):
+                return True
+        except ValueError:
+            continue
+    return False
+
+
+def resolve_ingress_attribution(
+    *,
+    peer_ip: str,
+    forwarded_for: "Sequence[str]" = (),
+    real_ip: Optional[str] = None,
+    trusted_proxies: "Sequence[str]" = (),
+) -> IngressAttribution:
+    """Resolve the real client attribution for an inbound request.
+
+    Pure and side-effect-free so it is unit-testable in isolation and every
+    seam (rate limiters, connection budget, operator id) consults the same
+    decision.
+
+    Args:
+        peer_ip: Raw socket peer (``request.client.host``).
+        forwarded_for: Parsed ``X-Forwarded-For`` chain, left→right
+            (client-first). See :func:`parse_forwarded_for`.
+        real_ip: Optional ``X-Real-IP`` value (used only when there is no
+            forwarded chain and the immediate hop is trusted).
+        trusted_proxies: CIDRs/IPs the operator declares trusted.
+
+    Returns:
+        An :class:`IngressAttribution`. The real client IP is resolved *only*
+        across hops in ``trusted_proxies``; anything proxy-shaped but
+        unattributable fails closed to the socket peer.
+
+    Resolution rules:
+        * No proxy headers → ``direct-local`` (loopback peer) or
+          ``direct-remote``; ``client_ip`` is the peer, never fail-closed.
+        * Proxy headers present but the socket peer is not a trusted hop →
+          ``unattributable-proxy``, ``fail_closed=True``, ``client_ip`` = peer
+          (a spoofable header is never trusted on a directly-reachable
+          gateway).
+        * Socket peer is a trusted hop → walk the forwarded chain right→left,
+          peeling trusted hops, and stop at the first untrusted address: that
+          address is the real client. If the chain is empty, fall back to
+          ``real_ip`` then the peer. The resolved client must be a literal IP —
+          a malformed/non-IP hop fails closed to the peer. A loopback trusted
+          hop is classified ``tunnel``; otherwise ``trusted-proxy``.
+    """
+    has_headers = bool(forwarded_for) or bool(real_ip)
+
+    # No forwarded headers at all — attribute straight to the socket peer.
+    if not has_headers:
+        trust: IngressTrust = "direct-local" if is_loopback(peer_ip) else "direct-remote"
+        return IngressAttribution(
+            trust=trust,
+            client_ip=peer_ip,
+            via_proxy=False,
+            fail_closed=False,
+        )
+
+    # Proxy-shaped. The immediate hop (socket peer) must itself be trusted for
+    # any header to be believed; otherwise fail closed to the socket peer.
+    if not _is_trusted_hop(peer_ip, trusted_proxies):
+        return IngressAttribution(
+            trust="unattributable-proxy",
+            client_ip=peer_ip,
+            via_proxy=True,
+            fail_closed=True,
+        )
+
+    # Trusted immediate hop. Walk the forwarded chain right→left, peeling hops
+    # that are themselves trusted; the first untrusted address is the real
+    # client. Stop as soon as we hit it so a client-supplied spoofed prefix
+    # cannot be peeled past.
+    client_ip = None
+    for hop in reversed(list(forwarded_for)):
+        if _is_trusted_hop(hop, trusted_proxies):
+            continue
+        client_ip = hop
+        break
+    if client_ip is None:
+        # Entire chain was trusted (or empty). Prefer X-Real-IP, then peer.
+        client_ip = (real_ip.strip() if real_ip else None) or peer_ip
+
+    # The resolved client must be a literal IP before it keys per-IP policy. A
+    # malformed hop (e.g. ``unknown`` or an arbitrary label injected upstream)
+    # would otherwise group unrelated callers or split one caller across
+    # buckets — fail closed to the socket peer instead (Greptile P2).
+    if not _is_valid_ip(client_ip):
+        return IngressAttribution(
+            trust="unattributable-proxy",
+            client_ip=peer_ip,
+            via_proxy=True,
+            fail_closed=True,
+        )
+
+    trust = "tunnel" if is_loopback(peer_ip) else "trusted-proxy"
+    return IngressAttribution(
+        trust=trust,
+        client_ip=client_ip,
+        via_proxy=True,
+        fail_closed=False,
+    )
+
+
 # ---------------------------------------------------------------------------
 # Weak / placeholder secret guard (Issue #3259)
 # ---------------------------------------------------------------------------
@@ -3074,6 +3293,57 @@ ReactionStatus = Literal["ok", "unsupported", "failed", "no_route"]
 """
 
 
+MessageActionStatus = Literal["ok", "unsupported", "failed", "no_route"]
+"""Closed set of outcomes for a message-mutation verb (``edit`` / ``delete``).
+
+Mirrors :data:`ReactionStatus` so every agent-callable message action returns
+the same typed shape:
+
+* ``ok`` — the message was edited/deleted.
+* ``unsupported`` — the channel has no matching capability
+  (``supports_edit`` / ``supports_delete``).
+* ``failed`` — the transport rejected the action (e.g. no such message, or the
+  message is not editable/deletable by this bot).
+* ``no_route`` — the target could not be resolved to a reachable channel.
+"""
+
+
+@dataclass
+class MessageActionResult:
+    """Outcome of an agent-initiated message mutation (Issue #5054).
+
+    Shared typed result for the ``edit`` and ``delete`` verbs of the unified
+    message-action surface. Every call resolves to exactly one
+    :data:`MessageActionStatus`, so a channel that cannot edit/delete returns a
+    typed ``unsupported`` outcome instead of raising — identical to how
+    :class:`ReactionResult` and :class:`ThreadResult` degrade.
+
+    Attributes:
+        status: The outcome (``ok`` / ``unsupported`` / ``failed`` /
+            ``no_route``).
+        target: The resolved target the action was routed to.
+        detail: Optional model-readable explanation.
+    """
+
+    status: MessageActionStatus
+    target: str = ""
+    detail: Optional[str] = None
+
+    @property
+    def ok(self) -> bool:
+        """Whether the action succeeded (``status == "ok"``)."""
+        return self.status == "ok"
+
+    def as_dict(self) -> Dict[str, Any]:
+        """Convert to a serializable dictionary for the tool return value."""
+        data: Dict[str, Any] = {"status": self.status}
+        if self.target:
+            data["target"] = self.target
+        if self.detail:
+            data["detail"] = self.detail
+        return data
+
+
 @dataclass
 class ReactionResult:
     """Outcome of an agent-initiated message reaction (Issue #3917).
@@ -3259,6 +3529,51 @@ class OutboundMessengerProtocol(Protocol):
         """
         ...
 
+    async def edit(
+        self,
+        target: str,
+        message_id: str,
+        text: str,
+    ) -> "MessageActionResult":
+        """Edit a prior message this agent/session sent (Issue #5054).
+
+        Lets an agent update its own live status message in place rather than
+        posting a follow-up — e.g. editing "Working…" to "Done ✅". Gated on the
+        channel's ``PlatformCapabilities.supports_edit``; a channel that cannot
+        edit returns a typed ``unsupported`` outcome instead of raising.
+
+        Args:
+            target: Symbolic target token ("origin", "<platform>",
+                "<platform>:<chat_id>[:<thread_id>]", or a friendly alias).
+            message_id: The id of the message to edit.
+            text: The new message text.
+
+        Returns:
+            A :class:`MessageActionResult` describing the outcome.
+        """
+        ...
+
+    async def delete(
+        self,
+        target: str,
+        message_id: str,
+    ) -> "MessageActionResult":
+        """Delete/unsend a prior message this agent/session sent (Issue #5054).
+
+        Lets an agent retract a message it should not have sent. Gated on the
+        channel's ``PlatformCapabilities.supports_delete``; a channel that
+        cannot delete returns a typed ``unsupported`` outcome instead of raising.
+
+        Args:
+            target: Symbolic target token ("origin", "<platform>",
+                "<platform>:<chat_id>[:<thread_id>]", or a friendly alias).
+            message_id: The id of the message to delete.
+
+        Returns:
+            A :class:`MessageActionResult` describing the outcome.
+        """
+        ...
+
 
 # ---------------------------------------------------------------------------
 # Agent-callable cross-conversation request/reply (Issue #3689)
@@ -3365,6 +3680,122 @@ class ConversationRequestProtocol(Protocol):
         ...
 
 
+# ---------------------------------------------------------------------------
+# Server→client interactive request/reply over the gateway transport
+# (Issue #5351)
+#
+# A blocked HITL turn (approval / choice / free-text input) needs to reach a
+# generic gateway client — a web dashboard, TUI, or custom app — and correlate
+# the client's answer back into the waiting turn. Structured elicitation today
+# only renders through per-channel button backends; a client on the gateway
+# WebSocket cannot see or answer one, and an in-flight prompt is lost when the
+# client drops (the ``since`` cursor replays *past events*, not *still-open
+# requests*).
+#
+# Core owns only the *shape*: the typed request (:class:`GatewayServerRequest`),
+# the correlated reply (:class:`GatewayServerReply`), and the protocol seam
+# (:class:`GatewayRequestChannelProtocol`) — including the ``open_requests``
+# contract that the resume path honours so a reconnecting client is re-sent
+# every still-open (and *only* still-open) request for its sessions. The socket
+# delivery, correlation, and durable open-request set are bound by the running
+# gateway (praisonai-bot) exactly as the outbound messenger is — no heavy
+# import lives in core.
+# ---------------------------------------------------------------------------
+
+GatewayRequestKind = Literal["approval", "choice", "input"]
+"""Closed set of server→client interactive request kinds.
+
+* ``approval`` — allow/deny; the reply ``value`` is ``"allow"`` or ``"deny"``.
+* ``choice`` — select one of ``options``; the reply ``value`` is the option.
+* ``input`` — free text; the reply ``value`` is the entered text.
+"""
+
+
+@dataclass
+class GatewayServerRequest:
+    """A typed server→client interactive request (Issue #5351).
+
+    Delivered to a connected gateway client as a ``server_request`` frame while
+    a turn is blocked awaiting a human answer, and re-issued verbatim on resume
+    for as long as it stays open.
+
+    Attributes:
+        request_id: Correlation id echoed back in the :class:`GatewayServerReply`.
+        kind: One of :data:`GatewayRequestKind`.
+        prompt: Human-readable prompt to render.
+        options: Selectable options, populated only for ``kind == "choice"``.
+        session_id: Session the request belongs to (used for resume replay).
+    """
+
+    request_id: str
+    kind: GatewayRequestKind
+    prompt: str
+    options: Optional[List[str]] = None
+    session_id: str = ""
+
+    def as_dict(self) -> Dict[str, Any]:
+        """Convert to a serializable dictionary for the ``server_request`` frame."""
+        data: Dict[str, Any] = {
+            "request_id": self.request_id,
+            "kind": self.kind,
+            "prompt": self.prompt,
+        }
+        if self.options is not None:
+            data["options"] = list(self.options)
+        if self.session_id:
+            data["session_id"] = self.session_id
+        return data
+
+
+@dataclass
+class GatewayServerReply:
+    """A client's correlated answer to a :class:`GatewayServerRequest`.
+
+    Attributes:
+        request_id: The id of the request being answered.
+        value: ``"allow"``/``"deny"`` for approval, the chosen option for
+            choice, or the entered text for input.
+    """
+
+    request_id: str
+    value: str
+
+    def as_dict(self) -> Dict[str, Any]:
+        """Convert to a serializable dictionary."""
+        return {"request_id": self.request_id, "value": self.value}
+
+
+@runtime_checkable
+class GatewayRequestChannelProtocol(Protocol):
+    """Protocol for server→client interactive request/reply over the transport.
+
+    A concrete implementation is bound by the running gateway (in praisonai-bot)
+    and registered into the per-turn context so a blocked HITL turn can deliver
+    an :class:`GatewayServerRequest` to the connected client and await the
+    correlated :class:`GatewayServerReply`. ``open_requests`` returns the
+    still-open requests for a session so the gateway's resume path can re-issue
+    them alongside the event replay — a dropped connection never wedges a turn.
+    """
+
+    async def request(
+        self,
+        req: "GatewayServerRequest",
+        *,
+        timeout_s: float = 120.0,
+    ) -> Optional["GatewayServerReply"]:
+        """Deliver ``req`` to the session's client and await its reply.
+
+        Returns the correlated :class:`GatewayServerReply`, or ``None`` when no
+        reply arrives within ``timeout_s`` (the request is then cleared from the
+        open set so it is not replayed on a later reconnect).
+        """
+        ...
+
+    def open_requests(self, session_id: str) -> List["GatewayServerRequest"]:
+        """Return the still-open (unanswered) requests for ``session_id``."""
+        ...
+
+
 # ---------------------------------------------------------------------------
 # Agent-callable live status/health (Issue #3688)
 #
@@ -7364,8 +7795,16 @@ def _register_core_gateway_methods() -> None:
         # the same action, so both names carry the WRITE scope in lockstep.
         "abort": OperatorScope.WRITE,
         "message_abort": OperatorScope.WRITE,
+        # Issue #5351: answering a server→client interactive request (approval /
+        # choice / input) on behalf of the blocked turn mutates it, so it needs
+        # the same WRITE scope as sending a message.
+        "server_reply": OperatorScope.WRITE,
         "session.status": OperatorScope.READ,
         "session.transcript": OperatorScope.READ,
+        # Resolve a friendly reference (short id / slug / label / ``main``
+        # sentinel, optionally agent-scoped) to a concrete session id (Issue
+        # #5383). Read-only lookup: it reveals no more than session.status.
+        "session.resolve": OperatorScope.READ,
         "approvals.resolve": OperatorScope.APPROVALS,
         "pairing.approve": OperatorScope.PAIRING,
         "pairing.revoke": OperatorScope.PAIRING,

src/praisonai-agents/praisonaiagents/gateway/session_projection.py

@@ -0,0 +1,365 @@
+"""Session-projection reducer for Gateway clients (Issue #5324).
+
+A pure, dependency-free state machine that folds an initial snapshot plus a
+stream of :class:`~praisonaiagents.gateway.protocols.GatewayEvent`\\ s into a
+consistent, de-duplicated, memory-bounded view of a session. It turns *a
+stream of frames* into *a correct, resumable view* so every gateway client
+(the bundled Python client, the ``praisonai-ts`` mirror, the realtime
+dashboard, third-party integrations) stops re-implementing the same fiddly
+reconciliation logic:
+
+- **Snapshot + fold**: apply a snapshot, then fold live events into ``entries``.
+- **Identity de-duplication**: a final assistant message that arrives both as a
+  streamed delta sequence *and* as a persisted transcript row renders once.
+- **Optimistic reconciliation**: a locally-echoed outbound message is replaced
+  (not duplicated) by its durable counterpart, keyed on ``request_id``.
+- **Gap handling**: a sequence gap flips ``has_transport_gap``; after a resync
+  the reducer converges rather than appending duplicates.
+- **Bounded retention**: tracked runs are capped (LRU) so long-lived sessions
+  stay memory-bounded; a run with an active (unfinished) stream is never
+  evicted.
+
+This module lives in **core** because it is a pure reducer built on types core
+already owns (``GatewayEvent``/``GatewayMessage``/``EventType``); it must not be
+trapped in the transport client so all clients can share it. It has **no**
+third-party dependencies and does no I/O.
+"""
+
+from __future__ import annotations
+
+from collections import OrderedDict
+from dataclasses import dataclass, field, replace
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from .protocols import EventType, GatewayEvent, GatewayMessage
+
+__all__ = [
+    "RunView",
+    "SessionProjectionState",
+    "SessionProjection",
+]
+
+
+def _event_type_value(event: GatewayEvent) -> str:
+    """Return the wire string for an event type (enum or raw str)."""
+    etype = event.type
+    return etype.value if isinstance(etype, EventType) else str(etype)
+
+
+def _run_id_of(event: GatewayEvent) -> Optional[str]:
+    """Best-effort run/turn identifier for a streaming event."""
+    data = event.data or {}
+    for key in ("run_id", "turn_id", "message_id", "response_id"):
+        value = data.get(key)
+        if value:
+            return str(value)
+    return None
+
+
+@dataclass(frozen=True)
+class RunView:
+    """Immutable view of a single agent run/turn within a session.
+
+    Attributes:
+        run_id: The run/turn identifier the streamed deltas belong to.
+        text: The accumulated streamed text so far.
+        done: Whether the stream has ended (``STREAM_END``) for this run.
+        final_message_id: ``message_id`` of the persisted final row, once the
+            durable transcript row for this run has been reconciled in.
+    """
+
+    run_id: str
+    text: str = ""
+    done: bool = False
+    final_message_id: Optional[str] = None
+
+
+@dataclass(frozen=True)
+class SessionProjectionState:
+    """Immutable, de-duplicated, bounded view of a session.
+
+    Attributes:
+        entries: Ordered transcript messages, de-duplicated by identity.
+        runs: Per-run streaming views, keyed by ``run_id``.
+        has_transport_gap: ``True`` when a sequence gap was observed and a
+            resync has not yet reconciled the state.
+    """
+
+    entries: Tuple[GatewayMessage, ...] = ()
+    runs: Dict[str, RunView] = field(default_factory=dict)
+    has_transport_gap: bool = False
+
+
+class SessionProjection:
+    """Fold a snapshot + :class:`GatewayEvent`\\ s into a consistent view.
+
+    The reducer is idempotent: applying the same persisted message twice (e.g.
+    once from the live stream's ``STREAM_END`` and once as a durable transcript
+    row on reconnect) yields a single entry. Feed it with
+    :meth:`apply_snapshot` then :meth:`apply`; read :attr:`state` at any time.
+
+    Args:
+        max_tracked_runs: Cap on how many finished runs are tracked before the
+            oldest are evicted (LRU). Runs with an active/unfinished stream are
+            never evicted. Must be positive.
+    """
+
+    def __init__(self, *, max_tracked_runs: int = 200) -> None:
+        if max_tracked_runs <= 0:
+            raise ValueError("max_tracked_runs must be positive")
+        self._max_tracked_runs = max_tracked_runs
+        self._entries: List[GatewayMessage] = []
+        # message_id -> index into self._entries, for identity de-dup.
+        self._by_message_id: Dict[str, int] = {}
+        # request_id -> index into self._entries, for optimistic reconciliation.
+        self._by_request_id: Dict[str, int] = {}
+        # run_id -> RunView, ordered for LRU eviction.
+        self._runs: "OrderedDict[str, RunView]" = OrderedDict()
+        self._expected_sequence: Optional[int] = None
+        self._has_gap: bool = False
+
+    # -- public API -------------------------------------------------------
+
+    @property
+    def state(self) -> SessionProjectionState:
+        """Return the current immutable projection state."""
+        return SessionProjectionState(
+            entries=tuple(self._entries),
+            runs=dict(self._runs),
+            has_transport_gap=self._has_gap,
+        )
+
+    def apply_snapshot(
+        self, snapshot: Union[Dict[str, Any], Any]
+    ) -> SessionProjectionState:
+        """Replace all tracked state from a full snapshot.
+
+        A snapshot resets gap tracking and rebuilds the transcript from the
+        persisted rows it carries, so a resync converges rather than appending
+        duplicates. Accepts a mapping (``{"messages"/"entries": [...],
+        "cursor"/"sequence": N}``) or any object exposing ``messages``/
+        ``entries`` and an optional ``sequence``/``cursor`` attribute.
+        """
+        self._entries = []
+        self._by_message_id = {}
+        self._by_request_id = {}
+        self._runs = OrderedDict()
+        self._has_gap = False
+        self._expected_sequence = None
+
+        rows = _snapshot_rows(snapshot)
+        for row in rows:
+            message = _coerce_message(row)
+            if message is not None:
+                self._upsert_message(message)
+
+        seq = _snapshot_sequence(snapshot)
+        if seq is not None:
+            self._expected_sequence = seq + 1
+        return self.state
+
+    def apply(self, event: GatewayEvent) -> SessionProjectionState:
+        """Fold a single event into the state and return the new state.
+
+        Idempotent for persisted messages (de-duped by ``message_id``) and for
+        optimistic echoes (reconciled by ``request_id``). Streaming deltas are
+        accumulated per run; ``STREAM_END`` closes the run. Sequence gaps set
+        :attr:`SessionProjectionState.has_transport_gap`.
+        """
+        self._track_sequence(event)
+
+        etype = _event_type_value(event)
+        if etype in _STREAM_DELTA_TYPES:
+            self._apply_delta(event)
+        elif etype == EventType.STREAM_END.value:
+            self._apply_stream_end(event)
+        elif etype in _MESSAGE_TYPES:
+            self._apply_message(event)
+        # Other event types (health, presence, typing, ...) carry no transcript
+        # state; they are intentionally ignored by the projection.
+        return self.state
+
+    # -- sequence / gap tracking -----------------------------------------
+
+    def _track_sequence(self, event: GatewayEvent) -> None:
+        seq = event.sequence
+        if seq is None:
+            return
+        if self._expected_sequence is not None and seq > self._expected_sequence:
+            # A hole in the monotonic sequence: mark the view as gapped until a
+            # snapshot/resync reconciles it.
+            self._has_gap = True
+        # Advance the watermark to the highest sequence seen.
+        if self._expected_sequence is None or seq >= self._expected_sequence:
+            self._expected_sequence = seq + 1
+
+    # -- message handling -------------------------------------------------
+
+    def _apply_message(self, event: GatewayEvent) -> None:
+        message = _coerce_message(event.data)
+        if message is None:
+            return
+        request_id = _request_id_of(event, message)
+        self._upsert_message(message, request_id=request_id)
+
+    def _upsert_message(
+        self, message: GatewayMessage, *, request_id: Optional[str] = None
+    ) -> None:
+        # Identity de-dup: same persisted row seen twice renders once.
+        existing = self._by_message_id.get(message.message_id)
+        if existing is not None:
+            self._entries[existing] = message
+            self._reindex(existing, message, request_id)
+            return
+
+        # Optimistic reconciliation: a durable row replaces its local echo.
+        if request_id is not None:
+            echoed = self._by_request_id.get(request_id)
+            if echoed is not None:
+                stale_id = self._entries[echoed].message_id
+                self._entries[echoed] = message
+                # Drop the echo's provisional message_id so a re-delivered echo
+                # cannot later overwrite the durable row via identity lookup.
+                if stale_id != message.message_id:
+                    self._by_message_id.pop(stale_id, None)
+                self._reindex(echoed, message, request_id)
+                return
+
+        self._entries.append(message)
+        idx = len(self._entries) - 1
+        self._by_message_id[message.message_id] = idx
+        if request_id is not None:
+            self._by_request_id[request_id] = idx
+
+    def _reindex(
+        self,
+        idx: int,
+        message: GatewayMessage,
+        request_id: Optional[str],
+    ) -> None:
+        self._by_message_id[message.message_id] = idx
+        if request_id is not None:
+            self._by_request_id[request_id] = idx
+
+    # -- streaming handling ----------------------------------------------
+
+    def _apply_delta(self, event: GatewayEvent) -> None:
+        run_id = _run_id_of(event)
+        if run_id is None:
+            return
+        chunk = _delta_text_of(event)
+        current = self._runs.get(run_id)
+        if current is None:
+            current = RunView(run_id=run_id, text=chunk)
+        else:
+            current = replace(current, text=current.text + chunk)
+        self._runs[run_id] = current
+        self._runs.move_to_end(run_id)
+        self._evict_runs()
+
+    def _apply_stream_end(self, event: GatewayEvent) -> None:
+        run_id = _run_id_of(event)
+        if run_id is None:
+            return
+        current = self._runs.get(run_id) or RunView(run_id=run_id)
+        final_id = None
+        data = event.data or {}
+        final_id = data.get("message_id") or data.get("final_message_id")
+        current = replace(
+            current,
+            done=True,
+            final_message_id=str(final_id) if final_id else current.final_message_id,
+        )
+        self._runs[run_id] = current
+        self._runs.move_to_end(run_id)
+        self._evict_runs()
+
+    def _evict_runs(self) -> None:
+        if len(self._runs) <= self._max_tracked_runs:
+            return
+        # Evict oldest finished runs first; never evict an active stream.
+        for run_id in list(self._runs.keys()):
+            if len(self._runs) <= self._max_tracked_runs:
+                break
+            if self._runs[run_id].done:
+                del self._runs[run_id]
+
+
+# Streaming delta event types whose text is accumulated per run.
+_STREAM_DELTA_TYPES = frozenset(
+    {
+        EventType.TOKEN_STREAM.value,
+        "delta_text",
+    }
+)
+
+# Event types carrying a persisted transcript message.
+_MESSAGE_TYPES = frozenset(
+    {
+        EventType.MESSAGE.value,
+        "message_persisted",
+    }
+)
+
+
+def _delta_text_of(event: GatewayEvent) -> str:
+    data = event.data or {}
+    for key in ("delta", "text", "content", "token", "chunk"):
+        value = data.get(key)
+        if isinstance(value, str):
+            return value
+    return ""
+
+
+def _request_id_of(
+    event: GatewayEvent, message: GatewayMessage
+) -> Optional[str]:
+    data = event.data or {}
+    rid = data.get("request_id") or message.metadata.get("request_id")
+    return str(rid) if rid else None
+
+
+def _coerce_message(row: Any) -> Optional[GatewayMessage]:
+    if isinstance(row, GatewayMessage):
+        return row
+    if isinstance(row, dict):
+        payload = row
+        # A "message" event wraps the row under a "message" key in some frames.
+        if "message" in row and isinstance(row["message"], (dict, GatewayMessage)):
+            inner = row["message"]
+            return inner if isinstance(inner, GatewayMessage) else _coerce_message(inner)
+        if "content" not in payload and "message_id" not in payload:
+            return None
+        try:
+            return GatewayMessage.from_dict(payload)
+        except Exception:
+            return None
+    return None
+
+
+def _snapshot_rows(snapshot: Any) -> List[Any]:
+    if isinstance(snapshot, dict):
+        for key in ("messages", "entries", "transcript"):
+            rows = snapshot.get(key)
+            if isinstance(rows, list):
+                return rows
+        return []
+    for attr in ("messages", "entries", "transcript"):
+        rows = getattr(snapshot, attr, None)
+        if isinstance(rows, list):
+            return rows
+    return []
+
+
+def _snapshot_sequence(snapshot: Any) -> Optional[int]:
+    if isinstance(snapshot, dict):
+        for key in ("sequence", "cursor", "seq"):
+            value = snapshot.get(key)
+            if isinstance(value, int):
+                return value
+        return None
+    for attr in ("sequence", "cursor", "seq"):
+        value = getattr(snapshot, attr, None)
+        if isinstance(value, int):
+            return value
+    return None

src/praisonai-agents/praisonaiagents/hooks/registry.py

@@ -53,6 +53,9 @@ class HookRegistry:
         }
         self._enabled = enabled
         self._global_timeout = 60.0
+        # Monotonic counter giving each registered hook a stable secondary sort
+        # key so equal-priority hooks keep deterministic registration order.
+        self._seq_counter = 0
         # Guard mutation/iteration of the per-event hook lists. The default
         # registry is a cross-agent process-wide singleton, so a plugin
         # (un)registering a hook on one thread can race a get_hooks() iteration
@@ -80,6 +83,8 @@ class HookRegistry:
             The hook ID
         """
         with self._lock:
+            hook._seq = self._seq_counter
+            self._seq_counter += 1
             self._hooks[hook.event].append(hook)
         logger.debug(f"Registered hook '{hook.name}' for event '{hook.event.value}'")
         return hook.id
@@ -93,7 +98,8 @@ class HookRegistry:
         description: Optional[str] = None,
         sequential: bool = False,
         timeout: float = 60.0,
-        is_async: bool = False
+        is_async: bool = False,
+        priority: int = 100
     ) -> str:
         """
         Register a Python function as a hook.
@@ -107,6 +113,7 @@ class HookRegistry:
             sequential: Whether to run sequentially
             timeout: Timeout in seconds
             is_async: Whether the function is async
+            priority: Execution order (lower runs earlier). Default 100
             
         Returns:
             The hook ID
@@ -119,7 +126,8 @@ class HookRegistry:
             description=description,
             sequential=sequential,
             timeout=timeout,
-            is_async=is_async
+            is_async=is_async,
+            priority=priority
         )
         return self.register(hook)
     
@@ -133,7 +141,8 @@ class HookRegistry:
         sequential: bool = False,
         timeout: float = 60.0,
         env: Optional[Dict[str, str]] = None,
-        shell: bool = True
+        shell: bool = True,
+        priority: int = 100
     ) -> str:
         """
         Register a shell command as a hook.
@@ -148,6 +157,7 @@ class HookRegistry:
             timeout: Timeout in seconds
             env: Additional environment variables
             shell: Whether to run in shell mode
+            priority: Execution order (lower runs earlier). Default 100
             
         Returns:
             The hook ID
@@ -161,7 +171,8 @@ class HookRegistry:
             sequential=sequential,
             timeout=timeout,
             env=env or {},
-            shell=shell
+            shell=shell,
+            priority=priority
         )
         return self.register(hook)
     
@@ -170,7 +181,8 @@ class HookRegistry:
         event: HookEvent,
         matcher: Optional[str] = None,
         sequential: bool = False,
-        timeout: float = 60.0
+        timeout: float = 60.0,
+        priority: int = 100
     ) -> Callable:
         """
         Decorator to register a function as a hook.
@@ -180,6 +192,7 @@ class HookRegistry:
             matcher: Optional regex pattern to match targets
             sequential: Whether to run sequentially
             timeout: Timeout in seconds
+            priority: Execution order (lower runs earlier). Default 100
             
         Returns:
             Decorator function
@@ -199,7 +212,8 @@ class HookRegistry:
                 matcher=matcher,
                 sequential=sequential,
                 timeout=timeout,
-                is_async=is_async
+                is_async=is_async,
+                priority=priority
             )
             
             @wraps(func)
@@ -253,6 +267,10 @@ class HookRegistry:
         with self._lock:
             hooks = list(self._hooks.get(event, []))
         
+        # Sort by priority (lower runs earlier); ties fall back to registration
+        # order via the monotonic _seq for deterministic, reproducible ordering.
+        hooks.sort(key=lambda h: (h.priority, h._seq))
+        
         if target is None:
             return [h for h in hooks if h.enabled]
         
@@ -284,20 +302,25 @@ class HookRegistry:
             Dictionary mapping event names to hook info
         """
         result = {}
-        for event in HookEvent:
-            hooks = self._hooks[event]
-            if hooks:
-                result[event.value] = [
-                    {
-                        "id": h.id,
-                        "name": h.name,
-                        "type": "command" if isinstance(h, CommandHook) else "function",
-                        "matcher": h.matcher,
-                        "enabled": h.enabled,
-                        "sequential": h.sequential
-                    }
-                    for h in hooks
-                ]
+        with self._lock:
+            for event in HookEvent:
+                hooks = list(self._hooks[event])
+                if hooks:
+                    # Present hooks in effective execution order so an operator
+                    # can see the sequence get_hooks() will run them in.
+                    hooks.sort(key=lambda h: (h.priority, h._seq))
+                    result[event.value] = [
+                        {
+                            "id": h.id,
+                            "name": h.name,
+                            "type": "command" if isinstance(h, CommandHook) else "function",
+                            "matcher": h.matcher,
+                            "enabled": h.enabled,
+                            "sequential": h.sequential,
+                            "priority": h.priority
+                        }
+                        for h in hooks
+                    ]
         return result
     
     def enable_hook(self, hook_id: str) -> bool:
@@ -359,7 +382,7 @@ def set_default_registry(registry: HookRegistry):
 def add_hook(
     event: Union[str, HookEvent],
     callback: Optional[Callable[[HookInput], HookResult]] = None,
-    priority: int = 10,
+    priority: int = 100,
     matcher: Optional[str] = None
 ) -> Union[str, Callable]:
     """Register a hook callback. Simplified API.
@@ -376,7 +399,9 @@ def add_hook(
     Args:
         event: Hook event name ('before_tool', 'after_llm', etc.) or HookEvent enum
         callback: Function to call when hook fires (optional when using as decorator)
-        priority: Execution order (lower = earlier). Default 10. (Reserved for future use)
+        priority: Execution order (lower = earlier). Default 100 — same default
+            bucket as ``register_function``/``on``, so hooks added through either
+            API keep registration order unless an explicit priority is given.
         matcher: Optional regex pattern to match specific targets (e.g., tool names)
         
     Returns:
@@ -402,7 +427,8 @@ def add_hook(
             get_default_registry().register_function(
                 event=event,
                 func=func,
-                matcher=matcher
+                matcher=matcher,
+                priority=priority
             )
             return func
         return decorator
@@ -410,7 +436,8 @@ def add_hook(
     return get_default_registry().register_function(
         event=event,
         func=callback,
-        matcher=matcher
+        matcher=matcher,
+        priority=priority
     )
 
 def remove_hook(hook_id: str) -> bool:

src/praisonai-agents/praisonaiagents/hooks/runner.py

@@ -11,6 +11,7 @@ import time
 import asyncio
 from praisonaiagents._logging import get_logger
 import subprocess
+from concurrent.futures import ThreadPoolExecutor
 from typing import List, Optional, Dict, Any
 
 from .types import (
@@ -72,6 +73,47 @@ class HookRunner:
         self._registry = registry if registry is not None else HookRegistry()
         self._default_timeout = default_timeout
         self._cwd = cwd or os.getcwd()
+        # Dedicated executor for sync hooks. A hook that times out cannot be
+        # cancelled and keeps its worker slot; isolating it here means a
+        # misbehaving hook can only exhaust hook capacity, never the loop's
+        # shared default executor that guardrails/tools/locks also rely on.
+        self._hook_pool: Optional[ThreadPoolExecutor] = None
+        # hook.id -> in-flight Future for a sync hook. The slot is reserved
+        # before submission and only cleared when the worker finishes, so a
+        # still-running (timed-out or cancelled) hook caps new launches at one
+        # thread per hook. Guarded by ``_stuck_lock`` so concurrent invocations
+        # of the same hook cannot both reserve the slot and race past the cap.
+        self._stuck: Dict[str, Any] = {}
+        self._stuck_lock_obj: Optional[asyncio.Lock] = None
+
+    @property
+    def _stuck_lock(self) -> asyncio.Lock:
+        """Lazily create the in-flight-slot lock on first use.
+
+        Created lazily (not in ``__init__``) because a ``HookRunner`` may be
+        constructed before any event loop exists; the lock is only needed once
+        we are already inside ``async def execute`` / ``_execute_function_hook``.
+        """
+        if self._stuck_lock_obj is None:
+            self._stuck_lock_obj = asyncio.Lock()
+        return self._stuck_lock_obj
+
+    def _release_stuck(self, hook_id: str, fut: Any) -> None:
+        """Clear the in-flight slot once the worker Future completes.
+
+        Only removes the entry if it still points at *this* Future, so a newer
+        reserved invocation is never clobbered by an older worker draining.
+        """
+        if self._stuck.get(hook_id) is fut:
+            self._stuck.pop(hook_id, None)
+
+    def _get_hook_pool(self) -> ThreadPoolExecutor:
+        """Lazily create the dedicated sync-hook executor."""
+        if self._hook_pool is None:
+            self._hook_pool = ThreadPoolExecutor(
+                max_workers=8, thread_name_prefix="praison-hook"
+            )
+        return self._hook_pool
     
     @property
     def registry(self) -> HookRegistry:
@@ -102,25 +144,41 @@ class HookRunner:
         if not hooks:
             return []
         
-        # Separate sequential and parallel hooks
-        sequential_hooks = [h for h in hooks if h.sequential]
-        parallel_hooks = [h for h in hooks if not h.sequential]
-        
-        results = []
-        
-        # Execute parallel hooks first
-        if parallel_hooks:
-            parallel_results = await self._execute_parallel(
-                parallel_hooks, event, input_data
-            )
-            results.extend(parallel_results)
-        
-        # Execute sequential hooks
-        if sequential_hooks:
-            sequential_results = await self._execute_sequential(
-                sequential_hooks, event, input_data
-            )
-            results.extend(sequential_results)
+        # The registry hands us hooks already sorted by (priority, _seq), so the
+        # list is in the exact order the operator asked them to run. Preserve
+        # that global order across parallel/sequential boundaries: walk the list
+        # and run each *contiguous* run of same-mode hooks together. A redact
+        # (sequential, priority 10) that must precede a trace (parallel,
+        # priority 20) therefore still runs first — splitting all parallel hooks
+        # to the front would have leaked the unredacted input to the tracer.
+        # Adjacent parallel hooks are still gathered concurrently; only their
+        # position relative to a lower-priority sequential hook is respected.
+        results: List[HookExecutionResult] = []
+        current_input = input_data
+        i = 0
+        n = len(hooks)
+        while i < n:
+            j = i + 1
+            is_seq = hooks[i].sequential
+            while j < n and hooks[j].sequential == is_seq:
+                j += 1
+            run = hooks[i:j]
+            if is_seq:
+                run_results = await self._execute_sequential(run, event, current_input)
+                # A sequential run may have blocked; stop the whole chain so a
+                # later parallel group never runs after a denial (matches the
+                # sequential-hook break semantics).
+                results.extend(run_results)
+                if any(
+                    r.output and r.output.is_denied() for r in run_results
+                ):
+                    break
+            else:
+                run_results = await self._execute_parallel(
+                    run, event, current_input
+                )
+                results.extend(run_results)
+            i = j
         
         return results
     
@@ -151,27 +209,30 @@ class HookRunner:
         Returns:
             List of execution results
         """
-        try:
-            loop = asyncio.get_running_loop()
-        except RuntimeError:
-            loop = None
-
-        if loop is not None and loop.is_running():
-            # Cannot execute sync in running event loop - would block the loop
-            # Guide users to use the async API instead
-            raise RuntimeError(
-                "execute_sync() cannot be called from within a running event loop. "
-                "Use 'await runner.execute(event, input_data, target)' instead in async contexts."
-            )
-
         # Fast path: no hooks → no loop creation needed
         # This mirrors the same check in execute() but avoids event loop creation overhead
         hooks = self._registry.get_hooks(event, target)
         if not hooks:
             return []
 
-        # No running loop and hooks exist — safe to create one
-        return asyncio.run(self.execute(event, input_data, target, _hooks=hooks))
+        # Use the canonical sync-to-async bridge so blocking policy/guardrail
+        # hooks still run (and can deny) from a running event loop — e.g. a
+        # FastAPI/aiohttp handler, Jupyter, or a bot callback invoking
+        # agent.start()/agent.chat(). Raising here would turn documented sync
+        # entry points into hard failures whenever any hook is registered.
+        #
+        # Bound the bridge wait so a hook that ignores its own per-hook timeout
+        # (e.g. a blocking C call on the worker loop) can never pin the caller's
+        # thread forever. Each hook already enforces ``default_timeout``
+        # internally; we give the whole chain headroom of one timeout per hook
+        # so legitimately slow sequential chains are not cut short, while still
+        # providing a hard backstop instead of ``timeout=None``.
+        from ..utils.async_bridge import run_coroutine_from_any_context
+        bridge_timeout = max(self._default_timeout, 1.0) * (len(hooks) + 1)
+        return run_coroutine_from_any_context(
+            self.execute(event, input_data, target, _hooks=hooks),
+            timeout=bridge_timeout,
+        )
     
     async def _execute_parallel(
         self,
@@ -378,11 +439,52 @@ class HookRunner:
                     timeout=timeout
                 )
             else:
-                # Execute sync function in thread pool with timeout
-                loop = asyncio.get_event_loop()
+                # Reserve the single in-flight slot for this hook *under a lock*
+                # and *before* submitting, so two concurrent invocations can't
+                # both see an empty ``_stuck`` and both launch a worker (which
+                # would exceed the intended one-worker-per-hook cap and double
+                # the side effects). The slot is only cleared once the worker
+                # finishes; a still-running previous worker fails this one
+                # closed immediately instead of leaking another thread.
+                async with self._stuck_lock:
+                    prev = self._stuck.get(hook.id)
+                    if prev is not None and not prev.done():
+                        duration = (time.time() - start_time) * 1000
+                        return HookExecutionResult(
+                            hook_id=hook.id,
+                            hook_name=hook.name or "unknown",
+                            event=event,
+                            success=False,
+                            error="previous invocation still running past its timeout",
+                            duration_ms=duration,
+                            output=HookResult(
+                                decision="deny",
+                                reason=f"Hook '{hook.name}' is still running past its timeout",
+                            ),
+                        )
+
+                    # Execute sync function in the dedicated hook pool with
+                    # timeout. ``copy_context_to_callable`` preserves
+                    # trace/session contextvars across the thread, matching
+                    # other executor sites.
+                    from ..trace.context_events import copy_context_to_callable
+                    loop = asyncio.get_event_loop()
+                    func = hook.func
+                    data = input_data
+                    fut = loop.run_in_executor(
+                        self._get_hook_pool(),
+                        copy_context_to_callable(lambda: func(data)),
+                    )
+                    # Reserve the slot now; a completion callback clears it only
+                    # when the worker actually finishes, so a timeout *or* a
+                    # cancellation leaves the slot held until the thread drains.
+                    self._stuck[hook.id] = fut
+                    fut.add_done_callback(
+                        lambda f, hid=hook.id: self._release_stuck(hid, f)
+                    )
+
                 result = await asyncio.wait_for(
-                    loop.run_in_executor(None, hook.func, input_data),
-                    timeout=timeout
+                    asyncio.shield(fut), timeout=timeout
                 )
             
             duration = (time.time() - start_time) * 1000

src/praisonai-agents/praisonaiagents/hooks/types.py

@@ -316,6 +316,8 @@ class HookDefinition:
     name: Optional[str] = None
     description: Optional[str] = None
     timeout: float = 60.0  # Timeout in seconds
+    priority: int = 100  # Execution order (lower runs earlier)
+    _seq: int = field(default=0, repr=False)  # Registration order for stable ties
     
     def matches(self, target: str) -> bool:
         """Check if this hook matches the target (tool name, etc.)."""

src/praisonai-agents/praisonaiagents/knowledge/adapters/factories.py

@@ -211,6 +211,8 @@ class ChromaKnowledgeAdapter:
                **kwargs: Any):
         """Search for relevant content in ChromaDB."""
         from ..models import SearchResult, SearchResultItem
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "search", backend="chroma")
         
         # Get embedding for query
         embedding_model, embedding_kwargs = self._embedding_call()
@@ -236,9 +238,15 @@ class ChromaKnowledgeAdapter:
         if run_id is not None:
             where_filter["run_id"] = run_id
         
-        # Merge with additional filters
+        # Merge with additional filters, but never let a caller-supplied filter
+        # overwrite the tenant-scope constraint: filters={"user_id": "bob"} must
+        # not widen an already-scoped query back onto another tenant's data
+        # (issue #5319).
         if filters:
-            where_filter.update(filters)
+            for key, value in filters.items():
+                if key in where_filter:
+                    continue
+                where_filter[key] = value
         
         # Search ChromaDB
         try:
@@ -289,6 +297,8 @@ class ChromaKnowledgeAdapter:
             **kwargs: Any):
         """Add content to ChromaDB."""
         from ..models import AddResult
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "add", backend="chroma")
         
         # Convert content to string
         content_str = str(content)
@@ -375,6 +385,8 @@ class ChromaKnowledgeAdapter:
                 run_id: Optional[str] = None, limit: int = 100, **kwargs: Any):
         """Get all items from ChromaDB."""
         from ..models import SearchResult, SearchResultItem
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "get_all", backend="chroma")
         
         try:
             # ChromaDB doesn't support get_all with filters easily, so we'll use peek
@@ -510,8 +522,11 @@ class SQLiteKnowledgeAdapter:
                **kwargs: Any):
         """Search for relevant content in SQLite."""
         from ..models import SearchResult, SearchResultItem
+        from ..protocols import require_scope
         import json
         
+        require_scope(user_id, agent_id, run_id, "search", backend="sqlite")
+        
         conn = self._get_conn()
         
         # Build query with filters
@@ -528,6 +543,19 @@ class SQLiteKnowledgeAdapter:
             sql += " AND run_id = ?"
             params.append(run_id)
         
+        # Apply metadata filters against the stored JSON metadata column so a
+        # filtered search returns the same subset chroma/mem0 already return
+        # instead of silently discarding the filters (issue #5319).
+        if filters:
+            for key, value in filters.items():
+                # Quote the key in the JSON path so a top-level metadata key that
+                # itself contains dots (e.g. "source.type") is matched literally
+                # instead of being interpreted as a nested path (issue #5319).
+                escaped_key = key.replace('"', '""')
+                sql += " AND json_extract(metadata, ?) = ?"
+                params.append(f'$."{escaped_key}"')
+                params.append(value)
+        
         sql += " LIMIT ?"
         params.append(limit)
         
@@ -551,9 +579,12 @@ class SQLiteKnowledgeAdapter:
             **kwargs: Any):
         """Add content to SQLite."""
         from ..models import AddResult
+        from ..protocols import require_scope
         import json
         import time
         
+        require_scope(user_id, agent_id, run_id, "add", backend="sqlite")
+        
         content_str = str(content)
         doc_id = str(time.time_ns())
         metadata_json = json.dumps(metadata or {})
@@ -596,8 +627,11 @@ class SQLiteKnowledgeAdapter:
                 run_id: Optional[str] = None, limit: int = 100, **kwargs: Any):
         """Get all items from SQLite."""
         from ..models import SearchResult, SearchResultItem
+        from ..protocols import require_scope
         import json
         
+        require_scope(user_id, agent_id, run_id, "get_all", backend="sqlite")
+        
         conn = self._get_conn()
         
         # Build query with filters

src/praisonai-agents/praisonaiagents/knowledge/adapters/mongodb_adapter.py

@@ -169,7 +169,12 @@ class MongoDBKnowledgeAdapter:
             
         Returns:
             Document ID of inserted content
+
+        Raises:
+            ScopeRequiredError: If no scope identifier is provided
         """
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "add", backend="mongodb")
         try:
             # Generate embedding if model is available
             embedding = None
@@ -210,6 +215,7 @@ class MongoDBKnowledgeAdapter:
         user_id: Optional[str] = None,
         agent_id: Optional[str] = None,
         run_id: Optional[str] = None,
+        filters: Optional[Dict[str, Any]] = None,
         **kwargs
     ) -> List[Dict[str, Any]]:
         """
@@ -221,11 +227,17 @@ class MongoDBKnowledgeAdapter:
             user_id: Optional user scope for tenant isolation
             agent_id: Optional agent scope for tenant isolation
             run_id: Optional run scope for tenant isolation
+            filters: Optional metadata filters applied to the query
             **kwargs: Additional search parameters
             
         Returns:
             List of matching documents
+
+        Raises:
+            ScopeRequiredError: If no scope identifier is provided
         """
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "search", backend="mongodb")
         try:
             results = []
 
@@ -237,6 +249,13 @@ class MongoDBKnowledgeAdapter:
                     "run_id": run_id,
                 }.items() if v is not None
             }
+
+            # Fold metadata filters in alongside the scope so a filtered search
+            # returns the same subset chroma/mem0 already return instead of
+            # silently discarding the filters (issue #5319).
+            if filters:
+                for key, value in filters.items():
+                    scope_filter[f"metadata.{key}"] = value
             
             # Try vector search first if available
             if self.use_vector_search and self.embedding_model and self._is_atlas_connection():
@@ -332,6 +351,127 @@ class MongoDBKnowledgeAdapter:
             logger.error(f"Failed to delete document {document_id}: {e}")
             return False
     
+    def get(self, item_id: str, **kwargs):
+        """Get a specific document by ID from the knowledge store.
+
+        Returns a ``SearchResultItem`` (or ``None``) so the result shape matches
+        the shared KnowledgeStoreProtocol and the other adapters — a caller that
+        reads ``.text``/``.metadata`` or calls ``.to_dict()`` must not break when
+        it switches to the MongoDB backend (issue #5319).
+        """
+        from ..models import SearchResultItem
+        try:
+            from bson import ObjectId
+            doc = self.collection.find_one({"_id": ObjectId(item_id)})
+            if not doc:
+                return None
+            return SearchResultItem(
+                id=str(doc["_id"]),
+                text=doc.get("content", ""),
+                metadata=doc.get("metadata", {}) or {},
+                score=1.0,
+            )
+        except Exception as e:
+            logger.error(f"Failed to get document {item_id}: {e}")
+            return None
+
+    def get_all(
+        self,
+        *,
+        user_id: Optional[str] = None,
+        agent_id: Optional[str] = None,
+        run_id: Optional[str] = None,
+        limit: int = 100,
+        **kwargs
+    ):
+        """Get all documents matching the given scope.
+
+        Returns a ``SearchResult`` to match the shared KnowledgeStoreProtocol
+        and the other adapters.
+
+        Raises:
+            ScopeRequiredError: If no scope identifier is provided
+        """
+        from ..models import SearchResult, SearchResultItem
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "get_all", backend="mongodb")
+        try:
+            scope_filter = {
+                k: v for k, v in {
+                    "user_id": user_id,
+                    "agent_id": agent_id,
+                    "run_id": run_id,
+                }.items() if v is not None
+            }
+            items = []
+            for doc in self.collection.find(scope_filter).limit(limit):
+                items.append(SearchResultItem(
+                    id=str(doc["_id"]),
+                    text=doc.get("content", ""),
+                    metadata=doc.get("metadata", {}) or {},
+                    score=1.0,
+                ))
+            return SearchResult(results=items)
+        except Exception as e:
+            logger.error(f"Failed to get_all from MongoDB: {e}")
+            return SearchResult(results=[])
+
+    def update(self, item_id: str, content: Any, **kwargs):
+        """Update an existing document's content/metadata by ID.
+
+        Returns an ``AddResult`` to match the shared KnowledgeStoreProtocol and
+        the other adapters (issue #5319).
+        """
+        from ..models import AddResult
+        try:
+            from bson import ObjectId
+            update_fields: Dict[str, Any] = {"content": str(content)}
+            if "metadata" in kwargs and kwargs["metadata"] is not None:
+                update_fields["metadata"] = kwargs["metadata"]
+            result = self.collection.update_one(
+                {"_id": ObjectId(item_id)},
+                {"$set": update_fields}
+            )
+            if result.matched_count > 0:
+                return AddResult(success=True, id=item_id)
+            return AddResult(success=False, message=f"No document matched id {item_id}")
+        except Exception as e:
+            logger.error(f"Failed to update document {item_id}: {e}")
+            return AddResult(success=False, message=str(e))
+
+    def delete_all(
+        self,
+        *,
+        user_id: Optional[str] = None,
+        agent_id: Optional[str] = None,
+        run_id: Optional[str] = None,
+        **kwargs
+    ) -> bool:
+        """Delete all documents matching the given scope.
+
+        Enforces the same tenant-scope contract as the mem0 backend: an
+        unscoped call raises ``ScopeRequiredError`` instead of silently wiping
+        every tenant's documents from the (typically shared) collection.
+
+        Raises:
+            ScopeRequiredError: If no scope identifier is provided
+        """
+        from ..protocols import require_scope
+        require_scope(user_id, agent_id, run_id, "delete_all", backend="mongodb")
+        try:
+            scope_filter = {
+                k: v for k, v in {
+                    "user_id": user_id,
+                    "agent_id": agent_id,
+                    "run_id": run_id,
+                }.items() if v is not None
+            }
+            self.collection.delete_many(scope_filter)
+            return True
+        except Exception as e:
+            logger.error(f"Failed to delete_all from MongoDB: {e}")
+            return False
+
     def close(self):
         """Close MongoDB connection."""
         if hasattr(self, 'client'):

src/praisonai-agents/praisonaiagents/llm/adapters/__init__.py

@@ -207,9 +207,10 @@ def _recover_json_tool_calls(response_text: str, tools: List[Dict[str, Any]]) ->
         if isinstance(response_json, list):
             tool_calls: List[Dict[str, Any]] = []
             for idx, tool_json in enumerate(response_json):
-                if isinstance(tool_json, dict) and tool_json.get("name") in allowed:
+                name = tool_json.get("name") if isinstance(tool_json, dict) else None
+                if name:
                     tool_calls.append(_make_tool_call(
-                        tool_json["name"], tool_json.get("arguments", {}), response_text, idx))
+                        name, tool_json.get("arguments", {}), response_text, idx))
             if tool_calls:
                 return tool_calls
     except (json.JSONDecodeError, TypeError, KeyError):

src/praisonai-agents/praisonaiagents/llm/error_classifier.py

@@ -9,6 +9,7 @@ Provides both legacy API (classify_error) and new structured classification
 """
 
 import re
+import math
 import random
 from dataclasses import dataclass
 from enum import Enum
@@ -588,12 +589,48 @@ def get_retry_delay(category: ErrorCategory, attempt: int = 1, base_delay: float
     return 0
 
 
-def extract_retry_after(error: Exception) -> Optional[float]:
+def _retry_after_http_date_delay(value: str) -> Optional[float]:
+    """Return seconds until an HTTP-date ``Retry-After`` value, else ``None``.
+
+    RFC 7231 allows ``Retry-After`` to be either delta-seconds or an absolute
+    HTTP-date. Delta-seconds is handled by the caller; this parses the date
+    form using the stdlib (no new dependency) and returns the non-negative
+    delay from now. A value in the past or unparseable yields ``None`` so the
+    caller falls through to its other signals.
+    """
+    try:
+        from email.utils import parsedate_to_datetime
+    except Exception:  # pragma: no cover - stdlib always present
+        return None
+    try:
+        when = parsedate_to_datetime(value.strip())
+    except (TypeError, ValueError):
+        return None
+    if when is None:
+        return None
+    import time as _time
+    from datetime import timezone
+
+    # An HTTP-date is UTC; a naive parse result is treated as UTC per the RFC.
+    if when.tzinfo is None:
+        when = when.replace(tzinfo=timezone.utc)
+    delay = when.timestamp() - _time.time()
+    return delay if delay > 0 else None
+
+
+def extract_retry_after(
+    error: Exception, cap_seconds: float = 300.0,
+) -> Optional[float]:
     """Extract Retry-After header value from rate limit errors.
-    
+
     Args:
         error: Exception potentially containing Retry-After info
-        
+        cap_seconds: Upper bound applied to the parsed value. Defaults to 300s
+            (5 minutes) for in-tick backoff callers. A scheduler *hold* passes a
+            larger cap so a job can be parked past a long provider reset window
+            (e.g. an hourly quota) instead of re-firing every tick — capping at
+            5 minutes there would defeat the park.
+
     Returns:
         Delay in seconds if found, None otherwise
     """
@@ -608,31 +645,60 @@ def extract_retry_after(error: Exception) -> Optional[float]:
             retry_after = None
         if retry_after is not None:
             try:
-                return min(float(retry_after), 300.0)  # Cap at 5 minutes
+                delay = float(retry_after)
+                if math.isfinite(delay) and delay >= 0:
+                    return min(delay, cap_seconds)
             except (ValueError, TypeError):
-                pass  # Not a plain number (could be an HTTP-date); fall through
+                # Not a plain delta-seconds value — RFC 7231 also permits an
+                # HTTP-date. Parse it so a provider that sends an absolute reset
+                # instant still yields a usable delay (else a long quota window
+                # would leave the job un-parked). A past/unparseable date yields
+                # no delay and falls through to the other signals.
+                delay = _retry_after_http_date_delay(str(retry_after))
+                if delay is not None:
+                    return min(delay, cap_seconds)
 
     # 2. Some SDKs expose a numeric ``retry_after`` attribute directly.
     retry_after_attr = getattr(error, "retry_after", None)
     if isinstance(retry_after_attr, (int, float)):
-        return min(float(retry_after_attr), 300.0)
+        try:
+            delay = float(retry_after_attr)
+        except OverflowError:
+            delay = float("inf")
+        if math.isfinite(delay) and delay >= 0:
+            return min(delay, cap_seconds)
 
     error_str = str(error)
     
     # 3. Fall back to parsing common Retry-After patterns from the message.
+    #    Capture the full signed token so a negative hint (e.g. "-3600 seconds")
+    #    is rejected by the finite/non-negative check below instead of being
+    #    silently read as a positive delay. Match a whole numeric token so
+    #    leading-dot fractions, exponents and malformed suffixes cannot be
+    #    mistaken for an unrelated positive integer.
+    # A period may terminate a sentence, but may not split an embedded token.
+    number = (
+        r"(?<![\w.+-])([+-]?(?:\d+(?:\.\d+)?|\.\d+)(?:[eE][+-]?\d+)?)"
+        r"(?![\deE+-]|\.(?!\s|$|[\"')\]}])|\.\s*(?:seconds?|s)\b)"
+    )
     patterns = [
-        r"retry.?after[:\s]+(\d+)",
-        r"retry[:\s]+(\d+)",
-        r"wait[:\s]+(\d+)",
-        r"(\d+).*second",
+        # Quoted duration values must close immediately after the unit.
+        r'"?retryDelay"?\s*:\s*"' + number + r's"',
+        r'"?retryDelay"?\s*:\s*' + number + r's(?=$|[\s,;}\]])',
+        r"try again in\s+" + number,
+        r"retry.?after[:\s]+" + number,
+        r"retry[:\s]+" + number,
+        r"wait[:\s]+" + number,
+        number + r"\s*second",
     ]
     
     for pattern in patterns:
-        match = re.search(pattern, error_str, re.IGNORECASE)
-        if match:
+        # Invalid earlier hints must not hide a later valid hint of this form.
+        for match in re.finditer(pattern, error_str, re.IGNORECASE):
             try:
                 delay = float(match.group(1))
-                return min(delay, 300.0)  # Cap at 5 minutes
+                if math.isfinite(delay) and delay >= 0:
+                    return min(delay, cap_seconds)
             except (ValueError, IndexError):
                 continue
     

src/praisonai-agents/praisonaiagents/llm/failover.py

@@ -359,10 +359,17 @@ class FailoverManager:
                 if profile.cooldown_until and current_time >= profile.cooldown_until:
                     profile.reset()
             
-            # Find first available profile
-            for profile in self._profiles:
-                if profile.is_available:
-                    return profile
+            # Find available profiles (already priority-sorted by add_profile).
+            available = [p for p in self._profiles if p.is_available]
+            if available:
+                if self.config.rotate_on_success:
+                    # Round-robin among the current highest-priority tier only,
+                    # so equal-priority credentials configured to spread load
+                    # actually share traffic instead of one absorbing all of it.
+                    top_priority = available[0].priority
+                    tier = [p for p in available if p.priority == top_priority]
+                    return tier[self._current_index % len(tier)]
+                return available[0]
             
             # If none available, return the one with shortest remaining cooldown
             available_profiles = [p for p in self._profiles if p.status != ProviderStatus.DISABLED]
@@ -443,6 +450,10 @@ class FailoverManager:
             profile: The profile that succeeded
         """
         with self._lock:
+            # Advance the round-robin cursor so the next get_next_profile()
+            # hands work to a different equal-priority credential.
+            if self.config.rotate_on_success:
+                self._current_index += 1
             if profile.status != ProviderStatus.AVAILABLE:
                 # Snapshot the cooldown this success is recovering from BEFORE
                 # resetting, so we can avoid clobbering a newer fleet-wide bench.

src/praisonai-agents/praisonaiagents/llm/openai_client.py

@@ -781,7 +781,9 @@ class OpenAIClient:
                         "output": msg.get("content", ""),
                     })
                 else:
-                    item = dict(msg)
+                    # Persisted history may carry local metadata/provider fields.
+                    # Responses message items accept role/content, not that record.
+                    item = {"role": role}
                     item["content"] = self._build_responses_content(
                         msg.get("content", "")
                     )
@@ -896,16 +898,52 @@ class OpenAIClient:
                 responses_content.append(part)
         return responses_content
 
+    @staticmethod
+    def _responses_incomplete_finish_reason(response) -> Optional[str]:
+        """Map known incomplete reasons to existing Chat Completions outcomes."""
+        status = response.get("status") if isinstance(response, dict) else getattr(response, "status", None)
+        if status != "incomplete":
+            return None
+        details = response.get("incomplete_details") if isinstance(response, dict) else getattr(response, "incomplete_details", None)
+        reason = details.get("reason") if isinstance(details, dict) else getattr(details, "reason", None)
+        return {"max_output_tokens": "length", "content_filter": "content_filter"}.get(reason)
+
+    @staticmethod
+    def _extract_responses_refusal(response) -> Optional[str]:
+        """Preserve refusal explanations separately from ordinary output text."""
+        output = response.get("output", []) if isinstance(response, dict) else getattr(response, "output", [])
+        refusals = []
+        for item in output or []:
+            item_type = item.get("type") if isinstance(item, dict) else getattr(item, "type", None)
+            if item_type != "message":
+                continue
+            content = item.get("content", []) if isinstance(item, dict) else getattr(item, "content", [])
+            for block in content or []:
+                block_type = block.get("type") if isinstance(block, dict) else getattr(block, "type", None)
+                if block_type == "refusal":
+                    refusal = block.get("refusal") if isinstance(block, dict) else getattr(block, "refusal", None)
+                    if isinstance(refusal, str) and refusal:
+                        refusals.append(refusal)
+        return "".join(refusals) or None
+
     def _responses_to_chat_completion(self, response) -> ChatCompletion:
         """
         Wrap a Responses API response into a ChatCompletion dataclass
         so that all downstream code (tool loop, display, callbacks)
         works without modification.
         """
+        status = response.get("status") if isinstance(response, dict) else getattr(response, "status", None)
+        if status == "failed":
+            from .llm import LLMResponseError
+            error = response.get("error") if isinstance(response, dict) else getattr(response, "error", None)
+            code = error.get("code") if isinstance(error, dict) else getattr(error, "code", None)
+            message = error.get("message") if isinstance(error, dict) else getattr(error, "message", None)
+            label = f"Responses API failed ({code})" if code else "Responses API failed"
+            raise LLMResponseError(f"{label}: {message or 'Provider returned a failed response.'}")
         response_text = ""
         tool_calls_list: List[ToolCall] = []
 
-        output_items = getattr(response, 'output', None) or []
+        output_items = (response.get('output') if isinstance(response, dict) else getattr(response, 'output', None)) or []
         for item in output_items:
             item_type = getattr(item, "type", "") if not isinstance(item, dict) else item.get("type", "")
 
@@ -940,22 +978,26 @@ class OpenAIClient:
                 ))
 
         # Build usage
-        raw_usage = getattr(response, 'usage', None)
+        raw_usage = response.get('usage') if isinstance(response, dict) else getattr(response, 'usage', None)
         usage = None
         if raw_usage:
+            def usage_value(name):
+                return raw_usage.get(name, 0) if isinstance(raw_usage, dict) else getattr(raw_usage, name, 0)
+
             usage = CompletionUsage(
-                prompt_tokens=getattr(raw_usage, 'input_tokens', 0),
-                completion_tokens=getattr(raw_usage, 'output_tokens', 0),
-                total_tokens=getattr(raw_usage, 'total_tokens', 0),
+                prompt_tokens=usage_value('input_tokens'),
+                completion_tokens=usage_value('output_tokens'),
+                total_tokens=usage_value('total_tokens'),
             )
 
         message = ChatCompletionMessage(
             content=response_text if response_text else None,
             role="assistant",
+            refusal=self._extract_responses_refusal(response),
             tool_calls=tool_calls_list if tool_calls_list else None,
         )
         choice = Choice(
-            finish_reason="tool_calls" if tool_calls_list else "stop",
+            finish_reason=self._responses_incomplete_finish_reason(response) or ("tool_calls" if tool_calls_list else "stop"),
             index=0,
             message=message,
         )
@@ -1053,6 +1095,7 @@ class OpenAIClient:
         Returns:
             ChatCompletion object or None if error
         """
+        raw = None
         # Lazy import StreamEvent types only when needed
         _emit = emit_events and stream_callback is not None
         if _emit:
@@ -1112,6 +1155,9 @@ class OpenAIClient:
                 except (FileNotFoundError, ValueError):
                     raise
                 except Exception as e:
+                    status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+                    if status == "failed":
+                        raise
                     self.logger.warning(f"Responses API streaming failed, falling back: {e}")
                     # Fall through to Chat Completions streaming
             
@@ -1282,6 +1328,9 @@ class OpenAIClient:
             return final_response
             
         except Exception as e:
+            status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+            if status == "failed":
+                raise
             self.logger.error(f"Error in stream processing: {e}")
             return None
     
@@ -1318,6 +1367,7 @@ class OpenAIClient:
         Returns:
             ChatCompletion object or None if error
         """
+        raw = None
         # Lazy import StreamEvent types only when needed
         _emit = emit_events and stream_callback is not None
         if _emit:
@@ -1386,6 +1436,9 @@ class OpenAIClient:
                 except (FileNotFoundError, ValueError):
                     raise
                 except Exception as e:
+                    status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+                    if status == "failed":
+                        raise
                     self.logger.warning(f"Responses API async streaming failed, falling back: {e}")
                     # Fall through to Chat Completions streaming
             
@@ -1548,6 +1601,9 @@ class OpenAIClient:
             return final_response
             
         except Exception as e:
+            status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+            if status == "failed":
+                raise
             self.logger.error(f"Error in async stream processing: {e}")
             return None
 
@@ -1652,6 +1708,7 @@ class OpenAIClient:
         """
         # ── Responses API path (non-streaming only) ────────────────────
         if not stream and self._use_responses_api(model):
+            raw = None
             try:
                 resp_params = self._build_responses_input(
                     messages, model, temperature, tools,
@@ -1662,6 +1719,9 @@ class OpenAIClient:
             except (FileNotFoundError, ValueError):
                 raise
             except Exception as e:
+                status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+                if status == "failed":
+                    raise
                 self.logger.warning(f"Responses API failed, falling back to Chat Completions: {e}")
                 # Fall through to Chat Completions
 
@@ -1716,6 +1776,7 @@ class OpenAIClient:
         """
         # ── Responses API path (non-streaming only) ────────────────────
         if not stream and self._use_responses_api(model):
+            raw = None
             try:
                 resp_params = self._build_responses_input(
                     messages, model, temperature, tools,
@@ -1726,6 +1787,9 @@ class OpenAIClient:
             except (FileNotFoundError, ValueError):
                 raise
             except Exception as e:
+                status = raw.get("status") if isinstance(raw, dict) else getattr(raw, "status", None)
+                if status == "failed":
+                    raise
                 self.logger.warning(f"Responses API failed, falling back to Chat Completions: {e}")
                 # Fall through to Chat Completions
 
@@ -2654,13 +2718,23 @@ class OpenAIClient:
                                 tool_result, messages, _tool_call_id, function_name,
                                 history_sink=deferred_history_sink,
                             )
-                            results_str = json.dumps(tool_result) if tool_result else "Function returned an empty output"
                         except ToolExecutionError:
                             raise
                         except Exception as e:
-                            results_str = f"Error executing function: {str(e)}"
+                            tool_result = {"error": str(e)}
                             if verbose:
                                 yield f"\n[Function error: {str(e)}]"
+                        # Serialize the result separately from execution so a tool
+                        # that SUCCEEDS but returns a non-JSON-serializable value
+                        # (datetime, set, bytes, DataFrame, custom object) is not
+                        # misreported as an execution error -- parity with the
+                        # sync/async non-streaming loops' (TypeError, ValueError)
+                        # fallback.
+                        try:
+                            results_str = json.dumps(tool_result) if tool_result else "Function returned an empty output"
+                        except (TypeError, ValueError):
+                            tool_result = {"result": str(tool_result)}
+                            results_str = json.dumps(tool_result)
                         
                         # Trigger callback with result
                         display_tool_call_fn(f"Function {function_name} returned: {results_str[:200]}{'...' if len(results_str) > 200 else ''}", console=None)

src/praisonai-agents/praisonaiagents/llm/rate_limiter.py

@@ -84,6 +84,7 @@ class RateLimiter:
 
     def __post_init__(self):
         """Initialize internal state."""
+        self.max_retry_delay = max(0, self.max_retry_delay)
         # Request-based limiting
         if self.requests_per_minute is not None:
             self._tokens = float(self.burst)

src/praisonai-agents/praisonaiagents/local/capabilities.py

@@ -53,6 +53,7 @@ class Cap(str, Enum):
     STREAMING_WITH_TOOLS = "streaming_with_tools"
     PARALLEL_TOOL_CALLS = "parallel_tool_calls"
     EMBEDDINGS_ENDPOINT = "embeddings_endpoint"
+    SYSTEM_ONE = "system_one"
 
 
 class Evidence(str, Enum):
@@ -105,6 +106,9 @@ _OLLAMA_CAP_MAP = {
     "thinking": (Cap.THINKING,),
     "embedding": (Cap.EMBEDDINGS_ENDPOINT,),
     "audio": (Cap.AUDIO_IN,),
+    "decision": (Cap.SYSTEM_ONE,),
+    "systemone": (Cap.SYSTEM_ONE,),
+    "system_one": (Cap.SYSTEM_ONE,),
 }
 
 

src/praisonai-agents/praisonaiagents/mcp/mcp.py

@@ -210,7 +210,27 @@ class MCPToolRunner(threading.Thread):
                 return f"Error: MCP tool call timed out after {self.timeout} seconds"
             if not success:
                 return f"Error: {result}"
-            
+
+            # A tool that fails at the application level returns a normal
+            # CallToolResult with isError=True per the MCP spec. Surface it as a
+            # structured error so retry/circuit-breaker/hook logic treats it as a
+            # failure instead of a successful string result (issue #5319).
+            if getattr(result, 'isError', False):
+                if hasattr(result, 'content') and result.content:
+                    if hasattr(result.content[0], 'text'):
+                        error_text = result.content[0].text
+                    else:
+                        error_text = str(result.content[0])
+                else:
+                    error_text = str(result)
+                # Guarantee a non-empty error string: an isError result with an
+                # empty text block would otherwise yield {"error": ""}, which
+                # retry/circuit-breaker/hook logic can misread as a success
+                # (issue #5319).
+                if not error_text:
+                    error_text = f"MCP tool '{tool_name}' reported an error (isError=True)"
+                return {"error": error_text, "mcp_error": True}
+
             # Process result
             if hasattr(result, 'content') and result.content:
                 if hasattr(result.content[0], 'text'):
@@ -401,8 +421,8 @@ class MCP:
         base = re.sub(r'\.(cmd|exe|bat|ps1)$', '', base, flags=re.IGNORECASE).lower()
         return base in cls._COLD_START_LAUNCHERS
 
-    def __init__(self, command_or_string=None, args=None, *, command=None, timeout=None, debug=False, 
-                 allowed_tools: Optional[List[str]] = None, disabled_tools: Optional[List[str]] = None, **kwargs):
+    def __init__(self, command_or_string=None, args=None, *, command=None, timeout=None, debug=False,
+                 allowed_tools: Optional[List[str]] = None, disabled_tools: Optional[List[str]] = None, max_response_bytes: Optional[int] = None, **kwargs):
         """
         Initialize the MCP connection and get tools.
         
@@ -422,6 +442,7 @@ class MCP:
             debug: Enable debug logging for MCP operations (default: False)
             allowed_tools: Include whitelist - only these tools will be available (default: None = all tools)
             disabled_tools: Exclude blacklist - these tools will be filtered out (default: None = no exclusions)
+            max_response_bytes: Maximum body size for each Streamable HTTP response.
             **kwargs: Additional parameters for StdioServerParameters
             
         Note:
@@ -439,6 +460,12 @@ class MCP:
         if command_or_string is None and command is not None:
             command_or_string = command
 
+        if max_response_bytes is not None:
+            if isinstance(max_response_bytes, bool) or not isinstance(max_response_bytes, int) or max_response_bytes <= 0:
+                raise ValueError("max_response_bytes must be a positive integer")
+            if not isinstance(command_or_string, str) or not re.match(r'^https?://', command_or_string):
+                raise ValueError("max_response_bytes is only supported for Streamable HTTP MCP servers")
+
         # Set up logging - default to WARNING level to hide INFO messages
         if debug:
             get_logger("mcp-wrapper").setLevel(logging.DEBUG)
@@ -510,6 +537,8 @@ class MCP:
             # single source of truth (see mcp_transport.get_transport_type).
             from .mcp_transport import get_transport_type
             if get_transport_type(command_or_string) == "sse" and 'transport_type' not in kwargs:
+                if max_response_bytes is not None:
+                    raise ValueError("max_response_bytes is only supported for Streamable HTTP MCP servers")
                 # Legacy SSE URL - use SSE transport for backward compatibility
                 from .mcp_sse import SSEMCPClient
                 self.sse_client = SSEMCPClient(command_or_string, debug=debug, timeout=timeout)
@@ -533,6 +562,8 @@ class MCP:
                     transport_options['session'] = kwargs.pop('session')
                 if 'resumability' in kwargs:
                     transport_options['resumability'] = kwargs.pop('resumability')
+                if max_response_bytes is not None:
+                    transport_options['max_response_bytes'] = max_response_bytes
                 
                 self.http_stream_client = HTTPStreamMCPClient(
                     command_or_string, 

src/praisonai-agents/praisonaiagents/mcp/mcp_auth_storage.py

@@ -17,6 +17,11 @@ import time
 from pathlib import Path
 from typing import Any, Dict, Optional
 
+from .._logging import get_logger
+from ..utils.atomic_io import update_json
+
+logger = get_logger(__name__)
+
 
 def get_default_auth_filepath() -> str:
     """
@@ -74,22 +79,56 @@ class MCPAuthStorage:
         parent.mkdir(parents=True, exist_ok=True)
     
     def _read(self) -> Dict[str, Any]:
-        """Read all entries from storage file."""
+        """Read all entries from storage file.
+
+        A corrupt file is moved aside to a unique ``.corrupt`` sidecar and
+        logged rather than silently treated as "no credentials", so a torn
+        write can never make every server's tokens vanish without a trace.
+
+        The quarantine happens *under the same FileLock the writers use*, and
+        the file is re-read inside the lock first: if a concurrent writer
+        repaired the store between the first decode error and acquiring the
+        lock, we return the repaired data instead of quarantining valid
+        credentials.
+        """
         try:
-            with open(self.filepath, 'r') as f:
+            with open(self.filepath, 'r', encoding='utf-8') as f:
                 return json.load(f)
-        except (FileNotFoundError, json.JSONDecodeError):
+        except FileNotFoundError:
+            return {}
+        except (json.JSONDecodeError, ValueError):
+            pass
+
+        # Corrupt on the optimistic (lock-free) read. Take the writer lock and
+        # re-read before quarantining so we never move a file another process
+        # just repaired.
+        from ..session.store import FileLock
+        from ..utils.atomic_io import quarantine_corrupt_file
+
+        try:
+            with FileLock(self.filepath, timeout=10):
+                try:
+                    with open(self.filepath, 'r', encoding='utf-8') as f:
+                        return json.load(f)
+                except FileNotFoundError:
+                    return {}
+                except (json.JSONDecodeError, ValueError):
+                    try:
+                        dest = quarantine_corrupt_file(self.filepath)
+                        logger.error(
+                            "MCP auth store was corrupt; moved aside to %s",
+                            dest,
+                        )
+                    except OSError:
+                        logger.error(
+                            "MCP auth store at %s is corrupt and could not be "
+                            "quarantined", self.filepath,
+                        )
+                    return {}
+        except (OSError, IOError):
+            # Could not acquire the lock; fall back to the previous behaviour
+            # of reporting empty rather than blocking a read path.
             return {}
-    
-    def _write(self, data: Dict[str, Any]) -> None:
-        """Write all entries to storage file with secure permissions."""
-        # Write to file
-        with open(self.filepath, 'w') as f:
-            json.dump(data, f, indent=2)
-        
-        # Set file permissions to 0600 (owner read/write only)
-        if os.name != 'nt':  # Unix-like systems
-            os.chmod(self.filepath, 0o600)
     
     def get(self, mcp_name: str) -> Optional[Dict[str, Any]]:
         """
@@ -143,20 +182,34 @@ class MCPAuthStorage:
     def _set(self, mcp_name: str, entry: Dict[str, Any], server_url: Optional[str] = None) -> None:
         """
         Set an auth entry for an MCP server.
-        
+
+        Uses a locked atomic read-modify-write so concurrent OAuth flows (across
+        threads or processes) no longer lose each other's entries, and a crash
+        mid-write leaves the previous file intact. The temp file created by
+        ``atomic_write_json`` is mode 0600, so secrets are never world-readable,
+        even for the moment between create and chmod.
+
         Args:
             mcp_name: Name of the MCP server
             entry: Auth entry dict
             server_url: Optional server URL to track
         """
-        data = self._read()
-        
-        # Update server_url if provided
         if server_url:
             entry["server_url"] = server_url
-        
-        data[mcp_name] = entry
-        self._write(data)
+
+        def mutate(data: Dict[str, Any]) -> None:
+            data[mcp_name] = entry
+
+        update_json(self.filepath, mutate, indent=2)
+
+        # Tighten to 0600 after the atomic swap. mkstemp already created the temp
+        # file as 0600 so there is no world-readable window; this keeps the mode
+        # correct across platforms and pre-existing files.
+        if os.name != 'nt':
+            try:
+                os.chmod(self.filepath, 0o600)
+            except OSError:
+                pass
     
     def set_tokens(
         self,
@@ -201,10 +254,15 @@ class MCPAuthStorage:
         Args:
             mcp_name: Name of the MCP server
         """
-        data = self._read()
-        if mcp_name in data:
-            del data[mcp_name]
-            self._write(data)
+        def mutate(data: Dict[str, Any]) -> None:
+            data.pop(mcp_name, None)
+
+        update_json(self.filepath, mutate, indent=2)
+        if os.name != 'nt':
+            try:
+                os.chmod(self.filepath, 0o600)
+            except OSError:
+                pass
     
     def set_code_verifier(self, mcp_name: str, code_verifier: str) -> None:
         """

src/praisonai-agents/praisonaiagents/mcp/mcp_http_stream.py

@@ -71,6 +71,61 @@ def _register_cleanup():
         atexit.register(_cleanup_all_clients)
         _cleanup_registered = True
 
+def _bounded_httpx_client_factory(max_response_bytes):
+    import httpx
+
+    class BoundedStream(httpx.AsyncByteStream):
+        def __init__(self, stream):
+            self.stream = stream
+
+        async def __aiter__(self):
+            total_bytes = 0
+            async for chunk in self.stream:
+                total_bytes += len(chunk)
+                if total_bytes > max_response_bytes:
+                    raise ValueError(f"MCP response exceeded maximum of {max_response_bytes} bytes")
+                yield chunk
+
+        async def aclose(self):
+            await self.stream.aclose()
+
+    class BoundedClient(httpx.AsyncClient):
+        async def send(self, request, *, stream=False, **kwargs):
+            response = await super().send(request, stream=True, **kwargs)
+            encoding = response.headers.get("content-encoding", "identity").lower()
+            if encoding not in ("", "identity"):
+                await response.aclose()
+                raise ValueError("Compressed MCP responses are not supported with a response byte limit")
+            content_length = response.headers.get("content-length")
+            try:
+                content_length = int(content_length) if content_length is not None else None
+            except ValueError:
+                content_length = None
+            if content_length is not None and content_length > max_response_bytes:
+                await response.aclose()
+                raise ValueError(f"MCP response exceeded maximum of {max_response_bytes} bytes")
+            response.stream = BoundedStream(response.stream)
+            if not stream:
+                try:
+                    await response.aread()
+                except BaseException:
+                    await response.aclose()
+                    raise
+                await response.aclose()
+            return response
+
+    def create_client(headers=None, timeout=None, auth=None):
+        request_headers = dict(headers or {})
+        request_headers["Accept-Encoding"] = "identity"
+        return BoundedClient(
+            headers=request_headers,
+            timeout=timeout,
+            auth=auth,
+            trust_env=True,
+        )
+
+    return create_client
+
 class HTTPStreamMCPTool:
     """A wrapper for an MCP tool that can be used with praisonaiagents."""
     
@@ -137,6 +192,8 @@ class HTTPStreamMCPTool:
         logger.debug(f"Async calling tool {self.name} with args: {kwargs}")
         try:
             result = await self.session.call_tool(self.name, kwargs)
+            if getattr(result, 'isError', False):
+                raise RuntimeError(f"MCP tool {self.name} failed: {result.content}")
             
             # Extract text from result
             if hasattr(result, 'content') and result.content:
@@ -490,6 +547,13 @@ class HTTPStreamMCPClient:
         self.debug = debug
         self.timeout = timeout
         self.options = options or {}
+        self.max_response_bytes = self.options.get('max_response_bytes')
+        if self.max_response_bytes is not None and (
+            isinstance(self.max_response_bytes, bool)
+            or not isinstance(self.max_response_bytes, int)
+            or self.max_response_bytes <= 0
+        ):
+            raise ValueError("max_response_bytes must be a positive integer")
         self.session = None
         self.tools = []
         self.transport = None
@@ -510,17 +574,16 @@ class HTTPStreamMCPClient:
         
     def _initialize(self):
         """Initialize the connection and tools."""
-        # Use the global event loop
         loop = get_event_loop()
-        
-        # Start a background thread to run the event loop
-        def run_event_loop():
-            asyncio.set_event_loop(loop)
-            loop.run_forever()
-        
-        self.loop_thread = threading.Thread(target=run_event_loop, daemon=True)
-        self.loop_thread.start()
-        
+        self.loop_thread = None
+        if not loop.is_running():
+            def run_event_loop():
+                asyncio.set_event_loop(loop)
+                loop.run_forever()
+
+            self.loop_thread = threading.Thread(target=run_event_loop, daemon=True)
+            self.loop_thread.start()
+
         # Run the initialization in the event loop
         future = asyncio.run_coroutine_threadsafe(self._async_initialize(), loop)
         self.tools = future.result(timeout=self.timeout)
@@ -542,11 +605,16 @@ class HTTPStreamMCPClient:
 
         headers = self.options.get('headers') or None
 
-        self._streams_context = streamablehttp_client(
-            url=self.base_url,
-            headers=headers,
-            timeout=self.timeout,
-        )
+        client_options = {
+            "url": self.base_url,
+            "headers": headers,
+            "timeout": self.timeout,
+        }
+        if self.max_response_bytes is not None:
+            client_options["httpx_client_factory"] = _bounded_httpx_client_factory(
+                self.max_response_bytes
+            )
+        self._streams_context = streamablehttp_client(**client_options)
 
         # If any step of the handshake fails after a context has been entered,
         # unwind the already-entered contexts so we don't leak the httpx/anyio
@@ -689,4 +757,4 @@ class HTTPStreamMCPClient:
                 self._force_cleanup()
         except Exception:
             # Never raise exceptions in __del__
-            pass
+            pass

src/praisonai-agents/praisonaiagents/memory/adapters/in_memory_adapter.py

@@ -6,6 +6,7 @@ No dependencies required - stores data as plain Python lists.
 """
 
 import threading
+from copy import deepcopy
 from typing import Any, Dict, List, Optional
 
 
@@ -15,6 +16,8 @@ class InMemoryAdapter:
 
     Stores short-term and long-term memories as plain Python lists.
     Data is not persisted across process restarts.
+    Metadata must support deepcopy; writes and reads own independent copies so
+    caller edits cannot change stored records outside the adapter's lock.
 
     Usage:
         adapter = InMemoryAdapter()
@@ -48,7 +51,7 @@ class InMemoryAdapter:
                 "id": str(self._next_id),
                 "text": text,
                 "type": "short",
-                "metadata": metadata,
+                "metadata": deepcopy(metadata),
             }
             self._next_id += 1
             self._data.append(entry)
@@ -64,7 +67,7 @@ class InMemoryAdapter:
                 for e in self._data
                 if e["type"] == "short" and query.lower() in e["text"].lower()
             ]
-            return results[:limit]
+            return deepcopy(results[:limit])
 
     def store_long_term(
         self, text: str, metadata: Optional[Dict[str, Any]] = None, **kwargs
@@ -74,7 +77,7 @@ class InMemoryAdapter:
                 "id": str(self._next_id),
                 "text": text,
                 "type": "long",
-                "metadata": metadata,
+                "metadata": deepcopy(metadata),
             }
             self._next_id += 1
             self._data.append(entry)
@@ -90,7 +93,7 @@ class InMemoryAdapter:
                 for e in self._data
                 if e["type"] == "long" and query.lower() in e["text"].lower()
             ]
-            return results[:limit]
+            return deepcopy(results[:limit])
 
     def delete_memory(self, memory_id: str, tier: Optional[str] = None, **kwargs) -> bool:
         """Delete a memory by ID. Returns True if an entry was removed.
@@ -123,5 +126,8 @@ class InMemoryAdapter:
 
     def get_all_memories(self, **kwargs) -> List[Dict[str, Any]]:
         with self._lock:
-            # Return defensive copy to prevent external mutation of internal state
-            return [dict(entry) for entry in self._data]
+            # Entries are owned on insertion and never mutated afterward.
+            # Keep their membership snapshot under lock, then copy nested data
+            # without blocking stores, deletes, or eviction.
+            snapshot = list(self._data)
+        return deepcopy(snapshot)

src/praisonai-agents/praisonaiagents/memory/adapters/sqlite_adapter.py

@@ -19,6 +19,16 @@ from ..protocols import MemoryProtocol
 
 logger = get_logger(__name__)
 
+
+def _metadata_user_id(raw):
+    """Match the existing Python JSON reader, including non-finite values."""
+    try:
+        metadata = json.loads(raw) if raw else {}
+        return str(metadata["user_id"]) if isinstance(metadata, dict) and metadata.get("user_id") is not None else None
+    except (TypeError, ValueError):
+        return None
+
+
 class SqliteMemoryAdapter:
     """
     SQLite-based memory adapter implementing MemoryProtocol.
@@ -73,12 +83,13 @@ class SqliteMemoryAdapter:
 
     def _get_stm_conn(self):
         """Get thread-local short-term memory connection."""
-        if not hasattr(self._local, 'stm_conn'):
+        if getattr(self._local, 'stm_conn', None) is None:
             self._local.stm_conn = sqlite3.connect(
                 self.short_db,
                 check_same_thread=False,
                 timeout=10.0
             )
+            self._local.stm_conn.create_function("memory_user_id", 1, _metadata_user_id, deterministic=True)
             self._local.stm_conn.execute("""
                 CREATE TABLE IF NOT EXISTS short_term_memory (
                     id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -97,12 +108,13 @@ class SqliteMemoryAdapter:
     
     def _get_ltm_conn(self):
         """Get thread-local long-term memory connection."""
-        if not hasattr(self._local, 'ltm_conn'):
+        if getattr(self._local, 'ltm_conn', None) is None:
             self._local.ltm_conn = sqlite3.connect(
                 self.long_db,
                 check_same_thread=False,
                 timeout=10.0
             )
+            self._local.ltm_conn.create_function("memory_user_id", 1, _metadata_user_id, deterministic=True)
             self._local.ltm_conn.execute("""
                 CREATE TABLE IF NOT EXISTS long_term_memory (
                     id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -143,10 +155,13 @@ class SqliteMemoryAdapter:
     ) -> List[Dict[str, Any]]:
         """Search short-term memory."""
         conn = self._get_stm_conn()
+        user_id = kwargs.get("user_id")
+        user_filter = " AND memory_user_id(metadata) = ?" if user_id is not None else ""
+        params = (f"%{query}%", str(user_id), limit) if user_id is not None else (f"%{query}%", limit)
         cursor = conn.execute(
             "SELECT id, content, metadata, timestamp FROM short_term_memory "
-            "WHERE content LIKE ? ORDER BY timestamp DESC LIMIT ?",
-            (f"%{query}%", limit)
+            "WHERE content LIKE ?" + user_filter + " ORDER BY timestamp DESC LIMIT ?",
+            params
         )
         
         results = []
@@ -184,10 +199,13 @@ class SqliteMemoryAdapter:
     ) -> List[Dict[str, Any]]:
         """Search long-term memory."""
         conn = self._get_ltm_conn()
+        user_id = kwargs.get("user_id")
+        user_filter = " AND memory_user_id(metadata) = ?" if user_id is not None else ""
+        params = (f"%{query}%", str(user_id), limit) if user_id is not None else (f"%{query}%", limit)
         cursor = conn.execute(
             "SELECT id, content, metadata, timestamp FROM long_term_memory "
-            "WHERE content LIKE ? ORDER BY timestamp DESC LIMIT ?",
-            (f"%{query}%", limit)
+            "WHERE content LIKE ?" + user_filter + " ORDER BY timestamp DESC LIMIT ?",
+            params
         )
         
         results = []
@@ -252,8 +270,10 @@ class SqliteMemoryAdapter:
 
     def get_all_memories(self, **kwargs) -> List[Dict[str, Any]]:
         """Get all memories from both short-term and long-term."""
-        short_memories = self.search_short_term("", limit=1000)
-        long_memories = self.search_long_term("", limit=1000)
+        # SQLite's negative LIMIT means no upper bound. Enumeration must return
+        # every row; ordinary searches retain their default/explicit limits.
+        short_memories = self.search_short_term("", limit=-1)
+        long_memories = self.search_long_term("", limit=-1)
         
         # Mark memory types
         for memory in short_memories:
@@ -272,4 +292,30 @@ class SqliteMemoryAdapter:
                     conn.close()
                 except Exception:
                     pass
-            self._all_connections.clear()
+            self._all_connections.clear()
+
+    def close_thread_connections(self):
+        """Close only the calling thread's connections.
+
+        Leaves the shared adapter and other threads' connections intact so the
+        adapter can be reused; the calling thread lazily reopens its own
+        connections on next access.
+
+        A connection is only removed from the registry and cleared from
+        thread-local storage after it closes successfully. If ``close()`` raises,
+        the connection stays tracked so a later ``close_connections()`` /
+        ``close_thread_connections()`` can retry it, and the failure is logged
+        rather than silently swallowed.
+        """
+        for attr in ("stm_conn", "ltm_conn"):
+            conn = getattr(self._local, attr, None)
+            if conn is None:
+                continue
+            try:
+                conn.close()
+            except Exception as e:
+                logger.warning(f"Failed to close {attr}; keeping it tracked for retry: {e}")
+                continue
+            with self._connection_lock:
+                self._all_connections.discard(conn)
+            setattr(self._local, attr, None)

src/praisonai-agents/praisonaiagents/memory/file_memory.py

@@ -266,11 +266,12 @@ class FileMemory:
             return True
         except (IOError, OSError) as e:
             self._log(f"Error writing {filepath}: {e}", logging.ERROR)
+            return False
+        finally:
             try:
                 tmp_path.unlink()
             except OSError:
                 pass
-            return False
 
     def commit_memory_batch(
         self,

src/praisonai-agents/praisonaiagents/memory/learn/stores.py

@@ -237,8 +237,8 @@ class BaseStore(ABC):
     def clear(self) -> int:
         """Clear all entries."""
         count = len(self._entries)
+        self._store._reset({})
         self._entries = {}
-        self._save()
         self._was_updated = True
         return count
     

src/praisonai-agents/praisonaiagents/memory/memory.py

@@ -500,18 +500,16 @@ class Memory(SearchMixin, MemoryCoreMixin):
     def _get_stm_conn(self):
         """Get thread-local short-term memory SQLite connection."""
         if not hasattr(self._local, 'stm_conn') or self._local.stm_conn is None:
-            self._local.stm_conn = sqlite3.connect(
+            # Hardened factory: WAL where the filesystem supports it, safe
+            # DELETE fallback on NFS/SMB/FUSE/virtiofs (Issue #5264).
+            from ..storage.sqlite import connect as _sqlite_connect
+
+            self._local.stm_conn = _sqlite_connect(
                 self.short_db,
                 check_same_thread=False,  # Allow cross-thread cleanup
-                timeout=30.0  # 30 second timeout for lock contention
+                timeout=30.0,  # 30 second timeout for lock contention
+                busy_timeout_ms=30000,  # 30 seconds
             )
-            # Configure busy timeout for better contention handling
-            self._local.stm_conn.execute("PRAGMA busy_timeout=30000")  # 30 seconds
-            
-            # Enable WAL mode for concurrent read/write without blocking
-            result = self._local.stm_conn.execute("PRAGMA journal_mode=WAL").fetchone()
-            if result and result[0].upper() != 'WAL':
-                logger.warning(f"WAL mode not enabled for STM, got: {result[0]}")
             self._local.stm_conn.commit()
             
             # Register connection for cleanup
@@ -522,18 +520,16 @@ class Memory(SearchMixin, MemoryCoreMixin):
     def _get_ltm_conn(self):
         """Get thread-local long-term memory SQLite connection."""
         if not hasattr(self._local, 'ltm_conn') or self._local.ltm_conn is None:
-            self._local.ltm_conn = sqlite3.connect(
+            # Hardened factory: WAL where the filesystem supports it, safe
+            # DELETE fallback on NFS/SMB/FUSE/virtiofs (Issue #5264).
+            from ..storage.sqlite import connect as _sqlite_connect
+
+            self._local.ltm_conn = _sqlite_connect(
                 self.long_db,
                 check_same_thread=False,  # Allow cross-thread cleanup
-                timeout=30.0  # 30 second timeout for lock contention
+                timeout=30.0,  # 30 second timeout for lock contention
+                busy_timeout_ms=30000,  # 30 seconds
             )
-            # Configure busy timeout for better contention handling
-            self._local.ltm_conn.execute("PRAGMA busy_timeout=30000")  # 30 seconds
-            
-            # Enable WAL mode for concurrent read/write without blocking
-            result = self._local.ltm_conn.execute("PRAGMA journal_mode=WAL").fetchone()
-            if result and result[0].upper() != 'WAL':
-                logger.warning(f"WAL mode not enabled for LTM, got: {result[0]}")
             self._local.ltm_conn.commit()
             
             # Register connection for cleanup
@@ -2536,16 +2532,31 @@ class Memory(SearchMixin, MemoryCoreMixin):
         # Other active threads should manage their own connections
         # Only clear full registry in __del__ or explicit shutdown
         
-        # Close memory adapter if it exists (protocol-driven path)
+        # Close memory adapter if it exists (protocol-driven path).
+        #
+        # SQLite is thread-scoped by contract (#1562/#1563): close only the
+        # calling thread's adapter connections and KEEP the shared adapter so
+        # other threads and subsequent same-thread searches keep using the
+        # adapter's tables. Tearing it down here would make searches fall back
+        # to legacy SQL against short_mem/long_mem (#5486).
         if hasattr(self, 'memory_adapter') and self.memory_adapter:
-            try:
-                if hasattr(self.memory_adapter, 'close') and callable(self.memory_adapter.close):
-                    self.memory_adapter.close()
-                    logger.debug("Memory adapter closed successfully")
-            except Exception as e:
-                logger.warning(f"Error closing memory adapter: {e}")
-            finally:
-                self.memory_adapter = None
+            if isinstance(self.memory_adapter, SqliteMemoryAdapter):
+                try:
+                    if hasattr(self.memory_adapter, 'close_thread_connections'):
+                        self.memory_adapter.close_thread_connections()
+                        logger.debug("SQLite adapter thread connections closed")
+                except Exception as e:
+                    logger.warning(f"Error closing SQLite adapter thread connections: {e}")
+                # Intentionally preserve self.memory_adapter for reuse.
+            else:
+                try:
+                    if hasattr(self.memory_adapter, 'close') and callable(self.memory_adapter.close):
+                        self.memory_adapter.close()
+                        logger.debug("Memory adapter closed successfully")
+                except Exception as e:
+                    logger.warning(f"Error closing memory adapter: {e}")
+                finally:
+                    self.memory_adapter = None
         
         # Close MongoDB client if it exists (legacy path)
         if hasattr(self, 'mongo_client') and self.mongo_client:

src/praisonai-agents/praisonaiagents/permissions/manager.py

@@ -113,12 +113,19 @@ class PermissionManager:
                 logger.warning(f"Failed to load rules: {e}")
     
     def _save_rules(self):
-        """Save rules to disk."""
+        """Save rules to disk atomically.
+
+        Serialises to a temp file and ``os.replace``\\ s it into place so a
+        non-serialisable value (``TypeError``), an OOM kill, or a crash
+        mid-write can never truncate the previous good rules file. A downgraded
+        ``deny`` rule after restart is a security regression, so the old bytes
+        must survive any failed save.
+        """
+        from ..utils.atomic_io import atomic_write_json
         path = self._get_rules_path()
         try:
-            with open(path, "w") as f:
-                json.dump([r.to_dict() for r in self._rules], f, indent=2)
-        except IOError as e:
+            atomic_write_json(path, [r.to_dict() for r in self._rules], indent=2)
+        except (OSError, TypeError, ValueError) as e:
             logger.error(f"Failed to save rules: {e}")
     
     def save_rules(self):
@@ -140,7 +147,8 @@ class PermissionManager:
                 logger.warning(f"Failed to load approvals: {e}")
     
     def _save_approvals(self):
-        """Save approvals to disk."""
+        """Save approvals to disk atomically (see :meth:`_save_rules`)."""
+        from ..utils.atomic_io import atomic_write_json
         path = self._get_approvals_path()
         try:
             # Filter out expired and one-time approvals
@@ -148,9 +156,8 @@ class PermissionManager:
                 a for a in self._approvals
                 if a.is_valid() and a.scope in ("session", "always")
             ]
-            with open(path, "w") as f:
-                json.dump([a.to_dict() for a in persistent], f, indent=2)
-        except IOError as e:
+            atomic_write_json(path, [a.to_dict() for a in persistent], indent=2)
+        except (OSError, TypeError, ValueError) as e:
             logger.error(f"Failed to save approvals: {e}")
     
     def add_rule(self, rule: PermissionRule) -> str:

src/praisonai-agents/praisonaiagents/planning/storage.py

@@ -108,10 +108,11 @@ class PlanStorage:
         
         # Convert plan to markdown
         markdown = plan.to_markdown()
-        
-        with open(path, "w", encoding="utf-8") as f:
-            f.write(markdown)
-            
+
+        # Atomic write so a crash mid-write never truncates the previous plan.
+        from ..utils.atomic_io import atomic_write_text
+        atomic_write_text(path, markdown)
+
         plan.file_path = path
         logger.debug(f"Saved plan to {path}")
         
@@ -227,10 +228,11 @@ class PlanStorage:
         """
         filename = f"{name}.json"
         path = os.path.join(self.todos_dir, filename)
-        
-        with open(path, "w", encoding="utf-8") as f:
-            f.write(todo.to_json())
-            
+
+        # Atomic write so a crash mid-write never truncates the previous todos.
+        from ..utils.atomic_io import atomic_write_text
+        atomic_write_text(path, todo.to_json())
+
         logger.debug(f"Saved todo list to {path}")
         return path
     

src/praisonai-agents/praisonaiagents/plugins/manager.py

@@ -617,6 +617,74 @@ class PluginManager:
     def get_all_tools(self) -> List[Any]:
         """Get all tools from all enabled plugins."""
         return [tool for _plugin_name, tool in self.get_all_tools_with_sources()]
+
+    def _enabled_of_type(self, plugin_type: "PluginType") -> List[Tuple[str, Plugin]]:
+        """Snapshot enabled ``(name, plugin)`` pairs of a given declared type.
+
+        Dispatch key for the typed subsystems: a plugin only contributes a
+        guardrail / skill / policy when it *declares* the matching
+        ``PluginType`` in its ``PluginInfo``. Snapshotting under the lock keeps
+        this safe against concurrent register/unregister in multi-agent runs.
+        """
+        from .plugin import PluginType
+
+        with self._lock:
+            result: List[Tuple[str, Plugin]] = []
+            for name, plugin in self._plugins.items():
+                if not self._enabled.get(name, False):
+                    continue
+                try:
+                    declared = getattr(plugin.info, "plugin_type", PluginType.HOOK)
+                except Exception:
+                    declared = PluginType.HOOK
+                if declared == plugin_type:
+                    result.append((name, plugin))
+            return result
+
+    def get_all_guardrails(self) -> List[Any]:
+        """Guardrail objects contributed by enabled ``GUARDRAIL`` plugins.
+
+        Calls ``as_guardrail()`` on each enabled plugin declared
+        ``PluginType.GUARDRAIL`` and returns the non-``None`` guardrail objects
+        the Agent folds into its ``GuardrailChain`` (so they see
+        ``validate_tool_call`` / ``validate_tool_result``, not just final text).
+        """
+        from .plugin import PluginType
+
+        guardrails: List[Any] = []
+        for name, plugin in self._enabled_of_type(PluginType.GUARDRAIL):
+            try:
+                g = plugin.as_guardrail()
+            except Exception as e:
+                logger.error(f"Error getting guardrail from plugin {name}: {e}")
+                continue
+            if g is not None:
+                guardrails.append(g)
+        return guardrails
+
+    def get_all_skills(self) -> List[Any]:
+        """Skill sources/dirs contributed by enabled ``SKILL`` plugins."""
+        from .plugin import PluginType
+
+        skills: List[Any] = []
+        for name, plugin in self._enabled_of_type(PluginType.SKILL):
+            try:
+                skills.extend(plugin.get_skills() or [])
+            except Exception as e:
+                logger.error(f"Error getting skills from plugin {name}: {e}")
+        return skills
+
+    def get_all_policies(self) -> List[Any]:
+        """Policy rules contributed by enabled ``POLICY`` plugins."""
+        from .plugin import PluginType
+
+        policies: List[Any] = []
+        for name, plugin in self._enabled_of_type(PluginType.POLICY):
+            try:
+                policies.extend(plugin.get_policies() or [])
+            except Exception as e:
+                logger.error(f"Error getting policies from plugin {name}: {e}")
+        return policies
     
     def shutdown(self):
         """Shutdown all plugins."""
@@ -1238,6 +1306,51 @@ def _adapt_plugin_hooks(plugin: Plugin) -> Iterator[Tuple["HookEvent", Callable]
             return HookResult.allow()
         yield HookEvent.CLI_BACKEND_EXECUTE, cli_backend_execute_hook
 
+    # Generic, declaration-driven tail. The explicit branches above give
+    # rewrite/deny semantics to the conversation/tool/channel events that need
+    # them. Everything else a plugin declares in ``PluginInfo.hooks`` — the
+    # gateway/schedule/kanban/compaction/subagent/model-fallback lifecycle — is
+    # observe-only, so a single loop bridges it by method name (``event.value``)
+    # without a hand-written branch per event. This keeps the coverage matrix
+    # from drifting: any future emitted event becomes first-class the moment a
+    # plugin declares it and defines the matching method.
+    yielded = {
+        HookEvent.BEFORE_AGENT, HookEvent.AFTER_AGENT,
+        HookEvent.BEFORE_LLM, HookEvent.AFTER_LLM,
+        HookEvent.BEFORE_TOOL, HookEvent.AFTER_TOOL,
+        HookEvent.BEFORE_TOOL_DEFINITIONS,
+        HookEvent.MESSAGE_RECEIVED, HookEvent.MESSAGE_SENDING,
+        HookEvent.MESSAGE_SENT, HookEvent.MESSAGE_UNDELIVERED,
+        HookEvent.ON_PERMISSION_ASK, HookEvent.ON_CONFIG, HookEvent.ON_AUTH,
+        HookEvent.SESSION_START, HookEvent.SESSION_END, HookEvent.ON_ERROR,
+        HookEvent.CLI_BACKEND_EXECUTE,
+    }
+
+    def _make_observe_hook(method):
+        def observe_hook(data, _m=method):
+            payload = data.to_dict() if hasattr(data, "to_dict") else {}
+            _m(payload)
+            return HookResult.allow()
+        return observe_hook
+
+    declared = getattr(plugin.info, "hooks", None) or []
+    seen: set = set()
+    for event in declared:
+        if not isinstance(event, HookEvent) or event in yielded or event in seen:
+            continue
+        seen.add(event)
+        method = getattr(plugin, event.value, None)
+        if not callable(method):
+            continue
+        parent = getattr(base, event.value, None)
+        own = getattr(type(plugin), event.value, None)
+        # A base no-op that the plugin did not override yields nothing (avoids
+        # registering dead hooks). Methods with no base counterpart (e.g.
+        # ``kanban_task_created``) are treated as genuine overrides.
+        if parent is not None and own is parent:
+            continue
+        yield event, _make_observe_hook(method)
+
 
 # Global plugin manager instance
 _default_manager: Optional[PluginManager] = None
@@ -1248,3 +1361,15 @@ def get_plugin_manager() -> PluginManager:
     if _default_manager is None:
         _default_manager = PluginManager()
     return _default_manager
+
+
+def has_registered_plugins() -> bool:
+    """Return whether the global manager has any plugins registered.
+
+    Cheap, allocation-free fast path for the hot Agent-init merge calls: when
+    the singleton has not been created or holds no plugins (the common case),
+    callers can skip constructing/scanning the manager entirely. Does NOT
+    create the singleton as a side effect.
+    """
+    manager = _default_manager
+    return manager is not None and bool(manager._plugins)

src/praisonai-agents/praisonaiagents/plugins/plugin.py

@@ -93,6 +93,7 @@ class PluginInfo:
     author: str = ""
     hooks: List[PluginHook] = field(default_factory=list)
     dependencies: List[str] = field(default_factory=list)
+    plugin_type: PluginType = PluginType.HOOK
     
     def to_dict(self) -> Dict[str, Any]:
         return {
@@ -102,6 +103,7 @@ class PluginInfo:
             "author": self.author,
             "hooks": [h.value for h in self.hooks],
             "dependencies": self.dependencies,
+            "plugin_type": self.plugin_type.value,
         }
 
 class Plugin(ABC):
@@ -246,6 +248,42 @@ class Plugin(ABC):
     def cli_backend_execute(self, context: Dict[str, Any]) -> None:
         """Called after a CLI backend delegates a turn. Observe-only."""
         pass
+
+    def gateway_start(self, context: Dict[str, Any]) -> None:
+        """Called when the gateway/BotOS starts. Observe-only (platforms, bot_count)."""
+        pass
+
+    def gateway_stop(self, context: Dict[str, Any]) -> None:
+        """Called when the gateway/BotOS stops. Observe-only (platforms, reason)."""
+        pass
+
+    def schedule_add(self, context: Dict[str, Any]) -> None:
+        """Called when a scheduled job is persisted. Observe-only."""
+        pass
+
+    def schedule_remove(self, context: Dict[str, Any]) -> None:
+        """Called when a scheduled job is deleted. Observe-only."""
+        pass
+
+    def schedule_trigger(self, context: Dict[str, Any]) -> None:
+        """Called when a scheduled job fires. Observe-only."""
+        pass
+
+    def before_compaction(self, context: Dict[str, Any]) -> None:
+        """Called before conversation memory is compacted. Observe-only."""
+        pass
+
+    def after_compaction(self, context: Dict[str, Any]) -> None:
+        """Called after conversation memory is compacted. Observe-only."""
+        pass
+
+    def subagent_stop(self, context: Dict[str, Any]) -> None:
+        """Called when a spawned subagent reaches a terminal state. Observe-only."""
+        pass
+
+    def model_fallback(self, context: Dict[str, Any]) -> None:
+        """Called when the runtime falls back to a secondary model. Observe-only."""
+        pass
     
     def on_permission_ask(self, target: str, reason: str) -> Optional[bool]:
         """Called when permission is requested. Return True/False to auto-approve/deny."""
@@ -269,6 +307,40 @@ class Plugin(ABC):
         """
         return []
 
+    def as_guardrail(self) -> Optional[Any]:
+        """Return a ``GuardrailProtocol`` object for a ``GUARDRAIL`` plugin.
+
+        Overriding this lets a plugin declared ``PluginType.GUARDRAIL``
+        participate in the Agent's ``GuardrailChain`` — including
+        ``validate_tool_call`` / ``validate_tool_result`` — instead of only
+        rewriting the model's final text through ``after_llm``. Return an
+        object exposing any of ``validate_input`` / ``validate_output`` /
+        ``validate_tool_call`` / ``validate_tool_result``, or ``None`` (default)
+        to opt out.
+        """
+        return None
+
+    def get_skills(self) -> List[Any]:
+        """Return skill directory paths for a ``SKILL`` plugin.
+
+        Mirrors :meth:`get_tools`. Entries are filesystem path **strings** to
+        skill directories (each containing a ``SKILL.md``); they are handed to
+        the Agent's ``SkillManager`` so an installed plugin contributes real,
+        discoverable skills. Non-string entries are ignored (the manager loads
+        by path). Empty by default (opt-in).
+        """
+        return []
+
+    def get_policies(self) -> List[Any]:
+        """Return ``Policy`` rules for a ``POLICY`` plugin.
+
+        Entries are ``praisonaiagents.policy.Policy`` objects added to the
+        Agent's ``PolicyEngine`` so a policy plugin enforces execution rules
+        rather than only answering ``on_permission_ask``. Empty by default
+        (opt-in).
+        """
+        return []
+
 class FunctionPlugin(Plugin):
     """
     A simple plugin that wraps functions for specific hooks.

src/praisonai-agents/praisonaiagents/policy/__init__.py

@@ -49,6 +49,7 @@ __all__ = [
     "create_deny_tools_policy",
     "create_allow_tools_policy",
     "create_read_only_policy",
+    "create_network_policy",
 ]
 
 
@@ -64,6 +65,7 @@ _LAZY_IMPORTS = {
     "create_deny_tools_policy": (f"{__name__}.engine", "create_deny_tools_policy"),
     "create_allow_tools_policy": (f"{__name__}.engine", "create_allow_tools_policy"),
     "create_read_only_policy": (f"{__name__}.engine", "create_read_only_policy"),
+    "create_network_policy": (f"{__name__}.engine", "create_network_policy"),
 }
 
 __getattr__ = create_lazy_getattr(_LAZY_IMPORTS, __name__)

src/praisonai-agents/praisonaiagents/policy/engine.py

@@ -13,6 +13,18 @@ from .config import PolicyConfig
 
 logger = get_logger(__name__)
 
+
+def _normalize_host(host: str) -> str:
+    """Normalize a hostname for consistent egress matching.
+
+    DNS hostnames are case-insensitive and a single trailing dot denotes the
+    root; treating ``API.Example.com.`` and ``api.example.com`` as the same
+    host prevents policy bypass via case/trailing-dot variants.
+    """
+    if not isinstance(host, str):
+        return host
+    return host.strip().rstrip(".").lower()
+
 class PolicyEngine:
     """
     Policy Engine for execution control.
@@ -185,6 +197,34 @@ class PolicyEngine:
             {"operation": operation, "file_path": file_path}
         )
     
+    def check_network(
+        self,
+        host: str,
+        context: Optional[Dict[str, Any]] = None
+    ) -> PolicyResult:
+        """
+        Check if an outbound network request to a host is allowed.
+        
+        This is the deterministic egress decision every outbound request
+        (tool HTTP call, MCP request, provider/LLM call) can be routed
+        through. It mirrors ``check_tool``/``check_file`` and evaluates a
+        ``net:<host>`` resource against the configured policies.
+        
+        Args:
+            host: Destination host (e.g., "api.example.com")
+            context: Additional context (e.g., purpose, agent_id)
+            
+        Returns:
+            PolicyResult indicating if the egress is allowed
+        """
+        normalized = _normalize_host(host)
+        # Destination host stays authoritative: a caller-supplied context can
+        # add metadata but must not override the host seen by rule conditions.
+        return self.check(
+            f"net:{normalized}",
+            {**(context or {}), "host": normalized}
+        )
+    
     def clear(self):
         """Remove all policies."""
         self._policies.clear()
@@ -307,6 +347,67 @@ def create_allow_tools_policy(
     
     return Policy(name=name, rules=rules, priority=10)
 
+def create_network_policy(
+    allow_hosts: Optional[List[str]] = None,
+    deny_hosts: Optional[List[str]] = None,
+    name: str = "network_egress"
+) -> Policy:
+    """
+    Create a network-egress policy for outbound host access.
+    
+    Semantics (evaluated by ``PolicyEngine.check_network``):
+    - ``deny_hosts`` wins: a match on the deny-list is always denied.
+    - A non-empty ``allow_hosts`` means default-deny: only hosts matching
+      the allow-list are permitted; everything else is denied.
+    - If ``allow_hosts`` is empty/None, hosts not on the deny-list fall
+      through to the engine's default action (allow unless strict_mode).
+    
+    Host patterns support fnmatch wildcards (e.g. ``*.mycorp.com``).
+    
+    Args:
+        allow_hosts: Hosts permitted egress (non-empty => default-deny)
+        deny_hosts: Hosts always denied (takes precedence over allow)
+        name: Policy name
+        
+    Returns:
+        Policy
+    """
+    # Normalize patterns so they match the normalized host that
+    # ``check_network`` evaluates (case-insensitive, no trailing dot).
+    allow_hosts = [_normalize_host(h) for h in (allow_hosts or [])]
+    deny_hosts = [_normalize_host(h) for h in (deny_hosts or [])]
+    rules: List[PolicyRule] = []
+    
+    # Deny-list wins: highest priority.
+    for pattern in deny_hosts:
+        rules.append(PolicyRule(
+            action=PolicyAction.DENY,
+            resource=f"net:{pattern}",
+            reason=f"Host denied by egress policy: {pattern}",
+            name=f"deny_host_{pattern}",
+            priority=100
+        ))
+    
+    if allow_hosts:
+        # Allow the listed hosts...
+        for pattern in allow_hosts:
+            rules.append(PolicyRule(
+                action=PolicyAction.ALLOW,
+                resource=f"net:{pattern}",
+                name=f"allow_host_{pattern}",
+                priority=10
+            ))
+        # ...and default-deny everything else.
+        rules.append(PolicyRule(
+            action=PolicyAction.DENY,
+            resource="net:*",
+            reason="Host not in egress allow-list",
+            name="deny_host_default",
+            priority=0
+        ))
+    
+    return Policy(name=name, rules=rules, priority=100)
+
 def create_read_only_policy(name: str = "read_only") -> Policy:
     """
     Create a read-only policy that denies write operations.

src/praisonai-agents/praisonaiagents/process/process.py

@@ -844,6 +844,13 @@ Subtask: {st.name}
                                 break
                 else:
                     logging.debug(f"No subtasks created yet for {current_task.name}")
+                    # Mirror the sync workflow(): default a mid-workflow loop
+                    # task (reached via next_tasks/condition, not the start task)
+                    # with no explicit input_file to the tasks.csv convention so
+                    # its subtasks are still expanded instead of being silently
+                    # skipped (issue #5319).
+                    if current_task is not start_task and not current_task.input_file:
+                        current_task.input_file = "tasks.csv"
                     # Create subtasks if needed
                     if current_task.input_file:
                         self._create_loop_subtasks(current_task)

src/praisonai-agents/praisonaiagents/push/__init__.py

@@ -7,6 +7,14 @@ re-exports them lazily so user code can keep importing from
 import importlib
 
 from .models import ChannelMessage
+from .policy import (
+    DEFAULT_PREFERENCE,
+    DetailLevel,
+    NotificationCategory,
+    NotificationDecision,
+    NotificationPreference,
+    evaluate,
+)
 from .protocols import PushTransportProtocol
 
 _lazy_cache: dict = {}
@@ -33,6 +41,12 @@ def __getattr__(name: str):
 __all__ = [
     "ChannelMessage",
     "PushTransportProtocol",
+    "NotificationCategory",
+    "DetailLevel",
+    "NotificationPreference",
+    "NotificationDecision",
+    "evaluate",
+    "DEFAULT_PREFERENCE",
     "PushClient",
     "WebSocketTransport",
     "PollingTransport",

src/praisonai-agents/praisonaiagents/push/policy.py

@@ -0,0 +1,212 @@
+"""Notification policy for gateway push — pure, dependency-free core.
+
+The push stack elsewhere is transport + delivery-guarantee only: it can get
+bytes to a device reliably, but it has no notion of *whether* or *what* to
+notify. This module supplies that missing policy layer as pure protocol code so
+every notifier (the gateway's delivery manager, scheduler completions,
+background-job alerts) can make the same decision identically:
+
+    decision = evaluate(category, prefs, now=..., is_urgent=..., content=...)
+    if decision.send:
+        deliver(recipient, decision.redacted_content)
+
+Three concerns compose into one :class:`NotificationDecision`:
+
+* **Categories** — a recipient opts in per event class (``approval_requested``,
+  ``agent_finished`` …); an un-opted category is suppressed.
+* **Quiet hours** — a per-recipient local window suppresses non-urgent
+  categories; urgent categories may override (``urgent_overrides_quiet``).
+* **Detail level** — ``private`` / ``identified`` / ``detailed`` redacts how
+  much message content leaves the gateway in a device notification.
+
+The heavy enforcement + durable preference store + CLI/YAML surface live in the
+``praisonai``/``praisonai-bot`` wrapper; this stays pure so it carries no heavy
+imports and is trivially testable.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime, time
+from enum import Enum
+from typing import Optional, Set
+
+__all__ = [
+    "NotificationCategory",
+    "DetailLevel",
+    "NotificationPreference",
+    "NotificationDecision",
+    "evaluate",
+    "DEFAULT_PREFERENCE",
+]
+
+
+class NotificationCategory(str, Enum):
+    """Event classes a recipient can subscribe to independently."""
+
+    APPROVAL_REQUESTED = "approval_requested"
+    AGENT_FINISHED = "agent_finished"
+    AGENT_QUESTION = "agent_question"
+    HUMAN_MENTIONED = "human_mentioned"
+    SCHEDULED_TASK_FAILED = "scheduled_task_failed"
+    BACKGROUND_TASK_FAILED = "background_task_failed"
+
+
+class DetailLevel(str, Enum):
+    """How much content leaves the gateway in a notification.
+
+    * ``PRIVATE`` — a bare "you have activity"; no agent/session, no content.
+    * ``IDENTIFIED`` — which agent/session, but no message content.
+    * ``DETAILED`` — content included verbatim.
+    """
+
+    PRIVATE = "private"
+    IDENTIFIED = "identified"
+    DETAILED = "detailed"
+
+
+def _in_quiet_window(now: time, start: time, end: time) -> bool:
+    """Whether ``now`` falls inside the ``[start, end)`` local window.
+
+    Handles a window that wraps past midnight (e.g. ``22:00``–``07:30``) by
+    treating start > end as "outside the daytime gap".
+    """
+    if start == end:
+        return False
+    if start < end:
+        return start <= now < end
+    return now >= start or now < end
+
+
+@dataclass
+class NotificationPreference:
+    """A recipient's durable notification policy.
+
+    Attributes:
+        categories: Opted-in categories; a category not present is suppressed.
+        quiet_hours: Optional ``(start, end)`` local time window during which
+            non-urgent categories are suppressed. ``None`` means no quiet hours.
+        timezone: IANA name used to localise ``now`` for the quiet-hours test.
+        detail: Redaction level applied to the notification content.
+        urgent_overrides_quiet: When True, an urgent event is delivered even
+            inside quiet hours.
+    """
+
+    categories: Set[NotificationCategory] = field(default_factory=set)
+    quiet_hours: Optional[tuple] = None
+    timezone: str = "UTC"
+    detail: DetailLevel = DetailLevel.IDENTIFIED
+    urgent_overrides_quiet: bool = True
+
+    def allows(self, category: NotificationCategory) -> bool:
+        """Whether this recipient has opted into ``category``."""
+        return category in self.categories
+
+
+# A sensible default applied when a recipient has set nothing: the two
+# interaction-blocking categories are on, at the identified (no-content) detail
+# level, with no quiet hours.
+DEFAULT_PREFERENCE = NotificationPreference(
+    categories={
+        NotificationCategory.APPROVAL_REQUESTED,
+        NotificationCategory.AGENT_QUESTION,
+    },
+    detail=DetailLevel.IDENTIFIED,
+)
+
+
+@dataclass
+class NotificationDecision:
+    """Outcome of :func:`evaluate` for one (category, recipient, moment).
+
+    Attributes:
+        send: Whether the notification should be delivered at all.
+        reason: Machine/operator-readable reason (e.g. ``"quiet_hours"``,
+            ``"category_not_subscribed"``, ``"ok"``).
+        redacted_content: The content to actually push, redacted per the
+            recipient's detail level (``None`` when ``send`` is False).
+    """
+
+    send: bool
+    reason: str
+    redacted_content: Optional[str] = None
+
+
+def _redact(detail: DetailLevel, content: Optional[str], identity: Optional[str]) -> str:
+    """Apply the detail-level redaction to ``content``.
+
+    ``identity`` is an agent/session label used at the ``identified`` level so a
+    recipient learns *what* has activity without the transcript leaving the
+    gateway.
+    """
+    if detail == DetailLevel.DETAILED:
+        return content if content is not None else "You have new activity"
+    if detail == DetailLevel.IDENTIFIED:
+        return f"Activity in {identity}" if identity else "You have new activity"
+    return "You have new activity"
+
+
+def evaluate(
+    category: NotificationCategory,
+    prefs: Optional[NotificationPreference] = None,
+    *,
+    now: Optional[datetime] = None,
+    is_urgent: bool = False,
+    content: Optional[str] = None,
+    identity: Optional[str] = None,
+) -> NotificationDecision:
+    """Decide whether/what to notify for one event — pure, no side effects.
+
+    Args:
+        category: The event class being considered.
+        prefs: The recipient's preference; ``DEFAULT_PREFERENCE`` when ``None``.
+        now: The moment of evaluation (defaults to ``datetime.now`` localised to
+            the recipient's timezone). A naive ``now`` is treated as already in
+            the recipient's local zone.
+        is_urgent: Whether this specific event is urgent (may override quiet
+            hours when the recipient allows it).
+        content: The full message content, redacted per ``prefs.detail``.
+        identity: An agent/session label used at the ``identified`` detail level.
+
+    Returns:
+        A :class:`NotificationDecision`. ``send`` is False (with a ``reason``)
+        when the category is not subscribed or quiet hours suppress it.
+    """
+    if prefs is None:
+        prefs = DEFAULT_PREFERENCE
+
+    if not prefs.allows(category):
+        return NotificationDecision(send=False, reason="category_not_subscribed")
+
+    if prefs.quiet_hours is not None:
+        local_now = _localised_time(now, prefs.timezone)
+        start, end = prefs.quiet_hours
+        if _in_quiet_window(local_now, start, end):
+            if not (is_urgent and prefs.urgent_overrides_quiet):
+                return NotificationDecision(send=False, reason="quiet_hours")
+
+    return NotificationDecision(
+        send=True,
+        reason="urgent_override" if is_urgent and prefs.quiet_hours else "ok",
+        redacted_content=_redact(prefs.detail, content, identity),
+    )
+
+
+def _localised_time(now: Optional[datetime], tz_name: str) -> time:
+    """Return the wall-clock ``time`` at ``now`` in ``tz_name``.
+
+    A ``None`` ``now`` uses the current time. An aware ``now`` is converted to
+    ``tz_name``; a naive ``now`` is assumed to already be local. ``zoneinfo`` is
+    stdlib (3.9+) so this stays dependency-free; an unknown zone falls back to
+    the naive time rather than raising.
+    """
+    if now is None:
+        now = datetime.now()
+    if now.tzinfo is not None:
+        try:
+            from zoneinfo import ZoneInfo
+
+            now = now.astimezone(ZoneInfo(tz_name))
+        except Exception:
+            pass
+    return now.timetz().replace(tzinfo=None) if now.tzinfo else now.time()

src/praisonai-agents/praisonaiagents/rooms/__init__.py

@@ -30,10 +30,13 @@ from .protocols import (
     RoundRobinRoomPlanner,
     extract_mentions,
 )
+from .runtime import Room, RoomConfig
 
 __all__ = [
     "RoomEvent",
     "RoomTurnPlannerProtocol",
     "RoundRobinRoomPlanner",
     "extract_mentions",
+    "RoomConfig",
+    "Room",
 ]

src/praisonai-agents/praisonaiagents/rooms/runtime.py

@@ -0,0 +1,244 @@
+"""Room config + a minimal, storage-agnostic runtime that *enacts* the planner.
+
+Issue #5067. The pure turn planner (:mod:`praisonaiagents.rooms.protocols`)
+decides *who* speaks next; it was previously orphaned — nothing consumed it. A
+room needs one more thing to be usable end-to-end: something that appends each
+message to a transcript and drains :meth:`RoomTurnPlannerProtocol.plan_next`,
+running the scheduled agent each turn.
+
+This module supplies that piece while staying faithful to the package's
+"no heavy imports, no I/O" contract: the transcript is an in-memory list and the
+agents are injected as a mapping of ``id -> async callable``. The *durable*
+transcript store, platform delivery and CLI/YAML surface remain a wrapper
+(``praisonai-bot``) concern; the wrapper can subclass or wrap :class:`Room`,
+persist :attr:`Room.transcript`, and replay it after a restart — determinism of
+the planner guarantees the identical schedule resumes.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+from typing import Awaitable, Callable, Dict, List, Optional, Sequence
+
+from .protocols import RoomEvent, RoomTurnPlannerProtocol, RoundRobinRoomPlanner
+
+__all__ = ["RoomConfig", "Room"]
+
+
+# An agent turn: given the rendered transcript, produce this speaker's reply.
+AgentReply = Callable[[str], Awaitable[str]]
+
+
+@dataclass
+class RoomConfig:
+    """Declarative description of a conversation room (roster + planner caps).
+
+    A room is the live, message-driven counterpart of ``AgentTeam``: a named
+    roster of agent ids that take bounded, ``@``-addressed turns in one shared
+    transcript. This is the small, pure config the wrapper binds to a
+    channel/thread exactly where a single ``agent`` is bound today; the planner
+    itself already exists in :mod:`praisonaiagents.rooms.protocols`.
+
+    Attributes:
+        name: Room identifier (used when binding to a channel, e.g. ``room:standup``).
+        roster: Ordered agent ids that take turns.
+        planner: Planner selector; ``"round_robin"`` -> :class:`RoundRobinRoomPlanner`.
+        max_rounds: Hard cap on turn-taking rounds (passed to the planner).
+        max_messages: Hard cap on messages per human-initiated activity.
+    """
+
+    name: str
+    roster: List[str] = field(default_factory=list)
+    planner: str = "round_robin"
+    max_rounds: int = 3
+    max_messages: int = 20
+
+    def __post_init__(self) -> None:
+        if not self.name:
+            raise ValueError("RoomConfig.name must be a non-empty string")
+        if self.planner != "round_robin":
+            raise ValueError(
+                f"Unknown room planner {self.planner!r}; expected 'round_robin'"
+            )
+        if self.max_rounds < 1:
+            raise ValueError("max_rounds must be >= 1")
+        if self.max_messages < 1:
+            raise ValueError("max_messages must be >= 1")
+
+    def build_planner(self) -> RoomTurnPlannerProtocol:
+        """Construct the planner this config selects."""
+        return RoundRobinRoomPlanner(
+            max_rounds=self.max_rounds, max_messages=self.max_messages
+        )
+
+    def to_dict(self) -> Dict[str, object]:
+        """Convert to a plain dict (for YAML round-tripping / diagnostics)."""
+        return {
+            "name": self.name,
+            "roster": list(self.roster),
+            "planner": self.planner,
+            "max_rounds": self.max_rounds,
+            "max_messages": self.max_messages,
+        }
+
+    @classmethod
+    def from_dict(cls, name: str, data: Dict[str, object]) -> "RoomConfig":
+        """Create from a parsed ``rooms.<name>`` YAML mapping.
+
+        Accepts either ``roster:`` or the issue's ``agents:`` key for the list of
+        agent ids so the declarative surface reads naturally.
+        """
+        data = data or {}
+        roster = data.get("roster")
+        key = "roster"
+        if roster is None:
+            roster = data.get("agents", [])
+            key = "agents"
+        if roster is None:
+            roster = []
+        # A roster must be a sequence of ids. Reject a bare string (which would
+        # silently split into character-sized entries, e.g. "researcher" ->
+        # ['r','e','s',...]) and mappings, with an actionable error instead of a
+        # room that is silently unusable.
+        if isinstance(roster, (str, bytes)) or isinstance(roster, Mapping):
+            raise ValueError(
+                f"RoomConfig {name!r}: {key!r} must be a list of agent ids, "
+                f"got {type(roster).__name__}; "
+                f"use a YAML list, e.g. {key}: [researcher, critic]"
+            )
+        if not isinstance(roster, Sequence):
+            raise ValueError(
+                f"RoomConfig {name!r}: {key!r} must be a list of agent ids, "
+                f"got {type(roster).__name__}"
+            )
+        return cls(
+            name=name,
+            roster=[str(a) for a in roster],
+            planner=str(data.get("planner", "round_robin")),
+            max_rounds=int(data.get("max_rounds", 3)),
+            max_messages=int(data.get("max_messages", 20)),
+        )
+
+
+class Room:
+    """Minimal storage-agnostic runtime that enacts the turn planner.
+
+    :class:`Room` owns the (in-memory) transcript and drains the planner: after
+    each incoming :class:`RoomEvent` it repeatedly asks the planner for the next
+    speaker and runs that agent, appending the reply, until the planner settles.
+    It performs no I/O and holds no durable state — the wrapper supplies durable
+    persistence/delivery by subclassing or wrapping this and persisting
+    :attr:`transcript`. Because the planner is pure, replaying a persisted
+    transcript resumes the identical schedule.
+
+    Args:
+        config: The room's roster + planner caps.
+        agents: Mapping of agent id -> async callable ``reply(prompt) -> str``.
+            Typically wraps ``Agent.achat``. Only ids in ``config.roster`` are
+            scheduled; unknown roster ids are skipped (treated as a pass).
+        planner: Optional planner override (defaults to ``config.build_planner()``).
+        render: Optional transcript renderer -> the prompt handed to each agent.
+    """
+
+    def __init__(
+        self,
+        config: RoomConfig,
+        agents: Dict[str, AgentReply],
+        planner: Optional[RoomTurnPlannerProtocol] = None,
+        render: Optional[Callable[[List[RoomEvent]], str]] = None,
+    ) -> None:
+        self.config = config
+        self.agents = agents
+        self.planner = planner or config.build_planner()
+        self._render = render or self._default_render
+        self.transcript: List[RoomEvent] = []
+        # Serialize the whole append-and-drain per room: concurrent
+        # ``on_message`` calls must not interleave against the shared transcript,
+        # or two drains could schedule and execute the same agent twice.
+        self._lock = asyncio.Lock()
+
+    @staticmethod
+    def _default_render(transcript: List[RoomEvent]) -> str:
+        """Render the transcript as ``speaker: content`` lines."""
+        return "\n".join(
+            f"{ev.speaker}: {ev.content}" for ev in transcript if ev.content
+        )
+
+    def _current_round(self) -> int:
+        """The round currently being filled in the active (post-human) slice."""
+        roster_set = set(self.config.roster)
+        last_human = -1
+        for idx, ev in enumerate(self.transcript):
+            if ev.speaker not in roster_set:
+                last_human = idx
+        activity = self.transcript[last_human:] if last_human >= 0 else []
+        return max((ev.round for ev in activity), default=0)
+
+    def _scheduled_round(self, speaker: str) -> int:
+        """Round the planner scheduled ``speaker`` into for the current activity.
+
+        Only :class:`RoundRobinRoomPlanner` defines round semantics (round 0
+        fan-out, later rounds admitted by ``@``-mentions). For any other custom
+        planner the round taxonomy is unknown, so we simply tag turns with the
+        activity's current round rather than inventing round-robin admission —
+        imposing round-robin rules on a custom planner would mislabel its turns
+        and could feed it an event that makes it re-select and loop unbounded.
+        """
+        if not isinstance(self.planner, RoundRobinRoomPlanner):
+            return self._current_round()
+
+        from .protocols import extract_mentions
+
+        roster_set = set(self.config.roster)
+        last_human = -1
+        for idx, ev in enumerate(self.transcript):
+            if ev.speaker not in roster_set:
+                last_human = idx
+        activity = self.transcript[last_human:] if last_human >= 0 else []
+        current_round = max((ev.round for ev in activity), default=0)
+        max_rounds = getattr(self.planner, "max_rounds", self.config.max_rounds)
+
+        for rnd in range(current_round, max_rounds):
+            spoken = {ev.speaker for ev in activity if ev.round == rnd}
+            if rnd == 0:
+                admitted = set(roster_set)
+            else:
+                admitted = set()
+                for ev in activity:
+                    if ev.round != rnd - 1 or ev.speaker not in roster_set:
+                        continue
+                    for name in extract_mentions(ev.content):
+                        if name in roster_set:
+                            admitted.add(name)
+            if speaker in admitted and speaker not in spoken:
+                return rnd
+        return current_round
+
+    async def on_message(self, event: RoomEvent) -> List[RoomEvent]:
+        """Append ``event`` then drain the plan, returning the agent replies.
+
+        Each new human (non-roster) message opens a fresh round-0 activity; the
+        planner scopes its accounting to messages since that last human turn, so
+        a settled room reliably restarts when the next human speaks.
+        """
+        async with self._lock:
+            self.transcript.append(event)
+            produced: List[RoomEvent] = []
+            while True:
+                nxt = self.planner.plan_next(self.config.roster, self.transcript)
+                if nxt is None:
+                    break
+                rnd = self._scheduled_round(nxt)
+                reply = self.agents.get(nxt)
+                if reply is None:
+                    # Unknown roster id: record a pass so the planner advances
+                    # instead of re-scheduling the same missing agent forever.
+                    turn = RoomEvent(speaker=nxt, content="", round=rnd, passed=True)
+                else:
+                    content = await reply(self._render(self.transcript))
+                    turn = RoomEvent(speaker=nxt, content=content or "", round=rnd)
+                self.transcript.append(turn)
+                produced.append(turn)
+            return produced

src/praisonai-agents/praisonaiagents/scheduler/__init__.py

@@ -25,6 +25,7 @@ if TYPE_CHECKING:
     from .store import FileScheduleStore
     from .config_store import ConfigYamlScheduleStore
     from .parser import parse_schedule
+    from .due import quota_hold_from_failure
     from .runner import ScheduleRunner
     from .loop import ScheduleLoop, InProcessScheduleProvider
     from .protocols import (
@@ -106,6 +107,11 @@ def __getattr__(name: str):
         _module_cache[name] = parse_schedule
         return parse_schedule
 
+    if name == "quota_hold_from_failure":
+        from .due import quota_hold_from_failure
+        _module_cache[name] = quota_hold_from_failure
+        return quota_hold_from_failure
+
     if name == "ScheduleRunner":
         from .runner import ScheduleRunner
         _module_cache[name] = ScheduleRunner
@@ -176,6 +182,7 @@ __all__ = [
     "FileScheduleStore",
     "ConfigYamlScheduleStore",
     "parse_schedule",
+    "quota_hold_from_failure",
     "ScheduleRunner",
     "ScheduleLoop",
     "InProcessScheduleProvider",

src/praisonai-agents/praisonaiagents/scheduler/config_store.py

@@ -15,7 +15,7 @@ import time
 from typing import Dict, Iterator, List, Optional
 
 from .models import ScheduleJob, RunRecord
-from .due import is_due as _is_due, resolve_schedule_timezone
+from .due import is_due as _is_due, is_misfire as _is_misfire, resolve_schedule_timezone
 from .hook_emit import emit_schedule_add, emit_schedule_remove
 
 logger = get_logger(__name__)
@@ -202,6 +202,10 @@ class ConfigYamlScheduleStore:
         # a later ``remove(job.id)`` by the runner is then a no-op, so a spent
         # one-shot yields exactly one SCHEDULE_REMOVE.
         auto_removed: List[ScheduleJob] = []
+        # (job_id, job_name) for occurrences suppressed by the misfire policy.
+        # Logged as ``missed`` history after the lock so a dropped occurrence is
+        # observable instead of vanishing silently.
+        missed: List[tuple] = []
         with self._lock, self._file_lock():
             # Re-read from disk so we observe cross-process claims/leases.
             self._reload_locked()
@@ -224,6 +228,16 @@ class ConfigYamlScheduleStore:
                         self._job_state.pop(job.id, None)
                         changed = True
                     continue
+                # Misfire policy: an occurrence that landed while the process
+                # was down and is now older than ``misfire_grace_seconds`` is
+                # recorded as ``missed`` (observable) rather than fired stale or
+                # dropped silently. Advance past the slot so it is not seen due
+                # again on the next poll.
+                if _is_misfire(job, now, self._default_timezone):
+                    job.last_run_at = now
+                    missed.append((job.id, job.name))
+                    changed = True
+                    continue
                 # Win the claim: pre-advance + lease atomically.
                 job.last_run_at = now
                 # Count the fire here (needed so ``should_retire()`` can drop a
@@ -254,6 +268,14 @@ class ConfigYamlScheduleStore:
                     return []
         for job in auto_removed:
             emit_schedule_remove(job)
+        # Record suppressed occurrences so a missed run is visible in history
+        # rather than silently dropped (log_run takes only the thread lock).
+        for job_id, job_name in missed:
+            self.log_run(
+                job_id=job_id,
+                status="missed",
+                job_name=job_name,
+            )
         return claimed
 
     def complete(self, job_id: str, owner_id: str) -> None:

src/praisonai-agents/praisonaiagents/scheduler/due.py

@@ -76,6 +76,158 @@ def next_fire_time(
     return croniter(cron_expr, base_datetime).get_next(datetime).timestamp()
 
 
+def scheduled_instant(
+    job: Any,
+    now: float,
+    default_timezone: str | None = None,
+) -> float | None:
+    """Return the epoch of the occurrence this due job is firing *for*.
+
+    When a recurring job becomes due at ``now`` it is firing for a specific
+    scheduled slot that may lie in the past (the process was down when the
+    slot arrived). This returns that slot's canonical instant so a misfire
+    policy can decide whether recovery is still within an acceptable grace
+    window. ``None`` is returned when there is no meaningful past slot (a
+    never-run interval job, or a kind/expression that cannot be evaluated).
+
+    - ``every``: the *first* slot that came due after ``last_run_at``
+      (``last_run_at + every_seconds``). Its age measures how long the job has
+      been overdue — the meaningful lateness for a coalesced recovery.
+    - ``cron``: the last cron fire at or before ``now`` (the slot being
+      recovered for a daily/weekly schedule).
+    - ``at``: the target instant itself.
+    """
+    sched = job.schedule
+    if sched.kind == "every":
+        if sched.every_seconds is None or job.last_run_at is None:
+            return None
+        if now - job.last_run_at < sched.every_seconds:
+            return None
+        return job.last_run_at + sched.every_seconds
+    if sched.kind == "at":
+        if sched.at is None:
+            return None
+        try:
+            return localize_wall_clock(
+                datetime.fromisoformat(sched.at),
+                sched.tz or default_timezone,
+            ).timestamp()
+        except (ValueError, TypeError):
+            return None
+    if sched.kind == "cron":
+        if sched.cron_expr is None:
+            return None
+        try:
+            from croniter import croniter  # type: ignore[import-untyped]
+        except ImportError:
+            return None
+        try:
+            base_datetime = datetime.fromtimestamp(
+                now, resolve_schedule_timezone(sched.tz or default_timezone)
+            )
+            return croniter(sched.cron_expr, base_datetime).get_prev(datetime).timestamp()
+        except (ValueError, KeyError, TypeError):
+            return None
+    return None
+
+
+def is_misfire(
+    job: Any,
+    now: float,
+    default_timezone: str | None = None,
+) -> bool:
+    """Whether a *due* job's occurrence should be suppressed as a misfire.
+
+    Shared by every atomic-claim store so the misfire decision is defined once.
+    Returns ``True`` when the occurrence the job is firing for is older than
+    ``misfire_grace_seconds`` — a stale run whose slot passed while the process
+    was down. Such an occurrence is recorded as ``missed`` (observable) instead
+    of firing a late, no-longer-useful run. An occurrence still within the
+    grace window (or a first run) coalesces into a single normal fire on
+    recovery, the pre-existing behaviour.
+
+    With no grace set this is always ``False`` — existing jobs keep firing
+    exactly as before. A first run (``last_run_at is None``) is never a misfire
+    for any kind: there is no missed *recurrence* to suppress, only the initial
+    fire, which must always run (matching ``is_due``'s never-run branches).
+    """
+    grace = getattr(job, "misfire_grace_seconds", None)
+    if grace is None:
+        return False
+    # A first run is the initial fire, not a recovered recurrence — always let
+    # it run. ``every`` already encodes this (``scheduled_instant`` returns
+    # ``None`` with no ``last_run_at``); make it explicit for ``cron``/``at``
+    # too so a never-run job past a stale slot is not marked missed unfired.
+    if getattr(job, "last_run_at", None) is None:
+        return False
+    # Tolerate a mis-typed grace (e.g. a hand-edited quoted YAML value that
+    # slipped past coercion): a bad policy value must never raise and abort the
+    # whole claim pass — treat it as "no misfire" so other due jobs still fire.
+    try:
+        grace_val = float(grace)
+    except (TypeError, ValueError):
+        return False
+    instant = scheduled_instant(job, now, default_timezone)
+    if instant is None:
+        return False
+    age = now - instant
+    if age <= 0:
+        return False
+    return age > grace_val
+
+
+# Upper bound on a single quota hold (24h) so a bogus/huge provider Retry-After
+# cannot park a recurring job indefinitely.
+_MAX_HOLD_SECONDS = 86_400.0
+
+
+def quota_hold_from_failure(
+    error: Any,
+    now: float,
+    slack_seconds: float = 60.0,
+) -> float | None:
+    """Return an epoch to park a job past a provider's rate-limit window.
+
+    Pure decision helper for the "provider-quota-aware hold": given a run's
+    failure it decides whether the job should stop re-firing until the
+    provider's own reset window elapses. Returns ``now + Retry-After + slack``
+    when the failure is a rate-limit / quota signal carrying a usable reset
+    hint, else ``None`` (no hold — a non-quota failure keeps today's behaviour).
+
+    Reuses the core error classifier so a 429/quota error is recognised by
+    exception type, HTTP status code, or message, and the reset window is read
+    from the provider's ``Retry-After`` header / ``retry_after`` attribute /
+    message text — no new parsing surface. The small ``slack_seconds`` avoids
+    re-firing the instant the window reopens (clock skew / provider rounding).
+
+    Args:
+        error: The failure — an ``Exception`` (preferred, so the classifier can
+            read the response headers) or an error string.
+        now: Epoch the failure was observed at.
+        slack_seconds: Extra seconds added past the provider's reset window.
+
+    Returns:
+        The epoch before which the job should be parked, or ``None`` when the
+        failure is not a quota signal or carries no usable reset window.
+    """
+    from praisonaiagents.llm.error_classifier import (
+        ErrorCategory,
+        classify_error,
+        extract_retry_after,
+    )
+
+    exc = error if isinstance(error, Exception) else Exception(str(error or ""))
+    if classify_error(exc) != ErrorCategory.RATE_LIMIT:
+        return None
+    # Honour the provider's full reset window (unlike in-tick backoff, which caps
+    # at 5 minutes) so a long quota window parks the job instead of re-firing —
+    # but bound it to 24h so a bogus/huge value can't park a job indefinitely.
+    retry_after = extract_retry_after(exc, cap_seconds=_MAX_HOLD_SECONDS)
+    if not retry_after or retry_after <= 0:
+        return None
+    return now + float(retry_after) + max(float(slack_seconds), 0.0)
+
+
 def is_due(
     job: Any,
     now: float,
@@ -101,6 +253,17 @@ def is_due(
     run_count = getattr(job, "run_count", 0)
     if max_runs is not None and run_count >= max_runs:
         return False
+    # Provider-quota hold short-circuits before the kind check: while a job is
+    # parked past a provider's known-closed window (a 429/quota ``Retry-After``
+    # captured on a prior failure) it is not due, so it stops re-firing and
+    # re-failing every tick against a benched provider. Any intervening fires
+    # coalesce — the job becomes due again at the first legal instant once the
+    # hold elapses. The executor clears ``hold_until`` on the first run that
+    # reaches the model (proof the window reopened).
+    hold_until = getattr(job, "hold_until", None)
+    if hold_until is not None and now < hold_until:
+        return False
+
     until = getattr(job, "until", None)
     if until is not None:
         try:

src/praisonai-agents/praisonaiagents/scheduler/models.py

@@ -10,6 +10,23 @@ from dataclasses import dataclass, field
 from typing import Any, Dict, List, Literal, Optional
 
 
+def _coerce_optional_float(value: Any) -> Optional[float]:
+    """Coerce a persisted/hand-edited value to ``Optional[float]``.
+
+    A hand-written config.yaml may quote a numeric field (``'3600'``) so it
+    arrives as a string. Left as-is it would raise ``TypeError`` in a later
+    numeric comparison (e.g. the misfire age check), aborting the whole claim
+    pass and starving every other due job. Coerce leniently: ``None`` stays
+    ``None``; a non-numeric value is treated as unset rather than exploding.
+    """
+    if value is None:
+        return None
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
 @dataclass
 class Schedule:
     """When to run a scheduled job.
@@ -198,12 +215,16 @@ class RunRecord:
         job_id: ID of the job that was executed.
         job_name: Human-readable name of the job.
         status: Execution status. One of ``"succeeded"``, ``"failed"``,
-                ``"skipped"``, or ``"no_change"``. ``"no_change"`` is the
-                stateful "monitor mode" outcome — a watched source was
+                ``"skipped"``, ``"no_change"``, or ``"missed"``. ``"no_change"``
+                is the stateful "monitor mode" outcome — a watched source was
                 unchanged since the last tick, so the model turn was suppressed
                 silently (no tokens, no delivery). It is distinct from
                 ``"skipped"`` (a generic gate go/no-go) so an operator can tell
                 "nothing changed" apart from "the gate said don't run".
+                ``"missed"`` records a recurring occurrence that landed while
+                the process was down and fell outside the job's
+                ``misfire_grace_seconds`` — it was not run, but is recorded
+                (not silently dropped) so an operator can see the gap.
         result: Agent response text (truncated if very long).
         error: Error message if status is ``"failed"``.
         duration: Wall-clock seconds for execution.
@@ -213,7 +234,7 @@ class RunRecord:
 
     job_id: str
     job_name: str = ""
-    status: Literal["succeeded", "failed", "skipped", "no_change"] = "succeeded"
+    status: Literal["succeeded", "failed", "skipped", "no_change", "missed"] = "succeeded"
     result: Optional[str] = None
     error: Optional[str] = None
     duration: float = 0.0
@@ -374,6 +395,28 @@ class ScheduleJob:
                  ``skipped`` (no tokens, no delivery) so a downstream never runs
                  on empty inputs. Consistent with the existing ``skipped``
                  outcome.
+        misfire_grace_seconds: Misfire policy for a recurring occurrence missed
+                 while the process was down — the maximum age (seconds) of the
+                 occurrence being recovered for it to still run. When the slot
+                 the job is firing for is older than this on recovery, the
+                 occurrence is recorded as ``missed`` (an observable
+                 :class:`RunRecord`) rather than fired — so a daily brief whose
+                 gateway was down for hours does not surface a stale run long
+                 after it was useful, yet the gap is never silently dropped. An
+                 occurrence still within the window (or a first run) coalesces
+                 into a single normal fire on recovery. ``None`` (default)
+                 disables the check, so existing jobs are byte-for-byte
+                 unchanged and keep firing exactly as before.
+        hold_until: Optional epoch instant before which the job is parked and
+                 not due, even when its schedule would otherwise fire. Set when
+                 a run fails with a provider rate-limit / quota signal (the
+                 provider's own ``Retry-After`` / reset window plus a small
+                 slack): the recurring job stops re-firing and re-failing every
+                 tick against a benched provider. Intervening fires coalesce
+                 into the first legal instant after the hold; the executor
+                 clears it on the first run that reaches the model (proof the
+                 window reopened). ``None`` (default) keeps today's behaviour,
+                 so jobs that never hit a quota wall are unchanged.
     """
 
     name: str = ""
@@ -403,8 +446,10 @@ class ScheduleJob:
     context_from: Optional[List[str]] = None
     context_max_chars: int = 4000
     on_missing_context: Literal["run", "skip"] = "run"
+    misfire_grace_seconds: Optional[float] = None
     backend: Optional[str] = None
     backend_options: Dict[str, Any] = field(default_factory=dict)
+    hold_until: Optional[float] = None
 
     # ── bounded-run retirement ───────────────────────────────────────
 
@@ -519,6 +564,10 @@ class ScheduleJob:
                 d["context_max_chars"] = self.context_max_chars
             if self.on_missing_context != "run":
                 d["on_missing_context"] = self.on_missing_context
+        # Misfire policy. Only persist when configured so a job with no misfire
+        # grace stays byte-for-byte unchanged.
+        if self.misfire_grace_seconds is not None:
+            d["misfire_grace_seconds"] = self.misfire_grace_seconds
         # External CLI backend action. Only persist when configured so agent
         # and command jobs stay byte-for-byte unchanged; options are opaque to
         # the core (the executor validates them against the backend registry).
@@ -526,6 +575,11 @@ class ScheduleJob:
             d["backend"] = self.backend
             if self.backend_options:
                 d["backend_options"] = dict(self.backend_options)
+        # Provider-quota hold. Only persist when parked so a job that never hit
+        # a quota wall stays byte-for-byte unchanged; the parked instant must
+        # survive a restart so a benched job stays parked across processes.
+        if self.hold_until is not None:
+            d["hold_until"] = self.hold_until
         # Atomic-claim lease metadata (set dynamically by stores that support
         # ``claim_due``). Persisted so a lease is visible across processes and
         # survives a restart; omitted when no lease is held.
@@ -577,12 +631,16 @@ class ScheduleJob:
             context_from=context_from,
             context_max_chars=d.get("context_max_chars", 4000),
             on_missing_context=d.get("on_missing_context", "run"),
+            misfire_grace_seconds=_coerce_optional_float(
+                d.get("misfire_grace_seconds")
+            ),
             backend=d.get("backend"),
             backend_options=(
                 dict(d["backend_options"])
                 if isinstance(d.get("backend_options"), dict)
                 else {}
             ),
+            hold_until=d.get("hold_until"),
         )
         # Restore atomic-claim lease metadata if present (see ``to_dict``).
         job._lease_until = d.get("lease_until", 0.0) or 0.0

src/praisonai-agents/praisonaiagents/scheduler/store.py

@@ -14,7 +14,7 @@ import time
 from typing import Dict, Iterator, List, Optional
 
 from .models import ScheduleJob, RunRecord
-from .due import is_due as _is_due
+from .due import is_due as _is_due, is_misfire as _is_misfire
 from .hook_emit import emit_schedule_add, emit_schedule_remove
 
 logger = get_logger(__name__)
@@ -187,6 +187,10 @@ class FileScheduleStore:
         # a later ``remove(job.id)`` by the runner is then a no-op, so a spent
         # one-shot yields exactly one SCHEDULE_REMOVE.
         auto_removed: List[ScheduleJob] = []
+        # (job_id, job_name) for occurrences suppressed by the misfire policy.
+        # Logged as ``missed`` history after the lock so a dropped occurrence is
+        # observable instead of vanishing silently.
+        missed: List[tuple] = []
         with self._lock, self._file_lock():
             # Re-read from disk so we observe cross-process claims/leases.
             self._reload_locked()
@@ -208,6 +212,18 @@ class FileScheduleStore:
                         auto_removed.append(self._jobs.pop(job.id))
                         changed = True
                     continue
+                # Misfire policy: the job is due, but the occurrence it is
+                # firing for may have landed while the process was down. Decide
+                # whether to run it now or record it as ``missed`` (observable)
+                # rather than firing a stale run or dropping it silently.
+                if _is_misfire(job, now):
+                    # Advance the schedule past the missed slot (so the next
+                    # poll does not see it as due again) and record the miss,
+                    # without claiming or firing.
+                    job.last_run_at = now
+                    missed.append((job.id, job.name))
+                    changed = True
+                    continue
                 # Win the claim: pre-advance + lease atomically.
                 job.last_run_at = now
                 # Count the fire here (needed so ``should_retire()`` can drop a
@@ -237,6 +253,15 @@ class FileScheduleStore:
                     return []
         for job in auto_removed:
             emit_schedule_remove(job)
+        # Record suppressed occurrences so a missed run is visible in history
+        # rather than silently dropped. Logged outside the file lock (log_run
+        # takes only the thread lock) to keep the critical section short.
+        for job_id, job_name in missed:
+            self.log_run(
+                job_id=job_id,
+                status="missed",
+                job_name=job_name,
+            )
         return claimed
 
     def complete(self, job_id: str, owner_id: str) -> None:

src/praisonai-agents/praisonaiagents/secrets.py

@@ -39,6 +39,8 @@ __all__ = [
     "register_resolver",
     "register_secret_for_redaction",
     "redact_secrets",
+    "redact_outbound",
+    "OutboundRedactor",
     "is_secret_ref",
 ]
 
@@ -287,3 +289,85 @@ def redact_secrets(text: str) -> str:
         if secret in text:
             text = text.replace(secret, _REDACTED)
     return text
+
+
+# ────────────────────────────────────────────────────────────────────────────
+# Outbound redaction — the safe-by-default reply-path primitive (Issue #5055).
+#
+# ``redact_secrets`` masks only values the framework has *seen* (resolved
+# credentials). An unregistered secret surfaced in a tool result or a document
+# would still slip through, so the outbound primitive additionally masks tokens
+# by *shape* (API keys, bearer tokens, ``AKIA…``, private-key blocks). This is
+# the counterpart to log redaction: it is applied to text on its way to a chat
+# user, before dispatch, by the wrapper's delivery seam.
+# ────────────────────────────────────────────────────────────────────────────
+
+# Credential-shaped patterns, matched by value (not key=value), so a naked token
+# echoed by the model or embedded in a tool result is caught even when never
+# registered. Kept deliberately narrow to avoid over-redacting ordinary text.
+_CREDENTIAL_SHAPE_PATTERNS = (
+    # OpenAI-style keys (sk-..., sk-proj-...), Anthropic (sk-ant-...)
+    r"\bsk-(?:ant-|proj-)?[A-Za-z0-9_-]{16,}\b",
+    # AWS access key id
+    r"\bAKIA[0-9A-Z]{16}\b",
+    # GitHub tokens (ghp_, gho_, ghu_, ghs_, ghr_, github_pat_)
+    r"\bgh[pousr]_[A-Za-z0-9]{16,}\b",
+    r"\bgithub_pat_[A-Za-z0-9_]{22,}\b",
+    # Google API key
+    r"\bAIza[0-9A-Za-z_-]{35}\b",
+    # Slack token
+    r"\bxox[baprs]-[0-9A-Za-z-]{10,}\b",
+    # Bearer token in an Authorization-style value
+    r"\bBearer\s+[A-Za-z0-9._-]{16,}\b",
+    # PEM private-key block
+    r"-----BEGIN (?:RSA |EC |OPENSSH |DSA |PGP )?PRIVATE KEY-----"
+    r"[\s\S]*?-----END (?:RSA |EC |OPENSSH |DSA |PGP )?PRIVATE KEY-----",
+)
+
+_credential_shape_re = None
+
+
+def _get_credential_shape_re():
+    """Compile the credential-shape matcher lazily (stdlib ``re`` only)."""
+    global _credential_shape_re
+    if _credential_shape_re is None:
+        import re
+
+        _credential_shape_re = re.compile(
+            "|".join(f"(?:{p})" for p in _CREDENTIAL_SHAPE_PATTERNS)
+        )
+    return _credential_shape_re
+
+
+def redact_outbound(text: str) -> str:
+    """Scrub secrets from text on its way to a chat user (safe-by-default).
+
+    Combines the registered-value registry (:func:`redact_secrets`, holding
+    every resolved gateway credential) with a credential-shape regex so an
+    *unregistered* token surfaced in a tool result is still masked. Personal
+    data is intentionally out of scope here — an opt-in PII policy layers on top
+    via the wrapper/plugin hook — so ordinary text is never over-redacted.
+
+    Zero dependency, never raises: a scrubber error returns the registered-value
+    result rather than blocking delivery.
+    """
+    if not text or not isinstance(text, str):
+        return text
+    text = redact_secrets(text)
+    try:
+        text = _get_credential_shape_re().sub(_REDACTED, text)
+    except Exception:  # pragma: no cover — never block delivery on a regex bug
+        pass
+    return text
+
+
+@runtime_checkable
+class OutboundRedactor(Protocol):
+    """Overridable contract for scrubbing outbound text before dispatch.
+
+    The default implementation is :func:`redact_outbound`. A deployment may
+    supply its own (e.g. one that also applies a PII policy) and inject it at
+    the wrapper's delivery seam without importing anything heavy into core.
+    """
+
+    def redact(self, text: str) -> str: ...

src/praisonai-agents/praisonaiagents/session/api.py

@@ -13,6 +13,7 @@ import uuid
 import json
 import time
 import logging
+from copy import deepcopy
 from typing import Any, Dict, List, Optional, TYPE_CHECKING
 
 logger = logging.getLogger(__name__)
@@ -332,7 +333,9 @@ class Session:
         histories = (getattr(data, "metadata", None) or {}).get(AGENT_HISTORY_KEY)
         if not isinstance(histories, dict):
             return {}
-        return {k: v for k, v in histories.items() if isinstance(v, list) and v}
+        # An empty list is an explicit clear, not a missing transcript. Keep
+        # it so current records override legacy and Memory fallback history.
+        return {k: v for k, v in histories.items() if isinstance(v, list)}
 
     def _merged_agent_histories(self, session_store) -> Dict[str, List[Dict[str, Any]]]:
         """All known sub-agent transcripts: legacy records under current ones.
@@ -502,6 +505,10 @@ class Session:
     ) -> bool:
         """Persist one sub-agent transcript onto this session's own record.
 
+        Built-in stores merge the entry under their session file lock. Older
+        custom stores retain the bounded read/write verification fallback below;
+        that fallback cannot guarantee atomic preservation of concurrent entries.
+
         The store replaces the whole ``AGENT_HISTORY_KEY`` map per write, so a
         naive read-then-replace would let a concurrent writer (another agent
         under the same parent) clobber this agent's entry, or vice versa. We
@@ -509,6 +516,14 @@ class Session:
         *other* key changed underneath us, retrying a bounded number of times.
         Legacy per-agent records are always merged in (they stay on disk).
         """
+        merge_map = getattr(session_store, "merge_session_metadata_map", None)
+        if callable(merge_map):
+            # Built-in stores merge inside their locked read/modify/write path.
+            # A post-write comparison cannot detect entries lost to a stale map.
+            return merge_map(
+                self.session_id, AGENT_HISTORY_KEY, {agent_key: messages},
+                defaults=self._legacy_agent_histories(session_store),
+            )
         update_meta = getattr(session_store, "update_session_metadata", None)
         if not callable(update_meta):
             return False
@@ -524,7 +539,7 @@ class Session:
             # read and write; if the persisted map matches what we wrote, we win.
             persisted = self._stored_agent_histories(session_store)
             if persisted.get(agent_key) == messages and all(
-                persisted.get(k) == v for k, v in current.items()
+                persisted.get(k) == v for k, v in current.items() if k != agent_key
             ):
                 return True
         logger.warning(
@@ -564,16 +579,12 @@ class Session:
                         {
                             "role": msg.get("role", "user"),
                             "content": msg.get("content", ""),
+                            **deepcopy(msg),
                         }
                         for msg in chat_history
                         if isinstance(msg, dict)
                     ]
-                    if not messages:
-                        logging.debug(
-                            f"No chat history to persist for agent {agent_key} "
-                            f"in session {self.session_id}"
-                        )
-                    elif not self._store_agent_history(
+                    if not self._store_agent_history(
                         session_store, agent_key, messages
                     ):
                         logging.warning(

src/praisonai-agents/praisonaiagents/session/context.py

@@ -39,6 +39,7 @@ from typing import TYPE_CHECKING, Dict, List, Optional, Any
 if TYPE_CHECKING:
     from ..gateway.protocols import (
         ConversationRequestProtocol,
+        GatewayRequestChannelProtocol,
         OutboundMessengerProtocol,
         SendPolicyProtocol,
         GatewayStatusProtocol,
@@ -273,6 +274,41 @@ def clear_conversation_requester(token: Token) -> None:
         _CONVERSATION_REQUESTER.set(None)
 
 
+# ---------------------------------------------------------------------------
+# Server→client interactive request channel (Issue #5351)
+#
+# The running gateway registers a concrete GatewayRequestChannelProtocol impl
+# into this task-local slot so a blocked HITL turn can deliver an approval /
+# choice / input request to the connected gateway client and await the
+# correlated reply. When unbound (CLI / one-shot runs) the elicitation simply
+# falls back to its existing path — a clean gate, never a hang.
+# ---------------------------------------------------------------------------
+
+_REQUEST_CHANNEL: ContextVar[Optional["GatewayRequestChannelProtocol"]] = ContextVar(
+    "praisonai_request_channel", default=None
+)
+
+
+def register_request_channel(
+    channel: Optional["GatewayRequestChannelProtocol"],
+) -> Token:
+    """Register the active server→client request channel for this task."""
+    return _REQUEST_CHANNEL.set(channel)
+
+
+def get_request_channel() -> Optional["GatewayRequestChannelProtocol"]:
+    """Return the active request channel, or ``None`` if no gateway is running."""
+    return _REQUEST_CHANNEL.get()
+
+
+def clear_request_channel(token: Token) -> None:
+    """Restore the previous request channel using the token from register."""
+    try:
+        _REQUEST_CHANNEL.reset(token)
+    except (LookupError, ValueError):
+        _REQUEST_CHANNEL.set(None)
+
+
 # ---------------------------------------------------------------------------
 # Outbound send-policy guard (Issue #2226)
 #
@@ -386,6 +422,9 @@ __all__ = [
     "register_conversation_requester",
     "get_conversation_requester",
     "clear_conversation_requester",
+    "register_request_channel",
+    "get_request_channel",
+    "clear_request_channel",
     "register_send_policy",
     "get_send_policy",
     "clear_send_policy",

src/praisonai-agents/praisonaiagents/session/handoff.py

@@ -59,17 +59,22 @@ def _progress_section(checkpoint: Optional[Dict[str, Any]]) -> List[str]:
     if not isinstance(checkpoint, dict):
         return []
     completed = checkpoint.get("completed_steps")
-    total = checkpoint.get("total_steps") or checkpoint.get("step_count")
-    last = checkpoint.get("last_step") or checkpoint.get("current_step")
-    if completed is None and last is None:
-        return []
-    if total:
-        head = f"PROGRESS: {completed or 0} of {total} workflow steps done"
-    else:
-        head = f"PROGRESS: {completed or 0} workflow steps done"
-    if last:
-        head += f"; last completed: {last}"
-    return [head]
+    total = checkpoint.get("total_steps")
+    if total is None:
+        total = checkpoint.get("step_count")
+    last = checkpoint.get("last_step")
+    current = checkpoint.get("current_step")
+    parts: List[str] = []
+    if completed is not None:
+        if total is not None:
+            parts.append(f"{completed} of {total} workflow steps done")
+        else:
+            parts.append(f"{completed} workflow steps done")
+    if last is not None and last != "":
+        parts.append(f"last completed: {last}")
+    if current is not None and current != "":
+        parts.append(f"current step: {current}")
+    return ["PROGRESS: " + "; ".join(parts)] if parts else []
 
 
 def _recent_actions(recap_text: str) -> List[str]:

src/praisonai-agents/praisonaiagents/session/hierarchy.py

@@ -6,6 +6,7 @@ forking, and revert capabilities.
 """
 
 import copy
+import hashlib
 import json
 import logging
 from praisonaiagents._logging import get_logger
@@ -13,17 +14,27 @@ import os
 import threading
 import time
 import uuid
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, fields
 from datetime import datetime, timezone
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple
 
-from .store import SessionData, SessionMessage, DefaultSessionStore, FileLock
+from .store import (
+    SessionData,
+    SessionMessage,
+    DefaultSessionStore,
+    FileLock,
+    _LRUSessionCache,
+)
 
 logger = get_logger(__name__)
 
 @dataclass
 class SessionSnapshot:
-    """A snapshot of session state at a point in time."""
+    """A transcript snapshot; old records may contain only a message index.
+
+    Durable records pool repeated transcript entries; portable exports inline them.
+    Legacy records cannot recover history already discarded before upgrading.
+    """
     
     id: str = field(default_factory=lambda: str(uuid.uuid4()))
     session_id: str = ""
@@ -31,9 +42,11 @@ class SessionSnapshot:
     created_at: float = field(default_factory=time.time)
     label: Optional[str] = None
     metadata: Dict[str, Any] = field(default_factory=dict)
+    transcript: Optional[Dict[str, Any]] = None
+    invalidated: bool = False
     
-    def to_dict(self) -> Dict[str, Any]:
-        return {
+    def _header_dict(self) -> Dict[str, Any]:
+        data = {
             "id": self.id,
             "session_id": self.session_id,
             "message_index": self.message_index,
@@ -41,6 +54,15 @@ class SessionSnapshot:
             "label": self.label,
             "metadata": self.metadata,
         }
+        if self.invalidated:
+            data["invalidated"] = True
+        return data
+
+    def to_dict(self) -> Dict[str, Any]:
+        data = self._header_dict()
+        if self.transcript is not None:
+            data["transcript"] = copy.deepcopy(self.transcript)
+        return data
     
     @classmethod
     def from_dict(cls, data: Dict[str, Any]) -> "SessionSnapshot":
@@ -51,6 +73,8 @@ class SessionSnapshot:
             created_at=data.get("created_at", time.time()),
             label=data.get("label"),
             metadata=data.get("metadata", {}),
+            transcript=copy.deepcopy(data.get("transcript")),
+            invalidated=data.get("invalidated", False),
         )
 
 @dataclass
@@ -75,30 +99,130 @@ class ExtendedSessionData(SessionData):
             "title": self.title,
         })
         return d
+
+    def _to_storage_dict(self) -> Dict[str, Any]:
+        """Pool snapshot records in the atomic session file, not portable exports."""
+        data = super().to_dict()
+        data.update({
+            "parent_id": self.parent_id,
+            "forked_from_message_id": self.forked_from_message_id,
+            "children_ids": self.children_ids,
+            "is_shared": self.is_shared,
+            "title": self.title,
+        })
+        records = {}
+        record_ids = {}
+
+        def intern(value):
+            # Most transcript records contain text plus scalar fields and empty
+            # metadata. Reuse hashes of their immutable strings rather than
+            # re-encoding every archived text for every snapshot on each write.
+            if isinstance(value, dict) and all(
+                item is None or isinstance(item, (str, bool, int, float))
+                or isinstance(item, (dict, list)) and not item
+                for item in value.values()
+            ):
+                identity = ("flat", tuple(
+                    (name, type(item), None if isinstance(item, (dict, list)) else item)
+                    for name, item in value.items()
+                ))
+            else:
+                identity = ("json", json.dumps(value, ensure_ascii=False,
+                                               separators=(",", ":")))
+            key = record_ids.get(identity)
+            if key is None:
+                key = str(len(records))
+                record_ids[identity] = key
+                records[key] = copy.deepcopy(value)
+            return key
+
+        snapshots = []
+        for snapshot in self.snapshots:
+            saved = snapshot._header_dict()
+            required = {"messages", "archived_messages", "last_compaction"}
+            if (snapshot.transcript is not None and required <= snapshot.transcript.keys()
+                    and all(isinstance(snapshot.transcript[name], list)
+                            and all(isinstance(item, dict) for item in snapshot.transcript[name])
+                            for name in ("messages", "archived_messages"))
+                    and (snapshot.transcript["last_compaction"] is None
+                         or isinstance(snapshot.transcript["last_compaction"], dict))):
+                refs = {}
+                for name, value in snapshot.transcript.items():
+                    if name in ("messages", "archived_messages") and isinstance(value, list):
+                        refs[name] = {"items": [intern(item) for item in value]}
+                    else:
+                        refs[name] = {"value": intern(value)}
+                saved["transcript_refs"] = refs
+                # Readers that only understand positional/inline snapshots
+                # must reject this record instead of restoring a stale index.
+                saved["invalidated"] = True
+                saved["transcript_invalidated"] = snapshot.invalidated
+            elif snapshot.transcript is not None:
+                # Preserve legacy inline shapes rather than encoding an
+                # incomplete transcript as a valid pooled snapshot.
+                saved["transcript"] = copy.deepcopy(snapshot.transcript)
+            snapshots.append(saved)
+        data["snapshots"] = snapshots
+        if any("transcript_refs" in snapshot for snapshot in snapshots):
+            data["snapshot_storage"] = {"version": 1, "records": records}
+        return data
     
     @classmethod
     def from_dict(cls, data: Dict[str, Any]) -> "ExtendedSessionData":
-        messages = [
-            SessionMessage.from_dict(m) 
-            for m in data.get("messages", [])
-        ]
-        archived = [
-            SessionMessage.from_dict(m)
-            for m in (data.get("archived_messages") or [])
-        ]
-        snapshots = [
-            SessionSnapshot.from_dict(s)
-            for s in data.get("snapshots", [])
-        ]
+        base = SessionData.from_dict(data)
+        snapshots = []
+        for raw in data.get("snapshots", []):
+            if "transcript_refs" in raw:
+                storage = data.get("snapshot_storage")
+                if (not isinstance(storage, dict) or storage.get("version") != 1
+                        or not isinstance(storage.get("records"), dict)
+                        or not isinstance(raw["transcript_refs"], dict)
+                        or "transcript" in raw):
+                    raise ValueError("invalid snapshot storage")
+                records = storage["records"]
+                if not {"messages", "archived_messages", "last_compaction"} <= raw["transcript_refs"].keys():
+                    raise ValueError("incomplete snapshot transcript references")
+                transcript = {}
+
+                def clone_record(value):
+                    # JSON object keys/scalars are immutable. Copy mutable
+                    # fields without traversing every archived text again.
+                    if isinstance(value, dict):
+                        return {key: copy.deepcopy(item) if isinstance(item, (dict, list)) else item
+                                for key, item in value.items()}
+                    return copy.deepcopy(value)
+
+                try:
+                    for name, ref in raw["transcript_refs"].items():
+                        if not isinstance(ref, dict):
+                            raise ValueError("invalid snapshot reference")
+                        if name in ("messages", "archived_messages") and (
+                                set(ref) != {"items"} or not isinstance(ref["items"], list)):
+                            raise ValueError("invalid snapshot message references")
+                        if name == "last_compaction" and set(ref) != {"value"}:
+                            raise ValueError("invalid snapshot checkpoint reference")
+                        if set(ref) == {"items"} and isinstance(ref["items"], list):
+                            transcript[name] = [clone_record(records[key]) for key in ref["items"]]
+                        elif set(ref) == {"value"}:
+                            transcript[name] = clone_record(records[ref["value"]])
+                        else:
+                            raise ValueError("invalid snapshot reference")
+                        if name in ("messages", "archived_messages") and not all(
+                                isinstance(item, dict) for item in transcript[name]):
+                            raise ValueError("invalid snapshot message record")
+                        if name == "last_compaction" and transcript[name] is not None and not isinstance(transcript[name], dict):
+                            raise ValueError("invalid snapshot checkpoint record")
+                except (KeyError, TypeError) as exc:
+                    raise ValueError("missing snapshot record") from exc
+                snapshot = SessionSnapshot.from_dict({
+                    **raw, "invalidated": raw.get("transcript_invalidated", False),
+                })
+                snapshot.transcript = transcript
+            else:
+                snapshot = SessionSnapshot.from_dict(raw)
+            snapshots.append(snapshot)
         return cls(
-            session_id=data.get("session_id", ""),
-            messages=messages,
-            created_at=data.get("created_at", datetime.now(timezone.utc).isoformat()),
-            updated_at=data.get("updated_at", datetime.now(timezone.utc).isoformat()),
-            agent_name=data.get("agent_name"),
-            user_id=data.get("user_id"),
-            metadata=data.get("metadata", {}),
-            archived_messages=archived,
+            **{descriptor.name: getattr(base, descriptor.name) for descriptor in fields(SessionData)},
             parent_id=data.get("parent_id"),
             forked_from_message_id=data.get("forked_from_message_id"),
             children_ids=data.get("children_ids", []),
@@ -111,13 +235,7 @@ class ExtendedSessionData(SessionData):
     def from_session_data(cls, session: SessionData) -> "ExtendedSessionData":
         """Convert a basic SessionData to ExtendedSessionData."""
         return cls(
-            session_id=session.session_id,
-            messages=session.messages,
-            created_at=session.created_at,
-            updated_at=session.updated_at,
-            agent_name=session.agent_name,
-            user_id=session.user_id,
-            metadata=session.metadata,
+            **{descriptor.name: getattr(session, descriptor.name) for descriptor in fields(SessionData)},
         )
 
 class HierarchicalSessionStore(DefaultSessionStore):
@@ -148,8 +266,21 @@ class HierarchicalSessionStore(DefaultSessionStore):
     
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        self._extended_cache: Dict[str, ExtendedSessionData] = {}
+        self._cache_fingerprints: Dict[str, Tuple[ExtendedSessionData, bytes]] = {}
+        # Bound the extended-session cache with the same LRU policy (and
+        # maxsize) as the base ``_cache`` so a long-lived process using this
+        # subclass does not retain every session it has ever loaded in RAM.
+        # Disk stays the source of truth, so an eviction just reloads on next
+        # access. When an entry is evicted its companion mtime is dropped too so
+        # ``_cache_mtimes`` stays bounded alongside the cache.
         self._cache_mtimes: Dict[str, float] = {}  # Track file modification times
+        self._extended_cache: Dict[str, ExtendedSessionData] = _LRUSessionCache(
+            getattr(self._cache, "maxsize", 0),
+            on_evict=lambda key: (
+                self._cache_mtimes.pop(key, None),
+                self._cache_fingerprints.pop(key, None),
+            ),
+        )
 
 
     def _load_session_from_disk(self, session_id: str, filepath: str) -> ExtendedSessionData:
@@ -171,9 +302,15 @@ class HierarchicalSessionStore(DefaultSessionStore):
         if not os.path.exists(filepath):
             return ExtendedSessionData(session_id=session_id)
         try:
-            with open(filepath, "r", encoding="utf-8") as f:
-                data = json.load(f)
-            return ExtendedSessionData.from_dict(data)
+            with open(filepath, "rb") as f:
+                raw = f.read()
+            data = json.loads(raw.decode("utf-8"))
+            session = ExtendedSessionData.from_dict(data)
+            # Bind the fingerprint to this parsed object and these exact bytes.
+            # A later writer must not certify an older payload via its stats.
+            with self._lock:
+                self._cache_fingerprints[session_id] = (session, hashlib.sha256(raw).digest())
+            return session
         except (json.JSONDecodeError, UnicodeDecodeError) as e:
             # Invalid UTF-8 (``UnicodeDecodeError``) is handled alongside
             # malformed JSON so a corrupt binary file is quarantined here too
@@ -194,6 +331,21 @@ class HierarchicalSessionStore(DefaultSessionStore):
                 f"refusing to overwrite existing data: {e}"
             )
             raise
+        except ValueError as e:
+            # Unsupported/damaged references must not turn into a fresh record
+            # or a legacy positional snapshot that restores unrelated history.
+            raise OSError("invalid snapshot storage; preserving session") from e
+
+    def _session_to_storage(self, session: SessionData) -> Dict[str, Any]:
+        if isinstance(session, ExtendedSessionData):
+            return session._to_storage_dict()
+        return super()._session_to_storage(session)
+
+    def _dump_session_json(self, data: Any, stream) -> None:
+        if isinstance(data, dict) and "snapshot_storage" in data:
+            json.dump(data, stream, ensure_ascii=False, separators=(",", ":"))
+        else:
+            super()._dump_session_json(data, stream)
 
     def _modify_session_locked(
         self,
@@ -201,33 +353,81 @@ class HierarchicalSessionStore(DefaultSessionStore):
         mutator,
         *,
         error_label: str = "modify session",
+        apply_retention: bool = True,
     ) -> bool:
         """Locked read-modify-write preserving extended session fields."""
-        result = super()._modify_session_locked(
-            session_id, mutator, error_label=error_label
-        )
-        if result:
-            with self._lock:
-                cached = self._cache.get(session_id)
-                if isinstance(cached, ExtendedSessionData):
-                    self._extended_cache[session_id] = cached
-        return result
-
-    def _is_cache_valid(self, session_id: str) -> bool:
-        """Check if cached session is still valid based on file mtime."""
-        if session_id not in self._extended_cache:
-            return False
-        
         filepath = self._get_session_path(session_id)
-        if not os.path.exists(filepath):
-            return False
-        
+        with FileLock(filepath, self.lock_timeout):
+            try:
+                session = self._load_session_from_disk(session_id, filepath)
+            except OSError:
+                logger.error("Failed to %s %s: could not read existing session", error_label, session_id)
+                return False
+            before = session.messages[:]
+            mutator(session)
+            self._invalidate_legacy_snapshots(session, before)
+            session.updated_at = datetime.now(timezone.utc).isoformat()
+            if apply_retention:
+                self._enforce_window(session)
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
+                logger.error("Failed to %s %s", error_label, session_id)
+                return False
+            self._cache_written_session(session, filepath)
+            return True
+
+    def _enforce_window(self, session: SessionData) -> None:
+        before = session.messages[:]
+        super()._enforce_window(session)
+        self._invalidate_legacy_snapshots(session, before)
+
+    @staticmethod
+    def _invalidate_legacy_snapshots(session: SessionData, before: List[SessionMessage]) -> None:
+        if isinstance(session, ExtendedSessionData):
+            for snapshot in session.snapshots:
+                index = snapshot.message_index
+                if snapshot.transcript is None and index >= 0 and (
+                    index >= len(session.messages)
+                    or before[:index + 1] != session.messages[:index + 1]
+                ):
+                    snapshot.invalidated = True
+
+    def _cache_written_session(self, session: ExtendedSessionData, filepath: str) -> None:
+        """Bind the written object to exact bytes while the caller holds FileLock."""
         try:
-            current_mtime = os.path.getmtime(filepath)
-            cached_mtime = self._cache_mtimes.get(session_id, 0)
-            return current_mtime <= cached_mtime
-        except (OSError, IOError):
+            with open(filepath, "rb") as f:
+                fingerprint = hashlib.sha256(f.read()).digest()
+        except OSError:
+            # The write succeeded; unavailable validation bytes only disable reuse.
+            fingerprint = None
+        with self._lock:
+            self._cache[session.session_id] = session
+            self._extended_cache[session.session_id] = session
+            if fingerprint is None:
+                self._cache_fingerprints.pop(session.session_id, None)
+            else:
+                self._cache_fingerprints[session.session_id] = (session, fingerprint)
+
+    def _is_cache_valid(self, session_id: str) -> Optional[bool]:
+        """Validate cached bytes; None means a transient validation failure."""
+        with self._lock:
+            if session_id not in self._extended_cache:
+                return False
+        filepath = self._get_session_path(session_id)
+        try:
+            with FileLock(filepath, self.lock_timeout):
+                with open(filepath, "rb") as f:
+                    fingerprint = hashlib.sha256(f.read()).digest()
+                with self._lock:
+                    cached = self._cache_fingerprints.get(session_id)
+                    return (
+                        cached is not None
+                        and cached[0] is self._extended_cache.get(session_id)
+                        and cached[1] == fingerprint
+                    )
+        except FileNotFoundError:
             return False
+        except OSError:
+            return None
     
     def _read_session_fresh(self, session_id: str) -> ExtendedSessionData:
         """Reload from disk and keep _cache and _extended_cache in sync."""
@@ -237,16 +437,8 @@ class HierarchicalSessionStore(DefaultSessionStore):
             with self._lock:
                 self._cache[session_id] = session
         
-        # Update cache with fresh file mtime
-        filepath = self._get_session_path(session_id)
-        try:
-            mtime = os.path.getmtime(filepath) if os.path.exists(filepath) else time.time()
-        except (OSError, IOError):
-            mtime = time.time()
-        
         with self._lock:
             self._extended_cache[session_id] = session
-            self._cache_mtimes[session_id] = mtime
         
         return session
     
@@ -279,31 +471,38 @@ class HierarchicalSessionStore(DefaultSessionStore):
 
         def _apply(session: SessionData) -> None:
             session.messages.append(message)
-            if len(session.messages) > self.max_messages:
-                session.messages = session.messages[-self.max_messages :]
 
         return self._modify_session_locked(
             session_id, _apply, error_label="add message to session"
         )
     
     def _load_extended_session(self, session_id: str, force_reload: bool = False) -> ExtendedSessionData:
-        """Load extended session with smart caching based on file modification time."""
+        """Reuse parsed extended data only when its source bytes are unchanged."""
         # Force reload bypasses cache validation
-        if force_reload or not self._is_cache_valid(session_id):
+        if force_reload:
+            return self._read_session_fresh(session_id)
+        valid = self._is_cache_valid(session_id)
+        if valid is False:
             return self._read_session_fresh(session_id)
         
-        # Cache is valid, return cached version
+        # Invalidation may have raced with the content check.
         with self._lock:
-            return self._extended_cache[session_id]
+            cached = self._extended_cache.get(session_id)
+        if cached is not None:
+            return cached
+        return self._read_session_fresh(session_id)
     
-    def _save_extended_session(self, session: ExtendedSessionData) -> bool:
+    def _save_extended_session(
+        self, session: ExtendedSessionData, *, apply_retention: bool = True
+    ) -> bool:
         """Save extended session to disk."""
         filepath = self._get_session_path(session.session_id)
         session.updated_at = datetime.now(timezone.utc).isoformat()
         
-        # Trim messages if over limit
-        if len(session.messages) > self.max_messages:
-            session.messages = session.messages[-self.max_messages:]
+        # Match the inherited retention policy, including non-destructive
+        # compaction and tool-exchange-aware truncation.
+        if apply_retention:
+            self._enforce_window(session)
         
         with FileLock(filepath, self.lock_timeout):
             try:
@@ -317,20 +516,12 @@ class HierarchicalSessionStore(DefaultSessionStore):
                     delete=False,
                     suffix=".tmp"
                 ) as f:
-                    json.dump(session.to_dict(), f, indent=2, ensure_ascii=False)
+                    self._dump_session_json(self._session_to_storage(session), f)
                     temp_path = f.name
                 
                 os.replace(temp_path, filepath)
                 
-                # Update cache with current file mtime after successful write
-                try:
-                    mtime = os.path.getmtime(filepath)
-                except (OSError, IOError):
-                    mtime = time.time()
-                
-                with self._lock:
-                    self._extended_cache[session.session_id] = session
-                    self._cache_mtimes[session.session_id] = mtime
+                self._cache_written_session(session, filepath)
                 
                 return True
             except (IOError, OSError) as e:
@@ -362,8 +553,15 @@ class HierarchicalSessionStore(DefaultSessionStore):
             
         Returns:
             The session ID
+
+        Raises:
+            ValueError: The session would be its own parent.
+            OSError: Saving the session or registering its parent failed.
+                A saved child is retained if parent registration fails.
         """
         sid = session_id or str(uuid.uuid4())
+        if parent_id == sid:
+            raise ValueError(f"Session {sid} cannot be its own parent")
         
         session = ExtendedSessionData(
             session_id=sid,
@@ -373,15 +571,18 @@ class HierarchicalSessionStore(DefaultSessionStore):
             metadata=metadata or {},
         )
         
-        # Update parent's children list without clobbering concurrent message writes
+        if not self._save_extended_session(session):
+            raise OSError(f"Failed to save session {sid}")
+
+        # Register only after the child exists on disk.
         if parent_id:
             def _apply(parent_session: SessionData) -> None:
                 assert isinstance(parent_session, ExtendedSessionData)
                 if sid not in parent_session.children_ids:
                     parent_session.children_ids.append(sid)
-            self._modify_session_locked(parent_id, _apply, error_label="update parent children")
+            if not self._modify_session_locked(parent_id, _apply, error_label="update parent children"):
+                raise OSError(f"Session {sid} was saved but registration with parent {parent_id} failed")
         
-        self._save_extended_session(session)
         return sid
     
     def fork_session(
@@ -395,14 +596,25 @@ class HierarchicalSessionStore(DefaultSessionStore):
         
         Args:
             session_id: The session to fork from
-            from_message_index: Message index to fork from (None = all messages)
+            from_message_index: Zero-based, inclusive message index to fork from
+                (None = all messages, including an empty session).
             title: Optional title for the forked session
             
         Returns:
             The new forked session ID
+
+        Raises:
+            ValueError: The explicit message index is outside the parent history.
+            OSError: Saving the fork or registering it with the parent failed.
+                A saved fork is retained if parent registration fails.
         """
         # Force reload to get latest messages from disk
         parent = self._load_extended_session(session_id, force_reload=True)
+
+        if from_message_index is not None and (
+            from_message_index < 0 or from_message_index >= len(parent.messages)
+        ):
+            raise ValueError(f"Invalid message index {from_message_index}")
         
         # Determine which messages to copy
         if from_message_index is None:
@@ -426,15 +638,17 @@ class HierarchicalSessionStore(DefaultSessionStore):
             metadata=copy.deepcopy(parent.metadata),
         )
         
-        self._save_extended_session(forked)
+        if not self._save_extended_session(forked):
+            raise OSError(f"Failed to save forked session {new_id}")
 
         def _register_fork(parent: SessionData) -> None:
             if new_id not in parent.children_ids:
                 parent.children_ids.append(new_id)
 
-        self._modify_session_locked(
+        if not self._modify_session_locked(
             session_id, _register_fork, error_label="register forked session"
-        )
+        ):
+            raise OSError(f"Forked session {new_id} was saved but registration with parent {session_id} failed")
         
         return new_id
     
@@ -476,6 +690,9 @@ class HierarchicalSessionStore(DefaultSessionStore):
     ) -> str:
         """
         Create a snapshot of the current session state.
+
+        Persist the active messages, archive and compaction checkpoint so
+        later retention cannot move the snapshot's restore point.
         
         Args:
             session_id: The session to snapshot
@@ -484,6 +701,9 @@ class HierarchicalSessionStore(DefaultSessionStore):
             
         Returns:
             The snapshot ID
+
+        Raises:
+            OSError: The snapshot could not be persisted.
         """
         snapshot = SessionSnapshot(
             session_id=session_id,
@@ -493,14 +713,30 @@ class HierarchicalSessionStore(DefaultSessionStore):
         )
 
         def _record_snapshot(session: SessionData) -> None:
-            snapshot.message_index = (
-                len(session.messages) - 1 if session.messages else -1
+            # The caller already holds the session write lock. Read the base
+            # transcript schema so existing durable checkpoint fields are not
+            # lost through the hierarchy's partial legacy field conversion.
+            captured = DefaultSessionStore._load_session_from_disk(
+                self, session_id, self._get_session_path(session_id)
             )
+            snapshot.message_index = (
+                len(captured.messages) - 1 if captured.messages else -1
+            )
+            snapshot.transcript = copy.deepcopy({
+                "messages": [message.to_dict() for message in captured.messages],
+                "archived_messages": [message.to_dict() for message in captured.archived_messages],
+                "last_compaction": captured.last_compaction.to_dict() if captured.last_compaction else None,
+            })
+            session.last_compaction = captured.last_compaction
             session.snapshots.append(snapshot)
 
-        self._modify_session_locked(
-            session_id, _record_snapshot, error_label="create snapshot"
-        )
+        # Recording a snapshot must not compact or truncate an imported live
+        # transcript, so skip retention on this write.
+        if not self._modify_session_locked(
+            session_id, _record_snapshot, error_label="create snapshot",
+            apply_retention=False,
+        ):
+            raise OSError(f"Failed to save snapshot {snapshot.id} for session {session_id}")
         
         return snapshot.id
     
@@ -518,7 +754,7 @@ class HierarchicalSessionStore(DefaultSessionStore):
             snapshot_id: The snapshot to revert to
             
         Returns:
-            True if successful
+            True if successful; False for missing or invalidated legacy snapshots.
         """
         def _apply(session: SessionData) -> None:
             assert isinstance(session, ExtendedSessionData)
@@ -534,14 +770,25 @@ class HierarchicalSessionStore(DefaultSessionStore):
                 logger.warning(f"Snapshot {snapshot_id} not found")
                 raise ValueError(f"Snapshot {snapshot_id} not found")
             
-            # Revert messages
-            if snapshot.message_index >= 0:
+            if snapshot.invalidated:
+                raise ValueError("Legacy snapshot position was invalidated by retention")
+            if snapshot.transcript is not None:
+                restored = SessionData.from_dict({"session_id": session_id, **snapshot.transcript})
+                session.messages = restored.messages
+                session.archived_messages = restored.archived_messages
+                session.last_compaction = restored.last_compaction
+            elif snapshot.message_index >= len(session.messages):
+                raise ValueError("Legacy snapshot position is outside the retained history")
+            elif snapshot.message_index >= 0:
                 session.messages = session.messages[:snapshot.message_index + 1]
             else:
                 session.messages = []
                 
         try:
-            return self._modify_session_locked(session_id, _apply, error_label="revert to snapshot")
+            return self._modify_session_locked(
+                session_id, _apply, error_label="revert to snapshot",
+                apply_retention=False,
+            )
         except ValueError:
             return False
     
@@ -673,12 +920,9 @@ class HierarchicalSessionStore(DefaultSessionStore):
     
     def get_extended_session(self, session_id: str,
                              force_reload: bool = False) -> ExtendedSessionData:
-        """Get extended session data, using the mtime cache when it is valid.
+        """Get extended data, reusing parsed data when the JSON bytes match.
 
-        This called _read_session_fresh unconditionally, so every read went to
-        disk and re-parsed the file: _is_cache_valid, _cache_mtimes and
-        _load_extended_session were all built and never reached from the public
-        API. Pass force_reload=True to re-read regardless.
+        Pass force_reload=True to reload regardless of content validation.
         """
         return self._load_extended_session(session_id, force_reload=force_reload)
 
@@ -688,9 +932,13 @@ class HierarchicalSessionStore(DefaultSessionStore):
             if session_id:
                 self._cache.pop(session_id, None)
                 self._extended_cache.pop(session_id, None)
+                self._cache_fingerprints.pop(session_id, None)
+                self._cache_mtimes.pop(session_id, None)
             else:
                 self._cache.clear()
                 self._extended_cache.clear()
+                self._cache_fingerprints.clear()
+                self._cache_mtimes.clear()
     
     def export_session(self, session_id: str) -> Dict[str, Any]:
         """
@@ -716,6 +964,9 @@ class HierarchicalSessionStore(DefaultSessionStore):
             
         Returns:
             The imported session ID
+
+        Raises:
+            OSError: The imported session could not be persisted.
         """
         session = ExtendedSessionData.from_dict(data)
         
@@ -729,7 +980,10 @@ class HierarchicalSessionStore(DefaultSessionStore):
         session.children_ids = []
         session.forked_from_message_id = None
         
-        self._save_extended_session(session)
+        # Restore the exported record in full, independent of the destination
+        # window. Retention applies to subsequent ordinary writes.
+        if not self._save_extended_session(session, apply_retention=False):
+            raise OSError(f"Failed to save imported session {session.session_id}")
         return session.session_id
 
 # Global hierarchical store instance

src/praisonai-agents/praisonaiagents/session/sqlite_store.py

@@ -30,11 +30,12 @@ Usage::
 
 import os
 import threading
+from contextlib import contextmanager
 from typing import Any, Dict, List, Optional
 
 from praisonaiagents._logging import get_logger
 
-from .store import DefaultSessionStore, SessionData
+from .store import DefaultSessionStore, FileLock, SessionData
 
 logger = get_logger(__name__)
 
@@ -74,10 +75,13 @@ class SqliteSessionStore(DefaultSessionStore):
             db_path = os.path.expanduser(db_path)
         self.db_path = db_path
         self._db_lock = threading.RLock()
+        self._backfill_lock = threading.RLock()
+        self._backfill_running = False
         self._conn = None
         self._fts_available = False
         self._db_ready = False
         self._backfilled = False
+        self._backfill_upgraded_files = {}
 
     # ── index lifecycle ───────────────────────────────────────────────
 
@@ -98,9 +102,14 @@ class SqliteSessionStore(DefaultSessionStore):
             try:
                 if self.db_path != ":memory:":
                     os.makedirs(os.path.dirname(self.db_path) or ".", exist_ok=True)
-                conn = sqlite3.connect(
-                    self.db_path, check_same_thread=False, isolation_level=None
-                )
+                # Hardened factory: WAL where the filesystem supports it, safe
+                # DELETE fallback on NFS/SMB/FUSE/virtiofs (Issue #5264).
+                from ..storage.sqlite import connect as _sqlite_connect
+
+                # guard=True: quick_check on open + forensic backup + bounded
+                # least-destructive repair so a malformed index file recovers
+                # instead of taking the durable path down (Issue #5387).
+                conn = _sqlite_connect(self.db_path, isolation_level=None, guard=True)
                 self._fts_available = self._init_schema(conn)
                 self._conn = conn
             except Exception as exc:
@@ -225,42 +234,60 @@ class SqliteSessionStore(DefaultSessionStore):
                 parts.append(str(content))
         return "\n".join(parts)
 
-    def _index_session(self, session: SessionData) -> None:
-        """Insert/replace a session's content in the index (best-effort)."""
+    def _index_session(self, session: SessionData) -> bool:
+        """Refresh the index best-effort, reporting whether it succeeded."""
         conn = self._connect()
         if conn is None:
-            return
+            return False
         content = self._flatten(session)
         sid = session.session_id
         gateway_session_id = getattr(session, "gateway_session_id", None)
         agent_id = getattr(session, "agent_id", None)
         try:
             with self._db_lock:
-                conn.execute("DELETE FROM session_fts WHERE session_id = ?", (sid,))
-                conn.execute(
-                    "INSERT INTO session_fts (session_id, content) VALUES (?, ?)",
-                    (sid, content),
-                )
-                conn.execute(
-                    "INSERT OR REPLACE INTO session_meta (session_id, updated_at) "
-                    "VALUES (?, ?)",
-                    (sid, session.updated_at),
-                )
-                # Keep the gateway/agent routing index in sync so inbound
-                # routing is an indexed lookup, not a full-directory scan.
-                if gateway_session_id or agent_id:
+                # A separate WAL reader must see the old or new index record,
+                # never the autocommitted gap between DELETE and INSERT.
+                owns_transaction = not conn.in_transaction
+                conn.execute("SAVEPOINT praisonai_index_refresh")
+                try:
+                    conn.execute("DELETE FROM session_fts WHERE session_id = ?", (sid,))
                     conn.execute(
-                        "INSERT OR REPLACE INTO session_route "
-                        "(session_id, gateway_session_id, agent_id) "
-                        "VALUES (?, ?, ?)",
-                        (sid, gateway_session_id, agent_id),
+                        "INSERT INTO session_fts (session_id, content) VALUES (?, ?)",
+                        (sid, content),
                     )
-                else:
                     conn.execute(
-                        "DELETE FROM session_route WHERE session_id = ?", (sid,)
+                        "INSERT OR REPLACE INTO session_meta (session_id, updated_at) "
+                        "VALUES (?, ?)",
+                        (sid, session.updated_at),
                     )
+                    if gateway_session_id or agent_id:
+                        conn.execute(
+                            "INSERT OR REPLACE INTO session_route "
+                            "(session_id, gateway_session_id, agent_id) "
+                            "VALUES (?, ?, ?)",
+                            (sid, gateway_session_id, agent_id),
+                        )
+                    else:
+                        conn.execute("DELETE FROM session_route WHERE session_id = ?", (sid,))
+                    conn.execute(
+                        "DELETE FROM session_index_meta WHERE key = ?",
+                        ("import_pending:" + sid,),
+                    )
+                    conn.execute("RELEASE praisonai_index_refresh")
+                except BaseException:
+                    if conn.in_transaction:
+                        if owns_transaction:
+                            # A failed outermost RELEASE leaves the transaction
+                            # active, including its uncommitted writes and locks.
+                            conn.execute("ROLLBACK")
+                        else:
+                            conn.execute("ROLLBACK TO praisonai_index_refresh")
+                            conn.execute("RELEASE praisonai_index_refresh")
+                    raise
         except Exception as exc:  # never let indexing break a write
             logger.debug("Session index update failed for %s: %s", sid, exc)
+            return False
+        return True
 
     def _deindex_session(self, session_id: str) -> None:
         conn = self._connect()
@@ -275,22 +302,60 @@ class SqliteSessionStore(DefaultSessionStore):
             logger.debug("Session de-index failed for %s: %s", session_id, exc)
 
     def _ensure_backfilled(self) -> None:
-        """Backfill the index from existing JSON transcripts exactly once.
+        """Backfill existing transcripts, retrying an incomplete pass.
 
-        Guarded by a one-time flag rather than by an empty-index check: a
+        Guarded by a successful-pass flag rather than an empty-index check: a
         single ``add_message`` on a *new* session could otherwise make the
         index non-empty and permanently skip backfilling pre-existing JSON
         transcripts, silently omitting legacy sessions from search results.
         Only sessions not already present in the index are (re)indexed, so the
         pass is cheap on a warm index.
         """
-        if self._backfilled:
-            return
-        with self._db_lock:
-            if self._backfilled:
+        # Do not hold the database lock while waiting for transcript locks:
+        # imports acquire their file lock before refreshing SQLite.
+        with self._backfill_lock:
+            if self._backfill_running:
                 return
-            self._backfilled = True
-            self._reindex_all()
+            # A corruption callback can query this store during the pass.
+            # It sees the index built so far; other threads wait for completion.
+            self._backfill_running = True
+            try:
+                self._retry_import_indexes()
+                if not self._backfilled:
+                    self._backfilled = self._reindex_all()
+            finally:
+                self._backfill_running = False
+
+    def _retry_import_indexes(self) -> None:
+        """Retry only failed restores, including after reopening the store."""
+        conn = self._connect()
+        if conn is None:
+            return
+        try:
+            with self._db_lock:
+                keys = conn.execute(
+                    "SELECT key FROM session_index_meta WHERE key GLOB 'import_pending:*'"
+                ).fetchall()
+            for (key,) in keys:
+                sid = key[len("import_pending:"):]
+                filepath = self._get_session_path(sid)
+                try:
+                    # Match the writer's file -> database order. File identity
+                    # may be unavailable; the lock still protects this read.
+                    with FileLock(filepath, self.lock_timeout):
+                        if not os.path.isfile(filepath):
+                            continue
+                        session = self._load_session_from_disk(sid, filepath)
+                        # The loader may quarantine a corrupt transcript and
+                        # return an empty placeholder. That is not recovery.
+                        if not os.path.isfile(filepath):
+                            continue
+                        self._reingest_spill(sid, session)
+                        self._index_session(session)
+                except Exception as exc:
+                    logger.debug("Import index retry failed for %s: %s", sid, exc)
+        except Exception as exc:
+            logger.debug("Cannot enumerate pending import indexes: %s", exc)
 
     def _indexed_ids(self) -> set:
         """Return the set of session_ids already fully indexed.
@@ -302,8 +367,7 @@ class SqliteSessionStore(DefaultSessionStore):
         otherwise have its already-content-indexed sessions skipped by backfill,
         leaving ``session_route`` empty and breaking gateway routing. Requiring a
         route row forces those sessions to be re-indexed so their route rows are
-        populated. Backfill runs at most once (guarded by ``_backfilled``), so
-        this stays a cheap one-time pass.
+        populated. Successful backfill runs once; incomplete passes can retry.
         """
         conn = self._connect()
         if conn is None:
@@ -322,7 +386,7 @@ class SqliteSessionStore(DefaultSessionStore):
         except Exception:
             return set()
 
-    def _reindex_all(self) -> None:
+    def _reindex_all(self) -> bool:
         """Backfill the index from existing JSON transcripts.
 
         Normally cheap: sessions already fully indexed are skipped. When the
@@ -331,31 +395,74 @@ class SqliteSessionStore(DefaultSessionStore):
         indexed, Issue #5031), every session's FTS row is rebuilt once so
         archived-only queries work immediately after upgrade rather than only
         after each session's next write. The new version is then persisted so
-        subsequent startups fall back to the cheap skip behaviour.
+        subsequent startups fall back to the cheap skip behaviour. Completed
+        upgrades are retained in memory across failed passes; retries skip
+        those sessions only while their complete index projection and locked
+        transcript generation still match.
         """
         try:
             filenames = os.listdir(self.session_dir)
         except (IOError, OSError):
-            return
+            return False
         conn = self._connect()
-        stale = (
-            conn is not None
-            and self._index_content_version(conn) != self.INDEX_CONTENT_VERSION
-        )
-        already = set() if stale else self._indexed_ids()
+        if conn is None:
+            # No index is available; readers already use the parent JSON scan.
+            return True
+        with self._db_lock:
+            stale = self._index_content_version(conn) != self.INDEX_CONTENT_VERSION
+            try:
+                pending = {
+                    row[0][len("import_pending:"):]
+                    for row in conn.execute(
+                        "SELECT key FROM session_index_meta WHERE key GLOB 'import_pending:*'"
+                    ).fetchall()
+                }
+            except Exception as exc:
+                logger.debug("Cannot backfill pending import indexes: %s", exc)
+                return False
+        indexed = self._indexed_ids()
+        # A completed row alone cannot prove that a JSON-only peer has not
+        # replaced its transcript between upgrade attempts.
+        already = indexed if not stale else set()
+        complete = True
         for filename in filenames:
             if not filename.endswith(".json"):
                 continue
             sid = filename[:-5]
+            if sid in pending:
+                # Failed restore retries must not be bypassed by a synthetic
+                # empty transcript from the initial/upgrade backfill path.
+                complete = False
+                continue
             if sid in already:
                 continue
             try:
-                session = self._read_session_fresh(sid)
+                filepath = self._get_session_path(sid)
+                with FileLock(filepath, self.lock_timeout):
+                    identity = self._session_file_identity(filepath)
+                    if (
+                        stale and sid in indexed and identity is not None
+                        and self._backfill_upgraded_files.get(sid) == identity
+                    ):
+                        continue
+                    # Match import's file -> database order and keep a peer
+                    # replacement from making the just-read projection stale.
+                    session = self._load_session_from_disk(sid, filepath)
+                    self._reingest_spill(sid, session)
+                    with self._lock:
+                        self._cache[sid] = session
+                    if self._index_session(session):
+                        if stale:
+                            self._backfill_upgraded_files[sid] = identity
+                    else:
+                        complete = False
             except Exception:
-                continue
-            self._index_session(session)
-        if conn is not None:
-            self._set_index_content_version(conn, self.INDEX_CONTENT_VERSION)
+                complete = False
+        if complete:
+            with self._db_lock:
+                self._set_index_content_version(conn, self.INDEX_CONTENT_VERSION)
+            self._backfill_upgraded_files.clear()
+        return complete
 
     # ── write path: keep the index in sync ────────────────────────────
 
@@ -365,6 +472,106 @@ class SqliteSessionStore(DefaultSessionStore):
             self._index_session(session)
         return ok
 
+    def _save_imported_session(self, session: SessionData, *, overwrite: bool = True) -> bool:
+        """Refresh recall and routing after a verbatim portable restore."""
+        ok = super()._save_imported_session(session, overwrite=overwrite)
+        if ok:
+            try:
+                filepath = self._get_session_path(session.session_id)
+                with FileLock(filepath, self.lock_timeout):
+                    # Read-only cache fallback is unsafe after an overwrite:
+                    # it can restore routing fields removed by the import.
+                    fresh = self._load_session_from_disk(session.session_id, filepath)
+                    self._reingest_spill(session.session_id, fresh)
+                    # Cleanup must describe the generation actually refreshed,
+                    # including a peer write that followed the durable import.
+                    session._import_file_identity = self._session_file_identity(filepath)
+                    session._import_index_session = fresh
+                    with self._lock:
+                        self._cache[session.session_id] = fresh
+                    if not self._index_session(fresh):
+                        raise RuntimeError("post-import index update failed")
+            except Exception as exc:
+                # The JSON write succeeded, but the old index is no longer a
+                # trustworthy view. Fail closed until a later refresh/rebuild.
+                self._invalidate_import_index(session)
+                logger.debug("Post-import index refresh failed for %s: %s", session.session_id, exc)
+        return ok
+
+    def _invalidate_import_index(self, session: SessionData) -> None:
+        """Clear a failed import index unless it matches the current transcript."""
+        identity = getattr(session, "_import_file_identity", None)
+        conn = self._connect()
+        if conn is None:
+            return
+        try:
+            with self._db_lock:
+                # Persist recovery separately from cleanup: unreadable or
+                # changing generations must remain retryable after restart.
+                conn.execute(
+                    "INSERT OR REPLACE INTO session_index_meta (key, value) VALUES (?, '1')",
+                    ("import_pending:" + session.session_id,),
+                )
+                # Serialize with other SQLite writers before checking the file.
+                # A peer saving afterward must index after this transaction;
+                # a peer already indexed has a different atomic file identity.
+                owns_transaction = not conn.in_transaction
+                conn.execute("SAVEPOINT praisonai_import_cleanup")
+                try:
+                    # SAVEPOINT alone is deferred. Acquire the SQLite writer
+                    # reservation before observing the transcript generation.
+                    conn.execute(
+                        "UPDATE session_index_meta SET value = value WHERE key = ?",
+                        ("import_pending:" + session.session_id,),
+                    )
+                    filepath = self._get_session_path(session.session_id)
+                    current_identity = self._session_file_identity(filepath)
+                    expected = None
+                    if identity is not None and current_identity == identity:
+                        expected = getattr(session, "_import_index_session", session)
+                    elif current_identity is not None:
+                        # A JSON-only peer may replace the file without updating
+                        # SQLite. Read without FileLock while holding the DB
+                        # transaction; never quarantine or mutate that file.
+                        import json
+                        with open(filepath, "r", encoding="utf-8") as handle:
+                            data = json.load(handle)
+                        if self._session_file_identity(filepath) == current_identity:
+                            expected = SessionData.from_dict(data)
+                    if expected is not None:
+                        sid = session.session_id
+                        content = conn.execute("SELECT content FROM session_fts WHERE session_id = ?", (sid,)).fetchall()
+                        metadata = conn.execute("SELECT updated_at FROM session_meta WHERE session_id = ?", (sid,)).fetchall()
+                        routes = conn.execute("SELECT gateway_session_id, agent_id FROM session_route WHERE session_id = ?", (sid,)).fetchall()
+                        expected_routes = (
+                            [(expected.gateway_session_id, expected.agent_id)]
+                            if expected.gateway_session_id or expected.agent_id else []
+                        )
+                        # A peer may have successfully indexed this same file
+                        # generation after our failed update. Preserve a complete
+                        # matching projection; file identity alone cannot tell
+                        # whether those index rows are already current.
+                        current = (
+                            content == [(self._flatten(expected),)]
+                            and metadata == [(expected.updated_at,)]
+                            and routes == expected_routes
+                        )
+                        if not current:
+                            conn.execute("DELETE FROM session_fts WHERE session_id = ?", (sid,))
+                            conn.execute("DELETE FROM session_meta WHERE session_id = ?", (sid,))
+                            conn.execute("DELETE FROM session_route WHERE session_id = ?", (sid,))
+                    conn.execute("RELEASE praisonai_import_cleanup")
+                except BaseException:
+                    if conn.in_transaction:
+                        if owns_transaction:
+                            conn.execute("ROLLBACK")
+                        else:
+                            conn.execute("ROLLBACK TO praisonai_import_cleanup")
+                            conn.execute("RELEASE praisonai_import_cleanup")
+                    raise
+        except Exception as exc:
+            logger.debug("Post-import index invalidation failed for %s: %s", session.session_id, exc)
+
     def _modify_session_locked(self, session_id, mutator, **kwargs) -> bool:
         """Refresh the index after any locked read-modify-write.
 
@@ -406,12 +613,6 @@ class SqliteSessionStore(DefaultSessionStore):
                 logger.debug("Post-add index refresh failed for %s: %s", session_id, exc)
         return ok
 
-    def clear_session(self, session_id: str) -> bool:
-        ok = super().clear_session(session_id)
-        if ok:
-            self._deindex_session(session_id)
-        return ok
-
     def delete_session(self, session_id: str) -> bool:
         ok = super().delete_session(session_id)
         if ok:
@@ -438,14 +639,14 @@ class SqliteSessionStore(DefaultSessionStore):
                     match = self._to_fts_query(query)
                     rows = conn.execute(
                         "SELECT session_id FROM session_fts WHERE session_fts "
-                        "MATCH ? ORDER BY bm25(session_fts) LIMIT ?",
+                        "MATCH ? ORDER BY bm25(session_fts), session_id LIMIT ?",
                         (match, fetch),
                     ).fetchall()
                 else:
                     like = "%" + query.replace("%", "").replace("_", "") + "%"
                     rows = conn.execute(
                         "SELECT session_id FROM session_fts WHERE content LIKE ? "
-                        "LIMIT ?",
+                        "ORDER BY session_id LIMIT ?",
                         (like, fetch),
                     ).fetchall()
         except Exception as exc:
@@ -463,14 +664,127 @@ class SqliteSessionStore(DefaultSessionStore):
             return '""'
         return " OR ".join('"%s"' % t for t in terms)
 
+    @contextmanager
+    def _search_id_connection(self, conn):
+        """Keep WAL candidate enumeration off the writer connection."""
+        import sqlite3
+        from pathlib import Path
+
+        with self._db_lock:
+            journal = conn.execute("PRAGMA journal_mode").fetchone()[0]
+        if self.db_path != ":memory:" and journal.lower() == "wal":
+            reader = sqlite3.connect(
+                Path(self.db_path).resolve().as_uri() + "?mode=ro",
+                uri=True, isolation_level=None,
+            )
+            try:
+                yield reader
+            finally:
+                reader.close()
+        else:
+            # Non-WAL stores retain the shared lock only for ID enumeration,
+            # never for transcript reads or corruption callbacks.
+            with self._db_lock:
+                yield conn
+
+    @contextmanager
+    def _search_candidate_ids(self, conn, query: str):
+        """Spool matching IDs in fixed batches; close SQL before file I/O.
+
+        A stable candidate set requires enumerating the matches. Spooling only
+        IDs bounds Python ID memory without copying the index or keeping a
+        SQLite read snapshot across transcript reads and corruption callbacks.
+        """
+        import json
+        import tempfile
+
+        with tempfile.TemporaryFile(mode="w+t", encoding="utf-8") as spool:
+            with self._search_id_connection(conn) as reader:
+                if self._fts_available:
+                    cursor = reader.execute(
+                        "SELECT session_id FROM session_fts WHERE session_fts "
+                        "MATCH ? ORDER BY bm25(session_fts), session_id",
+                        (self._to_fts_query(query),),
+                    )
+                else:
+                    like = "%" + query.replace("%", "").replace("_", "") + "%"
+                    cursor = reader.execute(
+                        "SELECT session_id FROM session_fts WHERE content LIKE ? "
+                        "ORDER BY session_id", (like,),
+                    )
+                try:
+                    while True:
+                        rows = cursor.fetchmany(128)
+                        if not rows:
+                            break
+                        for row in rows:
+                            spool.write(json.dumps(row[0], ensure_ascii=True) + "\n")
+                finally:
+                    cursor.close()
+            spool.seek(0)
+            yield (json.loads(line) for line in spool)
+
+    def _read_search_candidates(self, query: str, limit: int, consume=None):
+        """Read usable lineages outside the SQL lock and read snapshot.
+
+        A consumer scores each payload immediately instead of retaining every
+        continuation. Without a consumer, preserve the private list interface.
+        """
+        import json
+
+        conn = self._connect()
+        if conn is None:
+            return None
+        self._ensure_backfilled()
+        allowance = max(limit * 5, limit, 1)
+        candidates = []
+        lineages = set()
+        try:
+            with self._search_candidate_ids(conn, query) as ids:
+                for sid in ids:
+                    filepath = self._get_session_path(sid)
+                    try:
+                        with open(filepath, "r", encoding="utf-8") as handle:
+                            data = json.load(handle)
+                    except (UnicodeDecodeError, json.JSONDecodeError, OSError) as exc:
+                        self._report_unreadable_session(sid, filepath, exc)
+                        continue
+                    if consume is None:
+                        candidates.append((sid, data))
+                    else:
+                        consume(sid, data)
+                    lineage = self._lineage_key(data)
+                    lineages.add(("lineage", lineage) if lineage is not None else ("session", sid))
+                    if len(lineages) >= allowance:
+                        break
+        except Exception as exc:
+            logger.debug("Index query failed (%s); falling back to scan.", exc)
+            return None
+        return candidates
+
     # ── gateway/agent routing: indexed key → session lookup ───────────
 
+    def _read_indexed_route(self, session_id: str) -> Optional[SessionData]:
+        """Verify a route from durable JSON without a stale cache fallback."""
+        filepath = self._get_session_path(session_id)
+        try:
+            with FileLock(filepath, self.lock_timeout):
+                session = self._load_session_from_disk(session_id, filepath)
+                self._reingest_spill(session_id, session)
+            with self._lock:
+                self._cache[session_id] = session
+            return session
+        except (OSError, ValueError, TypeError) as exc:
+            logger.debug("Cannot verify indexed route for %s: %s", session_id, exc)
+            return None
+
     def get_by_gateway_session(self, gateway_session_id: str) -> Optional[SessionData]:
         """Resolve a session by ``gateway_session_id`` via the indexed route.
 
         Overrides the parent's O(N) ``os.listdir`` + JSON-parse-every-file scan
         with a single indexed ``SELECT`` on ``session_route`` so inbound routing
         latency is independent of the number of stored sessions (Issue #2956).
+        The loaded binding must still match; transient reads do not use cache.
         Falls back to the parent scan if the index is unavailable.
         """
         conn = self._connect()
@@ -489,33 +803,52 @@ class SqliteSessionStore(DefaultSessionStore):
             return super().get_by_gateway_session(gateway_session_id)
         if row is None:
             return None
-        try:
-            return self._read_session_fresh(row[0])
-        except Exception:
+        session = self._read_indexed_route(row[0])
+        if session is None or session.gateway_session_id != gateway_session_id:
             return None
+        return session
 
     def list_sessions_by_gateway_agent(self, agent_id: str, limit: int = 50) -> List[str]:
         """List session IDs for a gateway agent via the indexed route.
 
         Overrides the parent's full-directory scan with an indexed ``SELECT``
-        on ``session_route`` (Issue #2956). Falls back to the parent scan if the
-        index is unavailable.
+        on ``session_route`` (Issue #2956). Matching candidates are verified
+        against JSON outside the database lock, and only verified IDs count
+        toward the limit. Falls back to the parent scan if the index is unavailable.
         """
         conn = self._connect()
         if conn is None:
             return super().list_sessions_by_gateway_agent(agent_id, limit)
         self._ensure_backfilled()
+        if limit == 0:
+            return []
+        verified = []
+        offset = 0
         try:
-            with self._db_lock:
-                rows = conn.execute(
-                    "SELECT session_id FROM session_route WHERE agent_id = ? "
-                    "LIMIT ?",
-                    (agent_id, limit),
-                ).fetchall()
+            while True:
+                with self._db_lock:
+                    rows = conn.execute(
+                        "SELECT session_id FROM session_route WHERE agent_id = ? "
+                        "LIMIT ? OFFSET ?",
+                        (agent_id, limit, offset),
+                    ).fetchall()
+                if not rows:
+                    break
+                # File I/O stays outside the database lock. Unknown generations
+                # retain their index rows, but cannot supply obsolete bindings.
+                for (sid,) in rows:
+                    session = self._read_indexed_route(sid)
+                    if session is not None and session.agent_id == agent_id:
+                        verified.append(sid)
+                        if limit > 0 and len(verified) >= limit:
+                            return verified
+                if limit < 0:
+                    break
+                offset += len(rows)
         except Exception as exc:
             logger.debug("Route lookup failed (%s); falling back to scan.", exc)
             return super().list_sessions_by_gateway_agent(agent_id, limit)
-        return [r[0] for r in rows]
+        return verified
 
     def search(
         self,
@@ -538,29 +871,19 @@ class SqliteSessionStore(DefaultSessionStore):
         if not query:
             return []
 
-        candidate_ids = self._candidate_ids(query, limit)
-        if candidate_ids is None:
-            return super().search(query, limit=limit, window=window)
-        if not candidate_ids:
-            return []
-
         needle = query.lower()
         terms = [t for t in needle.split() if t]
-        hits: List[tuple] = []
-
-        for sid in candidate_ids:
-            filepath = self._get_session_path(sid)
-            try:
-                import json
-
-                with open(filepath, "r", encoding="utf-8") as f:
-                    data = json.load(f)
-            except (json.JSONDecodeError, IOError, OSError):
-                continue
+        hits: Dict[Any, tuple] = {}
+        position = 0
 
+        def consume(sid, data):
+            """Retain each lineage's best hit and its original tie position."""
+            nonlocal position
+            ordinal = position
+            position += 1
             messages = self._searchable_messages(data)
             if not messages:
-                continue
+                return
 
             best_index = -1
             best_score = 0.0
@@ -582,7 +905,7 @@ class SqliteSessionStore(DefaultSessionStore):
                     best_index = idx
 
             if best_index < 0:
-                continue
+                return
 
             start = max(0, best_index - window)
             end = min(len(messages), best_index + window + 1)
@@ -613,13 +936,26 @@ class SqliteSessionStore(DefaultSessionStore):
                 messages=context,
                 bookends=self._bookends(messages, self.BOOKEND_SIZE),
             )
-            hits.append((self._lineage_key(data), hit))
+            lineage = self._lineage_key(data)
+            key = ("lineage", lineage) if lineage is not None else ("session", sid)
+            previous = hits.get(key)
+            if previous is None or (hit.score, hit.when or "") > (
+                previous[1].score, previous[1].when or ""
+            ):
+                hits[key] = (lineage, hit, ordinal)
 
-        hits.sort(key=lambda item: (item[1].score, item[1].when or ""), reverse=True)
+        candidates = self._read_search_candidates(query, limit, consume=consume)
+        if candidates is None:
+            return super().search(query, limit=limit, window=window)
+        ranked = sorted(
+            hits.values(),
+            key=lambda item: (item[1].score, item[1].when or "", -item[2]),
+            reverse=True,
+        )
 
         deduped: List[Any] = []
         seen_lineage: set = set()
-        for lineage, hit in hits:
+        for lineage, hit, _ in ranked:
             if lineage is not None:
                 if lineage in seen_lineage:
                     continue

src/praisonai-agents/praisonaiagents/session/sqlite_transcript_store.py

@@ -92,20 +92,24 @@ class SqliteTranscriptStore(DefaultSessionStore):
         with self._db_lock:
             if self._db_ready:
                 return self._conn
-            import sqlite3  # lazy import — stdlib, no heavy dependency
+            import sqlite3 as _sqlite3  # noqa: F401 - availability guard
 
             if self.db_path != ":memory:":
                 os.makedirs(os.path.dirname(self.db_path) or ".", exist_ok=True)
-            conn = sqlite3.connect(
-                self.db_path, check_same_thread=False, isolation_level=None
+            # Hardened factory: WAL where the filesystem supports it, safe
+            # DELETE fallback on NFS/SMB/FUSE/virtiofs (Issue #5264). WAL lets
+            # readers proceed concurrently with a writer where available.
+            from ..storage.sqlite import connect as _sqlite_connect
+
+            # guard=True: quick_check on open + forensic backup + bounded
+            # least-destructive repair so a malformed transcript DB recovers
+            # instead of taking the durable path down (Issue #5387).
+            conn = _sqlite_connect(
+                self.db_path,
+                isolation_level=None,
+                busy_timeout_ms=int(self.lock_timeout * 1000),
+                guard=True,
             )
-            try:
-                # WAL lets readers proceed concurrently with a writer.
-                conn.execute("PRAGMA journal_mode=WAL")
-                conn.execute("PRAGMA synchronous=NORMAL")
-                conn.execute("PRAGMA busy_timeout=%d" % int(self.lock_timeout * 1000))
-            except Exception:  # pragma: no cover - PRAGMA best-effort
-                pass
             conn.execute(
                 "CREATE TABLE IF NOT EXISTS sessions ("
                 "  session_id TEXT PRIMARY KEY,"
@@ -154,7 +158,9 @@ class SqliteTranscriptStore(DefaultSessionStore):
         if not session_dir or not os.path.isdir(session_dir):
             return
         try:
-            filenames = [f for f in os.listdir(session_dir) if f.endswith(".json")]
+            filenames = sorted(
+                f for f in os.listdir(session_dir) if f.endswith(".json")
+            )
         except OSError:
             return
         if not filenames:
@@ -171,7 +177,7 @@ class SqliteTranscriptStore(DefaultSessionStore):
             try:
                 with open(filepath, "r", encoding="utf-8") as f:
                     data = json.load(f)
-            except (OSError, json.JSONDecodeError, TypeError):
+            except (OSError, UnicodeDecodeError, json.JSONDecodeError, TypeError):
                 continue
             if not isinstance(data, dict):
                 continue
@@ -210,7 +216,7 @@ class SqliteTranscriptStore(DefaultSessionStore):
         except (json.JSONDecodeError, TypeError):
             return None
 
-    def _write_row(self, session: SessionData, conn=None) -> bool:
+    def _write_row(self, session: SessionData, conn=None, *, overwrite: bool = True) -> bool:
         own_lock = conn is None
         if conn is None:
             conn = self._connect()
@@ -229,17 +235,21 @@ class SqliteTranscriptStore(DefaultSessionStore):
             getattr(session, "updated_at", None),
         )
         sql = (
-            "INSERT OR REPLACE INTO sessions "
+            ("INSERT OR REPLACE" if overwrite else "INSERT OR IGNORE") + " INTO sessions "
             "(session_id, data, agent_name, gateway_session_id, agent_id, updated_at) "
             "VALUES (?, ?, ?, ?, ?, ?)"
         )
         try:
             if own_lock:
                 with self._db_lock:
-                    conn.execute(sql, params)
+                    cursor = conn.execute(sql, params)
             else:
-                conn.execute(sql, params)
+                cursor = conn.execute(sql, params)
+            if not overwrite and cursor.rowcount == 0:
+                raise FileExistsError(session.session_id)
             return True
+        except FileExistsError:
+            raise
         except Exception as exc:
             logger.error("Failed to write session %s: %s", session.session_id, exc)
             return False
@@ -282,6 +292,52 @@ class SqliteTranscriptStore(DefaultSessionStore):
                 self._cache[session.session_id] = session
             return True
 
+    def _save_imported_session(self, session: SessionData, *, overwrite: bool = True) -> bool:
+        """Restore to the database without applying the destination's window."""
+        session.updated_at = datetime.now(timezone.utc).isoformat()
+        with self._db_lock:
+            # INSERT OR IGNORE makes no-clobber atomic across store instances;
+            # checking session_exists before a replace would still race.
+            if not self._write_row(session, overwrite=overwrite):
+                return False
+            with self._lock:
+                self._cache[session.session_id] = session
+            return True
+
+    def export_all(self) -> Dict[str, Any]:
+        """Export durable database rows, not legacy JSON sidecar files."""
+        return {
+            "version": self.PORTABLE_VERSION,
+            "sessions": [data for data in self._all_rows() if isinstance(data, dict)],
+        }
+
+    def _collect_lineage(
+        self, session: SessionData, *, exclude: str
+    ) -> List[Dict[str, Any]]:
+        """Collect persisted database records from the same conversation chain."""
+        lineage = self._lineage_key(session.to_dict())
+        if not lineage:
+            return []
+
+        def record_lineage(raw):
+            # Preserve the writer's Python JSON semantics (including NaN)
+            # and the shared lineage precedence, without retaining other rows.
+            try:
+                data = json.loads(raw)
+                return self._lineage_key(data) if isinstance(data, dict) else None
+            except (json.JSONDecodeError, TypeError, AttributeError):
+                return None
+
+        conn = self._connect()
+        with self._db_lock:
+            conn.create_function("portable_session_lineage", 1, record_lineage)
+            rows = conn.execute(
+                "SELECT data FROM sessions WHERE session_id != ? "
+                "AND portable_session_lineage(data) = ? ORDER BY updated_at DESC",
+                (exclude, lineage),
+            ).fetchall()
+        return [json.loads(row[0]) for row in rows]
+
     def _modify_session_locked(
         self,
         session_id: str,

src/praisonai-agents/praisonaiagents/session/store.py

@@ -21,6 +21,7 @@ try:
     _HAS_FCNTL = True
 except ImportError:
     _HAS_FCNTL = False
+from collections import OrderedDict
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Any, Callable, Dict, List, Optional
@@ -87,6 +88,48 @@ DEFAULT_RETENTION = RETENTION_COMPACT
 # past this many entries, so operators can spot runaway sessions on disk.
 ARCHIVE_WARN_THRESHOLD = 10_000
 
+# Bound the in-memory session cache so a long-lived gateway or bot process that
+# serves many users does not retain every session ever touched in RAM. The
+# on-disk file is already the source of truth, so an eviction just means the
+# next access reloads from disk. ``0`` or ``None`` disables the bound.
+DEFAULT_CACHE_MAXSIZE = 256
+
+
+class _LRUSessionCache(OrderedDict):
+    """An ``OrderedDict`` that evicts the least-recently-used entry once it
+    grows past ``maxsize``. Reads and writes refresh recency so the hot set
+    survives. ``maxsize <= 0`` disables eviction (unbounded, legacy behaviour).
+
+    All mutating/reading operations used by :class:`DefaultSessionStore`
+    (``in``, ``[]`` get/set, ``del``, ``pop``, ``clear``) work unchanged; only
+    insertion order is managed, so callers need no other changes.
+    """
+
+    def __init__(self, maxsize: int = DEFAULT_CACHE_MAXSIZE, on_evict=None):
+        super().__init__()
+        self.maxsize = maxsize
+        # Optional callback invoked with the key of each LRU-evicted entry, so a
+        # companion structure (e.g. the hierarchical store's _cache_mtimes) can
+        # drop the same key and stay bounded alongside this cache.
+        self._on_evict = on_evict
+
+    def __getitem__(self, key):
+        value = super().__getitem__(key)
+        self.move_to_end(key)
+        return value
+
+    def __setitem__(self, key, value):
+        if key in self:
+            super().__setitem__(key, value)
+            self.move_to_end(key)
+        else:
+            super().__setitem__(key, value)
+        if self.maxsize and self.maxsize > 0:
+            while len(self) > self.maxsize:
+                evicted_key, _ = self.popitem(last=False)
+                if self._on_evict is not None:
+                    self._on_evict(evicted_key)
+
 @dataclass
 class SessionMessage:
     """A single message in a session.
@@ -599,6 +642,7 @@ class DefaultSessionStore:
         retention: Optional[str] = None,
         active_window: Optional[int] = None,
         mirror: Optional[Any] = None,
+        cache_maxsize: int = DEFAULT_CACHE_MAXSIZE,
     ):
         """
         Initialize session store.
@@ -625,6 +669,11 @@ class DefaultSessionStore:
                 appended to the mirror on a background thread (local-first: the
                 local write always happens first and a mirror outage never
                 blocks or fails the turn). Left ``None`` there is zero overhead.
+            cache_maxsize: Maximum number of sessions kept in the in-memory LRU
+                cache (default 256). A long-lived multi-tenant process otherwise
+                retains every session ever touched in RAM. Disk is the source of
+                truth, so an evicted session simply reloads on next access. Pass
+                ``0`` to disable the bound (legacy unbounded cache).
         """
         self.session_dir = session_dir or _resolve_default_session_dir()
         self.max_messages = max_messages
@@ -649,7 +698,11 @@ class DefaultSessionStore:
         self.active_window = active_window if active_window is not None else max_messages
 
         self._lock = threading.RLock()
-        self._cache: Dict[str, SessionData] = {}
+        # LRU-bounded so a long-lived multi-tenant process does not retain every
+        # session ever touched in RAM. Pass cache_maxsize=0 for the legacy
+        # unbounded cache. Disk remains the source of truth, so an eviction just
+        # triggers a reload on next access.
+        self._cache: Dict[str, SessionData] = _LRUSessionCache(cache_maxsize)
 
         self._mirror = mirror
 
@@ -979,6 +1032,16 @@ class DefaultSessionStore:
         except Exception:  # pragma: no cover - observability must never break load
             logger.debug("SESSION corruption hook failed", exc_info=True)
 
+    def _report_unreadable_session(self, session_id: str, filepath: str, error: Exception) -> None:
+        """Report decode failures during scans without modifying the source file."""
+        if isinstance(error, (UnicodeDecodeError, json.JSONDecodeError)):
+            logger.warning("Skipping unreadable session file %s: %s", filepath, error)
+            self._fire_corruption_hook(session_id, str(error), None)
+
+    def _dump_session_json(self, data: Any, stream) -> None:
+        """Write durable JSON; subclasses can format private storage records."""
+        json.dump(data, stream, indent=2, ensure_ascii=False)
+
     def _atomic_write_json(self, filepath: str, data: Any) -> bool:
         """Atomically write JSON data to disk (temp file + os.replace)."""
         temp_path = None
@@ -993,7 +1056,7 @@ class DefaultSessionStore:
                 suffix=".tmp",
             ) as f:
                 temp_path = f.name
-                json.dump(data, f, indent=2, ensure_ascii=False)
+                self._dump_session_json(data, f)
 
             os.replace(temp_path, filepath)
             return True
@@ -1153,16 +1216,55 @@ class DefaultSessionStore:
         except (IOError, OSError):
             return
 
-        seen = {
-            (m.role, m.content, m.timestamp) for m in session.messages
-        }
+        def _freeze(value: Any) -> Any:
+            # A flat, iterative key avoids recursion both while converting and
+            # hashing deeply nested JSON. Sort object keys to preserve equality
+            # independently of insertion order; keep scalar types distinct.
+            tokens = []
+            stack = [("visit", value)]
+            ancestors = set()
+            while stack:
+                operation, item = stack.pop()
+                if operation == "leave":
+                    ancestors.remove(item)
+                elif operation == "key":
+                    tokens.append(("key", type(item), item))
+                elif isinstance(item, (list, dict)):
+                    identity = id(item)
+                    if identity in ancestors:
+                        raise ValueError("cyclic spill content")
+                    ancestors.add(identity)
+                    stack.append(("leave", identity))
+                    tokens.append((type(item), len(item)))
+                    if isinstance(item, list):
+                        stack.extend(("visit", child) for child in reversed(item))
+                    else:
+                        for key in sorted(item, reverse=True):
+                            stack.append(("visit", item[key]))
+                            stack.append(("key", key))
+                else:
+                    tokens.append((type(item), item))
+            return tuple(tokens)
+
+        def _key(message: SessionMessage) -> tuple:
+            return tuple(_freeze(value) for value in (
+                message.role, message.content, message.timestamp,
+            ))
+
+        seen = set()
+        for message in session.messages:
+            try:
+                seen.add(_key(message))
+            except (RecursionError, TypeError, ValueError):
+                # An unsupported existing value must not stop other salvage.
+                continue
         recovered: List[tuple] = []  # (filepath, [SessionMessage])
         for filename in candidates:
             filepath = os.path.join(spill_dir, filename)
             try:
                 with open(filepath, "r", encoding="utf-8") as f:
                     data = json.load(f)
-            except (json.JSONDecodeError, IOError, OSError):
+            except (json.JSONDecodeError, UnicodeDecodeError, RecursionError, IOError, OSError):
                 continue
             # A syntactically valid spill can still carry an unexpected shape
             # (non-object root, non-list messages, non-object message). Guard
@@ -1176,15 +1278,21 @@ class DefaultSessionStore:
             if not isinstance(raw_messages, list):
                 continue
             msgs = []
-            for raw in raw_messages:
-                if not isinstance(raw, dict):
-                    continue
-                msg = SessionMessage.from_dict(raw)
-                key = (msg.role, msg.content, msg.timestamp)
-                if key in seen:
-                    continue
-                seen.add(key)
-                msgs.append(msg)
+            pending_keys = set()
+            try:
+                for raw in raw_messages:
+                    if not isinstance(raw, dict):
+                        continue
+                    msg = SessionMessage.from_dict(raw)
+                    key = _key(msg)
+                    if key in seen or key in pending_keys:
+                        continue
+                    pending_keys.add(key)
+                    msgs.append(msg)
+            except (RecursionError, TypeError, ValueError):
+                # Retain the entire spill, without poisoning neighbor dedup.
+                continue
+            seen.update(pending_keys)
             recovered.append((filepath, msgs))
 
         if not recovered:
@@ -1201,7 +1309,7 @@ class DefaultSessionStore:
             # expose an oversized transcript that stays inconsistent until a
             # later mutation happens to compact it.
             self._enforce_window(session)
-            if not self._atomic_write_json(filepath, session.to_dict()):
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
                 # Could not fold the salvage back in durably — leave the spill
                 # files in place so a later load can retry.
                 return
@@ -1219,6 +1327,7 @@ class DefaultSessionStore:
         mutator: Callable[[SessionData], None],
         *,
         error_label: str = "modify session",
+        apply_retention: bool = True,
     ) -> bool:
         """Apply mutator after reloading from disk under FileLock."""
         filepath = self._get_session_path(session_id)
@@ -1234,9 +1343,10 @@ class DefaultSessionStore:
             mutator(session)
             session.updated_at = datetime.now(timezone.utc).isoformat()
 
-            self._enforce_window(session)
+            if apply_retention:
+                self._enforce_window(session)
 
-            if not self._atomic_write_json(filepath, session.to_dict()):
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
                 logger.error(f"Failed to {error_label} {session_id}")
                 return False
 
@@ -1245,6 +1355,10 @@ class DefaultSessionStore:
 
             return True
     
+    def _session_to_storage(self, session: SessionData) -> Dict[str, Any]:
+        """Serialize a durable record; subclasses may retain portable exports."""
+        return session.to_dict()
+
     def _save_session(self, session: SessionData) -> bool:
         """Save session to disk with atomic write."""
         filepath = self._get_session_path(session.session_id)
@@ -1254,7 +1368,7 @@ class DefaultSessionStore:
         self._enforce_window(session)
         
         with FileLock(filepath, self.lock_timeout):
-            if not self._atomic_write_json(filepath, session.to_dict()):
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
                 logger.error(f"Failed to save session {session.session_id}")
                 return False
             return True
@@ -1321,7 +1435,7 @@ class DefaultSessionStore:
             self._enforce_window(session)
             
             # Write atomically
-            if not self._atomic_write_json(filepath, session.to_dict()):
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
                 logger.error(f"Failed to save session {session_id}")
                 # Issue #3597: durable write failed (disk-full / corruption).
                 # Spill just this turn to a fallback file and fire the
@@ -1622,6 +1736,27 @@ class DefaultSessionStore:
             session_id, _apply, error_label="update session metadata"
         )
 
+    def merge_session_metadata_map(
+        self, session_id: str, key: str, updates: Dict[str, Any],
+        *, defaults: Optional[Dict[str, Any]] = None,
+    ) -> bool:
+        """Merge nested metadata entries using the freshest map under FileLock.
+
+        Defaults fill missing entries; persisted entries win over defaults,
+        and the supplied updates win over both.
+        """
+        def _apply(session: SessionData) -> None:
+            merged = dict(defaults or {})
+            current = session.metadata.get(key)
+            if isinstance(current, dict):
+                merged.update(current)
+            merged.update(updates)
+            session.metadata[key] = merged
+
+        return self._modify_session_locked(
+            session_id, _apply, error_label="merge session metadata map"
+        )
+
     def rename_session(self, session_id: str, title: str) -> bool:
         """Give a session a human-readable title (Issue #3737).
 
@@ -1693,7 +1828,8 @@ class DefaultSessionStore:
                             "total_tokens": data.get("total_tokens") or data.get("token_count") or (data.get("metadata") or {}).get("total_tokens"),
                             "cost": data.get("cost") or (data.get("metadata") or {}).get("cost"),
                         })
-                    except (json.JSONDecodeError, IOError):
+                    except (UnicodeDecodeError, json.JSONDecodeError, IOError) as exc:
+                        self._report_unreadable_session(filename[:-5], filepath, exc)
                         continue
         except (IOError, OSError):
             pass
@@ -1727,7 +1863,8 @@ class DefaultSessionStore:
                             data = json.load(f)
                         if data.get("agent_name") == agent_name:
                             session_ids.append(data.get("session_id", filename[:-5]))
-                    except (json.JSONDecodeError, IOError):
+                    except (UnicodeDecodeError, json.JSONDecodeError, IOError) as exc:
+                        self._report_unreadable_session(filename[:-5], filepath, exc)
                         continue
         except (IOError, OSError):
             pass
@@ -1835,7 +1972,8 @@ class DefaultSessionStore:
                             data = json.load(f)
                         if data.get("gateway_session_id") == gateway_session_id:
                             return SessionData.from_dict(data)
-                    except (json.JSONDecodeError, IOError):
+                    except (UnicodeDecodeError, json.JSONDecodeError, IOError) as exc:
+                        self._report_unreadable_session(filename[:-5], filepath, exc)
                         continue
         except (IOError, OSError):
             pass
@@ -1865,7 +2003,8 @@ class DefaultSessionStore:
                             data = json.load(f)
                         if data.get("agent_id") == agent_id:
                             session_ids.append(data.get("session_id", filename[:-5]))
-                    except (json.JSONDecodeError, IOError):
+                    except (UnicodeDecodeError, json.JSONDecodeError, IOError) as exc:
+                        self._report_unreadable_session(filename[:-5], filepath, exc)
                         continue
         except (IOError, OSError):
             pass
@@ -2075,7 +2214,8 @@ class DefaultSessionStore:
             try:
                 with open(filepath, "r", encoding="utf-8") as f:
                     data = json.load(f)
-            except (json.JSONDecodeError, IOError):
+            except (UnicodeDecodeError, json.JSONDecodeError, IOError) as exc:
+                self._report_unreadable_session(filename[:-5], filepath, exc)
                 continue
 
             messages = self._searchable_messages(data)
@@ -2256,12 +2396,9 @@ class DefaultSessionStore:
         return {"version": self.PORTABLE_VERSION, "sessions": sessions}
 
     def export_all(self) -> Dict[str, Any]:
-        """Export every stored session to a portable, versioned payload."""
+        """Export every stored session; raise OSError for an incomplete backup."""
         sessions: List[Dict[str, Any]] = []
-        try:
-            filenames = os.listdir(self.session_dir)
-        except (IOError, OSError):
-            filenames = []
+        filenames = os.listdir(self.session_dir)
         for filename in filenames:
             if not filename.endswith(".json"):
                 continue
@@ -2269,8 +2406,9 @@ class DefaultSessionStore:
             try:
                 with open(filepath, "r", encoding="utf-8") as f:
                     sessions.append(json.load(f))
-            except (json.JSONDecodeError, IOError, OSError):
-                continue
+            except (UnicodeDecodeError, json.JSONDecodeError, IOError, OSError) as exc:
+                self._report_unreadable_session(filename[:-5], filepath, exc)
+                raise OSError(f"Incomplete session export: cannot read {filename}") from exc
         return {"version": self.PORTABLE_VERSION, "sessions": sessions}
 
     def _collect_lineage(
@@ -2282,15 +2420,13 @@ class DefaultSessionStore:
         (``lineage_id`` / ``root_session_id`` / ``thread_id``) so a compacted /
         rotated continuation exports alongside its logical session. Returns the
         raw session dicts (already portable). Empty when no lineage is known.
+        Raises OSError when an unreadable record makes lineage coverage unknown.
         """
         lineage = self._lineage_key(session.to_dict())
         if not lineage:
             return []
         out: List[Dict[str, Any]] = []
-        try:
-            filenames = os.listdir(self.session_dir)
-        except (IOError, OSError):
-            return []
+        filenames = os.listdir(self.session_dir)
         for filename in filenames:
             if not filename.endswith(".json"):
                 continue
@@ -2298,15 +2434,16 @@ class DefaultSessionStore:
             try:
                 with open(filepath, "r", encoding="utf-8") as f:
                     data = json.load(f)
-            except (json.JSONDecodeError, IOError, OSError):
-                continue
+            except (UnicodeDecodeError, json.JSONDecodeError, IOError, OSError) as exc:
+                self._report_unreadable_session(filename[:-5], filepath, exc)
+                raise OSError(f"Incomplete session export: cannot read {filename}") from exc
             if data.get("session_id") == exclude:
                 continue
             if self._lineage_key(data) == lineage:
                 out.append(data)
         return out
 
-    def _save_imported_session(self, session: SessionData) -> bool:
+    def _save_imported_session(self, session: SessionData, *, overwrite: bool = True) -> bool:
         """Persist a restored session verbatim (no retention/window applied).
 
         Mirrors ``_save_session`` (timestamp + atomic, file-locked write) but
@@ -2317,11 +2454,35 @@ class DefaultSessionStore:
         filepath = self._get_session_path(session.session_id)
         session.updated_at = datetime.now(timezone.utc).isoformat()
         with FileLock(filepath, self.lock_timeout):
-            if not self._atomic_write_json(filepath, session.to_dict()):
+            # The initial import check can race with a peer creating this file.
+            # Check again inside the same lock that protects the replacement.
+            if not overwrite and os.path.exists(filepath):
+                raise FileExistsError(filepath)
+            if not self._atomic_write_json(filepath, self._session_to_storage(session)):
                 logger.error(f"Failed to save imported session {session.session_id}")
                 return False
+            session._import_file_identity = self._session_file_identity(filepath)
+            # Persistence hooks own their cache value; a subclass may refresh
+            # it from a newer durable generation before returning to import.
+            with self._lock:
+                self._cache[session.session_id] = session
             return True
 
+    @staticmethod
+    def _session_file_identity(filepath):
+        """Identify an atomic file generation without depending on wall-clock ordering."""
+        try:
+            stat = os.stat(filepath)
+        except OSError:
+            # A path metadata failure need not make the atomic generation
+            # unknowable: an opened descriptor can still identify that file.
+            try:
+                with open(filepath, "rb") as current:
+                    stat = os.fstat(current.fileno())
+            except OSError:
+                return None
+        return stat.st_dev, stat.st_ino, stat.st_mtime_ns, stat.st_size
+
     def import_sessions(
         self,
         payload: Dict[str, Any],
@@ -2416,14 +2577,16 @@ class DefaultSessionStore:
                 # Persist the imported record verbatim: an import is a restore,
                 # so the destination's retention/active_window must not truncate
                 # or compact a valid larger export before it lands on disk.
-                if not self._save_imported_session(session):
+                if not self._save_imported_session(session, overwrite=overwrite):
                     report.skipped.append(
                         {"session_id": session_id, "reason": "write failed"}
                     )
                     continue
-                with self._lock:
-                    self._cache[session_id] = session
                 report.imported += 1
+            except FileExistsError:
+                report.skipped.append(
+                    {"session_id": session_id, "reason": "already exists (use overwrite)"}
+                )
             except Exception as e:  # pragma: no cover - defensive; one bad record
                 report.skipped.append(
                     {"session_id": session_id, "reason": f"import error: {e}"}

src/praisonai-agents/praisonaiagents/session/title.py

@@ -66,6 +66,9 @@ def generate_title(
         ... )
         >>> # Returns something like: "Python Code Debugging Help"
     """
+    if max_length <= 0:
+        return ""
+
     # Fallback title from user message if generation fails
     fallback_title = _create_fallback_title(user_msg, max_length)
     
@@ -119,6 +122,9 @@ async def generate_title_async(
     Returns:
         Generated title string, or fallback based on user message
     """
+    if max_length <= 0:
+        return ""
+
     fallback_title = _create_fallback_title(user_msg, max_length)
     
     try:
@@ -152,8 +158,7 @@ Return ONLY the title text, no quotes, no explanation."""
                 title = title.replace('\n', ' ').replace('\r', '')
                 
                 # Truncate if too long
-                if len(title) > max_length:
-                    title = title[:max_length-3] + "..."
+                title = _truncate_title(title, max_length)
                 
                 # Return if non-empty
                 if title and len(title.strip()) > 0:
@@ -176,6 +181,17 @@ Return ONLY the title text, no quotes, no explanation."""
     return fallback_title
 
 
+def _truncate_title(title: str, max_length: int) -> str:
+    """Apply the character cap without using negative ellipsis slices."""
+    if max_length <= 0:
+        return ""
+    if len(title) <= max_length:
+        return title
+    if max_length <= 3:
+        return title[:max_length]
+    return title[:max_length - 3] + "..."
+
+
 def _create_fallback_title(user_msg: str, max_length: int) -> str:
     """Create a fallback title from the user message.
     
@@ -187,7 +203,7 @@ def _create_fallback_title(user_msg: str, max_length: int) -> str:
         Simple title based on user message
     """
     if not user_msg or not user_msg.strip():
-        return "Chat Session"
+        return _truncate_title("Chat Session", max_length)
     
     # Clean up the message
     clean_msg = user_msg.strip()
@@ -200,8 +216,4 @@ def _create_fallback_title(user_msg: str, max_length: int) -> str:
     if '.' in clean_msg:
         clean_msg = clean_msg.split('.')[0]
     
-    # Truncate if too long
-    if len(clean_msg) > max_length:
-        clean_msg = clean_msg[:max_length-3] + "..."
-    
-    return clean_msg if clean_msg else "Chat Session"
+    return _truncate_title(clean_msg or "Chat Session", max_length)

src/praisonai-agents/praisonaiagents/storage/__init__.py

@@ -29,6 +29,10 @@ __all__ = [
     "FileBackend",
     "SQLiteBackend", 
     "get_backend",
+    # Hardened stdlib SQLite connection factory (WAL-with-safe-fallback)
+    "sqlite_connect",
+    # Structural integrity probe for durable SQLite stores (Issue #5387)
+    "sqlite_quick_check",
 ]
 
 _LAZY_IMPORTS = {
@@ -46,6 +50,8 @@ _LAZY_IMPORTS = {
     "FileBackend": ("backends", "FileBackend"),
     "SQLiteBackend": ("backends", "SQLiteBackend"),
     "get_backend": ("backends", "get_backend"),
+    "sqlite_connect": ("sqlite", "connect"),
+    "sqlite_quick_check": ("sqlite", "quick_check"),
 }
 
 

src/praisonai-agents/praisonaiagents/storage/backends.py

@@ -60,6 +60,7 @@ class FileBackend:
         self.suffix = suffix
         self.pretty = pretty
         self._lock = threading.Lock()
+        self._unreadable_paths = set()
         
         # Create directory if needed
         self.storage_dir.mkdir(parents=True, exist_ok=True)
@@ -69,11 +70,26 @@ class FileBackend:
         # Sanitize key for filesystem
         safe_key = "".join(c if c.isalnum() or c in "-_" else "_" for c in key)
         return self.storage_dir / f"{safe_key}{self.suffix}"
+
+    def _path_to_key(self, file_path: Path) -> Optional[str]:
+        """Recognize records using the full configured suffix."""
+        if not file_path.is_file():
+            return None
+        if self.suffix:
+            if file_path.name.endswith(self.suffix):
+                return file_path.name[:-len(self.suffix)]
+            return None
+        # Preserve the existing no-extension selection for an empty suffix.
+        return file_path.name if not file_path.suffix else None
     
     def save(self, key: str, data: Dict[str, Any]) -> None:
         """Save data with the given key."""
         with self._lock:
             file_path = self._key_to_path(key)
+            if file_path in self._unreadable_paths:
+                if file_path.exists():
+                    raise OSError(f"Refusing to overwrite unreadable record: {file_path}")
+                self._unreadable_paths.discard(file_path)
             
             # Atomic write via temp file
             try:
@@ -84,11 +100,11 @@ class FileBackend:
                     delete=False,
                     suffix=".tmp"
                 ) as f:
+                    temp_path = f.name
                     if self.pretty:
                         json.dump(data, f, indent=2, default=str, ensure_ascii=False)
                     else:
                         json.dump(data, f, default=str, ensure_ascii=False)
-                    temp_path = f.name
                 
                 os.replace(temp_path, file_path)
             except Exception as e:
@@ -101,18 +117,25 @@ class FileBackend:
                 raise
     
     def load(self, key: str) -> Optional[Dict[str, Any]]:
-        """Load data by key."""
+        """Load data by key; failed reads protect the record from later saves.
+
+        A successful reload or deletion permits writes again. Returning None
+        for a failed read must not let a caller persist defaults over its source.
+        """
         file_path = self._key_to_path(key)
-        
-        if not file_path.exists():
-            return None
-        
-        try:
-            with open(file_path, "r", encoding="utf-8") as f:
-                return json.load(f)
-        except (json.JSONDecodeError, IOError) as e:
-            logger.warning(f"Failed to load {key}: {e}")
-            return None
+        with self._lock:
+            if not file_path.exists():
+                self._unreadable_paths.discard(file_path)
+                return None
+            try:
+                with open(file_path, "r", encoding="utf-8") as f:
+                    data = json.load(f)
+                self._unreadable_paths.discard(file_path)
+                return data
+            except (UnicodeDecodeError, json.JSONDecodeError, IOError) as e:
+                self._unreadable_paths.add(file_path)
+                logger.warning(f"Failed to load {key}: {e}")
+                return None
     
     def delete(self, key: str) -> bool:
         """Delete data by key."""
@@ -122,6 +145,7 @@ class FileBackend:
             if file_path.exists():
                 try:
                     file_path.unlink()
+                    self._unreadable_paths.discard(file_path)
                     return True
                 except Exception as e:
                     logger.error(f"Failed to delete {key}: {e}")
@@ -133,8 +157,8 @@ class FileBackend:
         keys = []
         
         for file_path in self.storage_dir.iterdir():
-            if file_path.is_file() and file_path.suffix == self.suffix:
-                key = file_path.stem
+            key = self._path_to_key(file_path)
+            if key is not None:
                 if not prefix or key.startswith(prefix):
                     keys.append(key)
         
@@ -149,9 +173,10 @@ class FileBackend:
         count = 0
         with self._lock:
             for file_path in self.storage_dir.iterdir():
-                if file_path.is_file() and file_path.suffix == self.suffix:
+                if self._path_to_key(file_path) is not None:
                     try:
                         file_path.unlink()
+                        self._unreadable_paths.discard(file_path)
                         count += 1
                     except Exception:
                         pass
@@ -191,28 +216,34 @@ class SQLiteBackend:
         if not isinstance(table_name, str) or not _re.match(r'^[a-zA-Z0-9_]+$', table_name):
             raise ValueError("table_name must contain only alphanumeric characters and underscores")
         self.table_name = table_name
+        # Quote identifiers so validated names that are SQL keywords or start
+        # with a digit (e.g. "select", "9table") are valid in all statements.
+        self._quoted_table = f'"{table_name}"'
+        self._quoted_index = f'"idx_{table_name}_key"'
         self._local = threading.local()
         
         # Ensure directory exists
         Path(self.db_path).parent.mkdir(parents=True, exist_ok=True)
         
         if auto_create:
-            self._create_table()
+            try:
+                self._create_table()
+            except Exception:
+                self.close()
+                raise
     
     def _get_conn(self):
         """Get thread-local connection."""
         # Lazy import sqlite3
         import sqlite3
-        
+
         if not hasattr(self._local, "conn") or self._local.conn is None:
-            self._local.conn = sqlite3.connect(
-                self.db_path,
-                check_same_thread=False
-            )
+            # Hardened factory: WAL where the filesystem supports it, safe
+            # DELETE fallback on NFS/SMB/FUSE/virtiofs (Issue #5264).
+            from .sqlite import connect as _sqlite_connect
+
+            self._local.conn = _sqlite_connect(self.db_path, isolation_level=None)
             self._local.conn.row_factory = sqlite3.Row
-            # Enable WAL mode for better concurrent write safety
-            self._local.conn.execute("PRAGMA journal_mode=WAL")
-            self._local.conn.execute("PRAGMA synchronous=NORMAL")
             self._local.conn.commit()
         return self._local.conn
     
@@ -222,7 +253,7 @@ class SQLiteBackend:
         cur = conn.cursor()
         
         cur.execute(f"""
-            CREATE TABLE IF NOT EXISTS {self.table_name} (
+            CREATE TABLE IF NOT EXISTS {self._quoted_table} (
                 key TEXT PRIMARY KEY,
                 data TEXT NOT NULL,
                 created_at REAL DEFAULT (strftime('%s', 'now')),
@@ -232,8 +263,8 @@ class SQLiteBackend:
         
         # Index for prefix queries
         cur.execute(f"""
-            CREATE INDEX IF NOT EXISTS idx_{self.table_name}_key 
-            ON {self.table_name}(key)
+            CREATE INDEX IF NOT EXISTS {self._quoted_index} 
+            ON {self._quoted_table}(key)
         """)
         
         conn.commit()
@@ -246,7 +277,7 @@ class SQLiteBackend:
         json_data = json.dumps(data, default=str, ensure_ascii=False)
         
         cur.execute(f"""
-            INSERT INTO {self.table_name} (key, data, updated_at)
+            INSERT INTO {self._quoted_table} (key, data, updated_at)
             VALUES (?, ?, strftime('%s', 'now'))
             ON CONFLICT(key) DO UPDATE SET
                 data = excluded.data,
@@ -261,7 +292,7 @@ class SQLiteBackend:
         cur = conn.cursor()
         
         cur.execute(f"""
-            SELECT data FROM {self.table_name} WHERE key = ?
+            SELECT data FROM {self._quoted_table} WHERE key = ?
         """, (key,))
         
         row = cur.fetchone()
@@ -278,7 +309,7 @@ class SQLiteBackend:
         cur = conn.cursor()
         
         cur.execute(f"""
-            DELETE FROM {self.table_name} WHERE key = ?
+            DELETE FROM {self._quoted_table} WHERE key = ?
         """, (key,))
         
         deleted = cur.rowcount > 0
@@ -292,13 +323,13 @@ class SQLiteBackend:
         
         if prefix:
             cur.execute(f"""
-                SELECT key FROM {self.table_name}
-                WHERE key LIKE ?
+                SELECT key FROM {self._quoted_table}
+                WHERE instr(key, ?) = 1
                 ORDER BY key
-            """, (f"{prefix}%",))
+            """, (prefix,))
         else:
             cur.execute(f"""
-                SELECT key FROM {self.table_name}
+                SELECT key FROM {self._quoted_table}
                 ORDER BY key
             """)
         
@@ -310,7 +341,7 @@ class SQLiteBackend:
         cur = conn.cursor()
         
         cur.execute(f"""
-            SELECT 1 FROM {self.table_name} WHERE key = ? LIMIT 1
+            SELECT 1 FROM {self._quoted_table} WHERE key = ? LIMIT 1
         """, (key,))
         
         return cur.fetchone() is not None
@@ -320,10 +351,8 @@ class SQLiteBackend:
         conn = self._get_conn()
         cur = conn.cursor()
         
-        cur.execute(f"SELECT COUNT(*) as count FROM {self.table_name}")
-        count = cur.fetchone()["count"]
-        
-        cur.execute(f"DELETE FROM {self.table_name}")
+        cur.execute(f"DELETE FROM {self._quoted_table}")
+        count = cur.rowcount
         conn.commit()
         
         return count

src/praisonai-agents/praisonaiagents/storage/base.py

@@ -191,6 +191,7 @@ class BaseJSONStore:
         self._lock = threading.Lock()
         self._data: Dict[str, Any] = {}
         self._backend = backend
+        self._read_failed = False
         
         if backend is None:
             # File-based storage (default)
@@ -236,7 +237,9 @@ class BaseJSONStore:
             else:
                 with open(self.storage_path, "r", encoding="utf-8") as f:
                     self._data = json.load(f)
-        except (json.JSONDecodeError, IOError) as e:
+            self._read_failed = False
+        except (UnicodeDecodeError, json.JSONDecodeError, IOError) as e:
+            self._read_failed = not isinstance(e, FileNotFoundError)
             logger.warning(f"Failed to load {self.storage_path}: {e}")
             self._data = self._default_data()
     
@@ -250,6 +253,13 @@ class BaseJSONStore:
                 raise
             return
         
+        # Defaults from a failed read must not replace their unreadable source.
+        # A successful reload or explicit reset/deletion permits writes again.
+        if self._read_failed:
+            if self.storage_path.exists():
+                raise OSError(f"Refusing to overwrite unreadable record: {self.storage_path}")
+            self._read_failed = False
+
         try:
             dir_path = self.storage_path.parent
             dir_path.mkdir(parents=True, exist_ok=True)
@@ -262,8 +272,8 @@ class BaseJSONStore:
                 delete=False,
                 suffix=".tmp"
             ) as f:
-                json.dump(self._data, f, indent=2, default=str, ensure_ascii=False)
                 temp_path = f.name
+                json.dump(self._data, f, indent=2, default=str, ensure_ascii=False)
             
             # Atomic rename
             if self.use_file_lock:
@@ -312,9 +322,18 @@ class BaseJSONStore:
     
     def clear(self) -> None:
         """Clear all stored data."""
+        self._reset(self._default_data())
+
+    def _reset(self, data: Dict[str, Any]) -> None:
+        """Atomically replace data for an explicit clear, restoring guards on failure."""
         with self._lock:
-            self._data = self._default_data()
-            self._save()
+            previous_data, previous_failed = self._data, self._read_failed
+            self._data, self._read_failed = data, False
+            try:
+                self._save()
+            except BaseException:
+                self._data, self._read_failed = previous_data, previous_failed
+                raise
     
     def delete(self) -> bool:
         """
@@ -334,6 +353,7 @@ class BaseJSONStore:
                 if self.storage_path.exists():
                     self.storage_path.unlink()
                     self._data = self._default_data()
+                    self._read_failed = False
                     return True
                 return False
             except Exception as e:
@@ -343,7 +363,7 @@ class BaseJSONStore:
 def list_json_sessions(
     storage_dir: Path,
     suffix: str = ".json",
-    limit: int = 50,
+    limit: Optional[int] = 50,
 ) -> List[BaseSessionInfo]:
     """
     List all sessions in a directory.
@@ -353,7 +373,7 @@ def list_json_sessions(
     Args:
         storage_dir: Directory to search
         suffix: File suffix to filter by
-        limit: Maximum number of sessions to return
+        limit: Maximum number of sessions to return, or None for all sessions
         
     Returns:
         List of BaseSessionInfo, sorted by modification time (newest first)
@@ -370,7 +390,7 @@ def list_json_sessions(
                 # Try to get item count from file
                 if suffix == ".json":
                     try:
-                        with open(file_path, "r") as f:
+                        with open(file_path, "r", encoding="utf-8") as f:
                             data = json.load(f)
                             # Common patterns for item count
                             if "iterations" in data:
@@ -383,7 +403,7 @@ def list_json_sessions(
                         pass
                 elif suffix == ".jsonl":
                     try:
-                        with open(file_path, "r") as f:
+                        with open(file_path, "r", encoding="utf-8") as f:
                             info.item_count = sum(1 for _ in f)
                     except Exception:
                         pass
@@ -424,7 +444,7 @@ def cleanup_old_sessions(
     now = datetime.now()
     
     # Get all sessions sorted by age (oldest first)
-    sessions = list_json_sessions(storage_dir, suffix, limit=10000)
+    sessions = list_json_sessions(storage_dir, suffix, limit=None)
     sessions.sort(key=lambda s: s.modified_at)
     
     # Delete old sessions
@@ -484,6 +504,7 @@ class AsyncBaseJSONStore:
         self.storage_path = Path(storage_path)
         self._backend = backend
         self._data: Dict[str, Any] = {}
+        self._read_failed = False
         self._async_lock: Optional[Any] = None  # Lazy init asyncio.Lock
         self._async_lock_init: threading.Lock = threading.Lock()  # Thread-safe initialization
         
@@ -524,7 +545,12 @@ class AsyncBaseJSONStore:
                     async with aiofiles.open(self.storage_path, "r", encoding="utf-8") as f:
                         content = await f.read()
                         self._data = json.loads(content)
-                except (FileNotFoundError, json.JSONDecodeError):
+                    self._read_failed = False
+                except FileNotFoundError:
+                    self._read_failed = False
+                    self._data = self._default_data()
+                except (UnicodeDecodeError, json.JSONDecodeError):
+                    self._read_failed = True
                     self._data = self._default_data()
             
             return self._data.copy()
@@ -541,11 +567,34 @@ class AsyncBaseJSONStore:
                 else:
                     self._backend.save(self._storage_key, self._data)
             else:
-                # File-based async write
+                if self._read_failed:
+                    if self.storage_path.exists():
+                        raise OSError(f"Refusing to overwrite unreadable record: {self.storage_path}")
+                    self._read_failed = False
+                # File-based async write. Serialise to a string FIRST, write it
+                # to a temp file, then os.replace it into place so a crash/kill
+                # mid-write cannot truncate the previous good file (writing
+                # directly to the target truncates it before any bytes land).
                 import aiofiles
                 self.storage_path.parent.mkdir(parents=True, exist_ok=True)
-                async with aiofiles.open(self.storage_path, "w", encoding="utf-8") as f:
-                    await f.write(json.dumps(self._data, indent=2, default=str, ensure_ascii=False))
+                payload = json.dumps(self._data, indent=2, default=str, ensure_ascii=False)
+                dir_path = self.storage_path.parent
+                fd, temp_path = tempfile.mkstemp(
+                    dir=str(dir_path),
+                    prefix=self.storage_path.name + ".",
+                    suffix=".tmp",
+                )
+                os.close(fd)
+                try:
+                    async with aiofiles.open(temp_path, "w", encoding="utf-8") as f:
+                        await f.write(payload)
+                    os.replace(temp_path, self.storage_path)
+                except BaseException:
+                    try:
+                        os.remove(temp_path)
+                    except OSError:
+                        pass
+                    raise
     
     async def exists_async(self) -> bool:
         """Check if storage exists asynchronously."""
@@ -572,6 +621,7 @@ class AsyncBaseJSONStore:
             if self.storage_path.exists():
                 self.storage_path.unlink()
                 self._data = self._default_data()
+                self._read_failed = False
                 return True
             return False
 

src/praisonai-agents/praisonaiagents/storage/sqlite.py

@@ -0,0 +1,557 @@
+"""Hardened stdlib-only SQLite connection helper for core durable stores.
+
+Every SQLite-backed store in the gateway/session/memory/run subsystems used to
+open its connection with a bare ``PRAGMA journal_mode=WAL`` duplicated inline.
+WAL relies on a shared-memory (``-shm``) sidecar and byte-range locking that
+several common *deployment substrates* do not honour: NFS/SMB home
+directories, FUSE / Docker-Desktop gRPC-FUSE mounts, overlayfs, virtiofs/9p on
+cross-VM setups. On those filesystems WAL either errors (``disk I/O error``,
+``database is locked``, ``unable to open database file``) or — worst case on a
+cross-VM share — silently corrupts.
+
+This module provides a single, lightweight, dependency-free factory that every
+store can call so the hardening lives in one place:
+
+    from praisonaiagents.storage.sqlite import connect
+    conn = connect(db_path)
+
+It attempts WAL and, on a filesystem that rejects it, transparently falls back
+to ``journal_mode=DELETE`` — but **never live-downgrades a database whose
+on-disk header already reports WAL** (a peer connection may hold uncheckpointed
+commits). A read-only header probe runs first so initialisation never unlinks
+sidecars another connection is using.
+
+Only the standard library is used (``sqlite3`` is stdlib), so this adds zero
+heavy dependencies and no import-time cost — it is lazy-imported by callers.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Optional
+
+from praisonaiagents._logging import get_logger
+
+logger = get_logger(__name__)
+
+# SQLite database header: bytes 18 (write) / 19 (read) hold the file-format
+# version numbers; both are 2 for a WAL database and 1 for rollback-journal
+# (DELETE/TRUNCATE/PERSIST). Reading them lets us detect an existing WAL header
+# without opening a write connection, so we never live-downgrade a DB a peer
+# may hold uncheckpointed commits in.
+_WAL_FILE_FORMAT = 2
+
+
+def _resolve_probe_path(path: str) -> Optional[str]:
+    """Resolve a connection target to the on-disk file to probe, or None.
+
+    ``:memory:`` and anonymous/temporary databases have no header to read.
+    ``file:`` URIs (used with ``uri=True``) embed the real filename plus a query
+    string; parsing them back to the filesystem path keeps the no-downgrade
+    header probe honest instead of stat-ing the URI text as a literal filename.
+    """
+    if not path or path == ":memory:":
+        return None
+    if path.startswith("file:"):
+        from urllib.parse import unquote, urlparse
+
+        parsed = urlparse(path)
+        # In-memory / anonymous URIs (``file::memory:``, ``file:?...``) have no
+        # backing file to probe.
+        if not parsed.path or parsed.path == ":memory:":
+            return None
+        return unquote(parsed.path)
+    return path
+
+
+def _header_is_wal(path: str) -> Optional[bool]:
+    """Return True/False if the on-disk header reports WAL, or None if unknown.
+
+    Reads only the fixed 100-byte SQLite header, so it never touches sidecars
+    or takes a lock. Returns None when the file is absent/empty/too short to
+    have a header yet (a fresh DB), so the caller treats it as "free to set".
+    Accepts either a plain filesystem path or a ``file:`` URI.
+    """
+    probe = _resolve_probe_path(path)
+    if probe is None:
+        return None
+    try:
+        with open(probe, "rb") as fh:
+            header = fh.read(20)
+    except (IOError, OSError):
+        return None
+    if len(header) < 20:
+        return None
+    return header[18] == _WAL_FILE_FORMAT and header[19] == _WAL_FILE_FORMAT
+
+
+def apply_wal_with_fallback(
+    conn, path: str, busy_timeout_ms: int = 5000, synchronous: str = "FULL"
+) -> str:
+    """Set a durable journal mode, falling back off WAL on hostile filesystems.
+
+    Attempts ``journal_mode=WAL``. If the filesystem rejects it (a locking or
+    ``disk I/O`` style error, common on NFS/SMB/FUSE/virtiofs), falls back to
+    ``journal_mode=DELETE`` — unless the on-disk header already reports WAL, in
+    which case the DB is left in WAL so a peer connection's uncheckpointed
+    commits are never stranded. Returns the journal mode actually in effect.
+
+    ``synchronous`` controls the ``PRAGMA synchronous`` durability level and
+    defaults to ``FULL`` so committed writes survive an OS/power crash — this
+    preserves the pre-hardening durability of the core stores (Issue #5264).
+    Callers that favour throughput over the last-transaction durability window
+    may pass ``synchronous="NORMAL"`` (safe under WAL against process crashes).
+    """
+    sync = (synchronous or "FULL").upper()
+
+    try:
+        conn.execute(f"PRAGMA busy_timeout={int(busy_timeout_ms)}")
+    except Exception:  # pragma: no cover - busy_timeout is universally supported
+        pass
+
+    try:
+        row = conn.execute("PRAGMA journal_mode=WAL").fetchone()
+        mode = (row[0] if row else "").lower()
+        if mode == "wal":
+            try:
+                conn.execute(f"PRAGMA synchronous={sync}")
+            except Exception:
+                pass
+            return "wal"
+    except Exception as exc:
+        logger.info(
+            "WAL journal mode unavailable for %s (%s); trying DELETE fallback.",
+            path,
+            exc,
+        )
+
+    # WAL did not take. Never live-downgrade a DB whose header already reports
+    # WAL — another connection may hold uncheckpointed commits in the -wal file.
+    # ``_header_is_wal`` resolves ``file:`` URIs and rejects ``:memory:`` itself.
+    if _header_is_wal(path):
+        logger.warning(
+            "WAL requested but not applied for %s and its header already "
+            "reports WAL; leaving as-is to avoid stranding peer commits.",
+            path,
+        )
+        return "wal"
+
+    try:
+        row = conn.execute("PRAGMA journal_mode=DELETE").fetchone()
+        mode = (row[0] if row else "delete").lower()
+    except Exception as exc:
+        logger.warning("Could not set DELETE journal mode for %s: %s", path, exc)
+        return "unknown"
+    try:
+        # FULL is the safe default for rollback-journal mode on unreliable
+        # filesystems; NORMAL is only durable under WAL.
+        conn.execute("PRAGMA synchronous=FULL")
+    except Exception:
+        pass
+    return mode
+
+
+# Maximum forensic backups of a malformed file kept before the oldest is
+# pruned, so a boot-loop on a permanently-bad file cannot fill the disk.
+_MAX_FORENSIC_BACKUPS = 3
+# A malformed file is repaired at most this many times before the guard gives
+# up (records the exhaustion in a sidecar ledger) and stops re-repairing on
+# every boot, so a permanently-bad file is not churned each start.
+_MAX_REPAIR_ATTEMPTS = 3
+
+
+def _is_transient_lock_error(exc: Exception) -> bool:
+    """True if an error is a transient lock/busy condition, not corruption.
+
+    A healthy database that a peer momentarily locks raises ``database is
+    locked`` / ``database is busy``. Treating that as corruption would trigger a
+    needless forensic copy and repair on a file that was never corrupt, so the
+    guard must distinguish it from a genuine "malformed image" error.
+    """
+    text = str(exc).lower()
+    return "locked" in text or "database is busy" in text
+
+
+def quick_check(conn) -> bool:
+    """Return True if ``PRAGMA quick_check`` reports the database is intact.
+
+    ``quick_check`` is the cheap structural probe (it skips the exhaustive
+    per-row index cross-check that ``integrity_check`` does), so it is safe to
+    run once on open without touching the hot path. Any SQLite-level error
+    (a malformed header, "database disk image is malformed", an I/O error) is
+    treated as *not intact* so the caller can route to backup/repair. A
+    transient lock/busy error is re-raised so callers can tell "locked" (leave
+    alone) apart from "corrupt" (back up + repair).
+    """
+    try:
+        row = conn.execute("PRAGMA quick_check(1)").fetchone()
+    except Exception as exc:  # DatabaseError / OperationalError on a bad file
+        if _is_transient_lock_error(exc):
+            raise
+        logger.warning("quick_check raised for database (%s); treating as corrupt.", exc)
+        return False
+    return bool(row) and str(row[0]).lower() == "ok"
+
+
+def _sidecar_paths(path: str):
+    """Return the WAL/SHM sidecar paths for a database file."""
+    return [path + "-wal", path + "-shm"]
+
+
+def _forensic_backup(path: str) -> Optional[str]:
+    """Copy a malformed database (+WAL/SHM sidecars) aside for forensics.
+
+    Mirrors the JSON store's ``_quarantine_corrupt`` behaviour: the raw,
+    possibly-recoverable bytes are preserved as ``<file>.corrupt-<epoch_ms>``
+    instead of being clobbered by a repair. Fails **closed** — returns ``None``
+    without copying — when free disk space is below the file size plus a margin,
+    so a corruption event never itself fills the disk. Retention-capped so a
+    boot-loop cannot accumulate unbounded copies.
+    """
+    import shutil
+    import time as _time
+
+    try:
+        size = os.path.getsize(path)
+    except OSError:
+        return None
+    try:
+        free = shutil.disk_usage(os.path.dirname(path) or ".").free
+    except OSError:
+        free = None
+    # Fail closed on low disk: need room for the file plus a small margin.
+    if free is not None and free < size * 2 + (1 << 20):
+        logger.error(
+            "Skipping forensic backup of %s: insufficient free disk space.", path
+        )
+        return None
+
+    base = f"{path}.corrupt-{int(_time.time() * 1000)}"
+    dest = base
+    attempt = 1
+    while os.path.exists(dest):
+        dest = f"{base}-{attempt}"
+        attempt += 1
+    try:
+        shutil.copy2(path, dest)
+    except OSError as exc:
+        logger.error("Forensic backup of %s failed: %s", path, exc)
+        return None
+    # Best-effort sidecar preservation; their absence is not fatal. Each sidecar
+    # is ``path + suffix`` (e.g. ``-wal``), copied next to the primary backup so
+    # a later analysis can replay the write-ahead log.
+    for sidecar in _sidecar_paths(path):
+        if os.path.exists(sidecar):
+            suffix = sidecar[len(path):]
+            try:
+                shutil.copy2(sidecar, dest + suffix)
+            except OSError:
+                pass
+    _prune_forensic_backups(path)
+    return dest
+
+
+def _prune_forensic_backups(path: str) -> None:
+    """Keep only the newest ``_MAX_FORENSIC_BACKUPS`` forensic copies."""
+    directory = os.path.dirname(path) or "."
+    prefix = os.path.basename(path) + ".corrupt-"
+    try:
+        entries = [
+            os.path.join(directory, name)
+            for name in os.listdir(directory)
+            if name.startswith(prefix)
+        ]
+    except OSError:
+        return
+    # Only prune the primary DB copies (no sidecar suffix like ``-wal``).
+    copies = [p for p in entries if not (p.endswith("-wal") or p.endswith("-shm"))]
+    if len(copies) <= _MAX_FORENSIC_BACKUPS:
+        return
+    copies.sort(key=lambda p: os.path.getmtime(p))
+    for stale in copies[: len(copies) - _MAX_FORENSIC_BACKUPS]:
+        # Remove the primary copy together with its own ``-wal``/``-shm``
+        # sidecars so a WAL database's forensic sidecars cannot accumulate past
+        # the retention cap.
+        for target in (stale, stale + "-wal", stale + "-shm"):
+            try:
+                os.remove(target)
+            except OSError:
+                pass
+
+
+def _repair_ledger_path(path: str) -> str:
+    return path + ".repair"
+
+
+def _repair_budget_ok(path: str) -> bool:
+    """Return True while the file still has repair attempts left in its budget.
+
+    A sidecar ledger (``<file>.repair``) records how many repairs have been
+    attempted so a permanently-malformed file is not re-repaired on every boot
+    (bounded persistent retry). Missing/unreadable ledger means a fresh budget.
+    """
+    try:
+        with open(_repair_ledger_path(path), "r", encoding="utf-8") as fh:
+            attempts = int(fh.read().strip() or "0")
+    except (OSError, ValueError):
+        attempts = 0
+    return attempts < _MAX_REPAIR_ATTEMPTS
+
+
+def _record_repair_attempt(path: str) -> None:
+    try:
+        with open(_repair_ledger_path(path), "r", encoding="utf-8") as fh:
+            attempts = int(fh.read().strip() or "0")
+    except (OSError, ValueError):
+        attempts = 0
+    try:
+        with open(_repair_ledger_path(path), "w", encoding="utf-8") as fh:
+            fh.write(str(attempts + 1))
+    except OSError:
+        pass
+
+
+def _clear_repair_ledger(path: str) -> None:
+    for p in (_repair_ledger_path(path),):
+        try:
+            os.remove(p)
+        except OSError:
+            pass
+
+
+def _repair_via_online_backup(path: str) -> bool:
+    """Attempt a least-destructive repair of a malformed database in place.
+
+    Recovers what SQLite's online-backup API can read into a fresh snapshot and
+    promotes it back with an atomic ``os.replace`` so the canonical file is only
+    swapped for a *verified-intact* rebuild — a failed attempt never touches the
+    original bytes (which are already preserved by :func:`_forensic_backup`).
+    Returns True only when the rebuilt file itself passes ``quick_check``.
+    """
+    import sqlite3
+
+    _record_repair_attempt(path)
+    # A per-attempt-unique snapshot name so two concurrent repairs of the same
+    # file never delete or promote each other's in-flight rebuild.
+    snapshot = f"{path}.repair-tmp-{os.getpid()}-{id(path)}"
+    try:
+        os.remove(snapshot)
+    except OSError:
+        pass
+    src = dst = None
+    try:
+        src = sqlite3.connect(path)
+        dst = sqlite3.connect(snapshot)
+        src.backup(dst)  # copies all readable pages into the fresh file
+        dst.commit()
+    except Exception as exc:
+        logger.warning("Online-backup repair of %s failed to read pages: %s", path, exc)
+        for c in (src, dst):
+            try:
+                if c is not None:
+                    c.close()
+            except Exception:
+                pass
+        try:
+            os.remove(snapshot)
+        except OSError:
+            pass
+        return False
+    finally:
+        for c in (src, dst):
+            try:
+                if c is not None:
+                    c.close()
+            except Exception:
+                pass
+
+    # Verify the rebuilt snapshot before promoting it over the canonical file.
+    verify = None
+    try:
+        verify = sqlite3.connect(snapshot)
+        intact = quick_check(verify)
+    except Exception:
+        intact = False
+    finally:
+        try:
+            if verify is not None:
+                verify.close()
+        except Exception:
+            pass
+    if not intact:
+        try:
+            os.remove(snapshot)
+        except OSError:
+            pass
+        return False
+
+    try:
+        # Drop stale sidecars of the malformed file so they cannot be replayed
+        # over the rebuilt database. The malformed original + its sidecars are
+        # already forensically backed up before we get here.
+        for sidecar in _sidecar_paths(path):
+            try:
+                os.remove(sidecar)
+            except OSError:
+                pass
+        os.replace(snapshot, path)
+    except OSError as exc:
+        logger.error("Could not promote repaired snapshot for %s: %s", path, exc)
+        return False
+    _clear_repair_ledger(path)
+    logger.warning("Repaired malformed SQLite database %s via online backup.", path)
+    return True
+
+
+def _quarantine_unrepairable(path: str) -> None:
+    """Move an unrepairable malformed file aside so a fresh DB can be created.
+
+    When the online-backup API cannot rebuild an intact snapshot (e.g. a
+    corrupt header that reads as "file is not a database"), the malformed bytes
+    are already preserved by :func:`_forensic_backup`; renaming the canonical
+    file out of the way lets the store open a fresh, empty database instead of
+    crashing on the first ``CREATE TABLE`` — mirroring the JSON store's
+    quarantine-and-continue behaviour (Issue #5387). Sidecars go too so a stale
+    ``-wal`` cannot be replayed over the new file.
+    """
+    for sidecar in _sidecar_paths(path):
+        try:
+            os.remove(sidecar)
+        except OSError:
+            pass
+    try:
+        os.remove(path)
+    except OSError as exc:
+        logger.error("Could not clear unrepairable malformed file %s: %s", path, exc)
+
+
+def _guard_integrity(path: str, *, repair: bool, backup: bool) -> None:
+    """Detect and (optionally) repair a malformed database before first use.
+
+    Runs a cheap ``quick_check`` on open; on failure it forensically backs up
+    the malformed file (+sidecars) and attempts a bounded, least-destructive
+    online-backup repair. If the file cannot be repaired, its (already backed
+    up) bytes are quarantined so the store can open a fresh database rather than
+    crash on first use. A no-op for ``:memory:``/absent/empty files. A transient
+    lock/busy error is treated as "healthy but busy" and left untouched.
+    """
+    import sqlite3
+
+    if not path or path == ":memory:" or not os.path.exists(path):
+        return
+    try:
+        if os.path.getsize(path) == 0:
+            return
+    except OSError:
+        return
+
+    probe = None
+    try:
+        probe = sqlite3.connect(path)
+        if quick_check(probe):
+            return
+    except Exception as exc:
+        # A transient lock is not corruption — never back up or repair a file
+        # that a peer merely holds open. Leave it exactly as-is.
+        if _is_transient_lock_error(exc):
+            logger.info(
+                "SQLite database %s is locked/busy on probe; skipping guard.", path
+            )
+            return
+    finally:
+        try:
+            if probe is not None:
+                probe.close()
+        except Exception:
+            pass
+
+    logger.error("SQLite database %s failed integrity check; corrupt.", path)
+
+    # Fail closed: if a forensic backup was requested but could not be taken
+    # (low disk, copy error), do NOT proceed to repair — repair's atomic
+    # promotion would otherwise destroy the only copy of the malformed bytes.
+    if backup:
+        if _forensic_backup(path) is None:
+            logger.error(
+                "Forensic backup of %s failed; skipping repair to preserve "
+                "the malformed file for recovery.",
+                path,
+            )
+            return
+
+    if repair and _repair_budget_ok(path):
+        if _repair_via_online_backup(path):
+            return
+        # Repair could not rebuild an intact snapshot. The malformed bytes are
+        # safely backed up, so quarantine the canonical file to let the durable
+        # store open a fresh database instead of crashing on first query.
+        _quarantine_unrepairable(path)
+
+
+def connect(
+    path,
+    *,
+    check_same_thread: bool = False,
+    isolation_level: Optional[str] = "",
+    busy_timeout_ms: int = 5000,
+    synchronous: str = "FULL",
+    guard: bool = False,
+    repair: bool = True,
+    backup: bool = True,
+    **kwargs,
+):
+    """Open a hardened SQLite connection used by core durable stores.
+
+    Applies :func:`apply_wal_with_fallback` so the connection uses WAL where the
+    filesystem supports it and DELETE where it does not, with a sensible
+    ``busy_timeout``. All keyword arguments other than the journal handling are
+    forwarded to :func:`sqlite3.connect`.
+
+    Args:
+        path: Database path (``":memory:"`` supported).
+        check_same_thread: Forwarded to ``sqlite3.connect`` (defaults False so
+            stores may share a connection across threads under their own lock).
+        isolation_level: Forwarded to ``sqlite3.connect``. Defaults to ``""``
+            (sqlite3's default autocommit-off); pass ``None`` for autocommit.
+        busy_timeout_ms: ``PRAGMA busy_timeout`` in milliseconds.
+        synchronous: ``PRAGMA synchronous`` level. Defaults to ``FULL`` so
+            committed writes survive an OS/power crash (matches pre-hardening
+            durability). Pass ``"NORMAL"`` for higher throughput under WAL.
+        guard: When True, run a one-time ``quick_check`` on open and, on a
+            malformed file, forensically back it up and attempt a bounded
+            least-destructive online-backup repair before connecting (Issue
+            #5387). Off by default so existing callers are unchanged.
+        repair: Attempt an online-backup repair on corruption (only when
+            ``guard`` is set). Bounded by a sidecar attempt-ledger.
+        backup: Take a retention-capped, disk-space-aware forensic backup of the
+            malformed file before repair (only when ``guard`` is set).
+        **kwargs: Forwarded to ``sqlite3.connect``.
+    """
+    import sqlite3  # lazy import — stdlib, no heavy dependency
+
+    db_path = path
+    if isinstance(path, (str, os.PathLike)) and str(path) != ":memory:":
+        db_path = str(path)
+
+    if guard:
+        try:
+            _guard_integrity(str(db_path), repair=repair, backup=backup)
+        except Exception as exc:  # detection/repair must never block open
+            logger.debug("Integrity guard skipped for %s: %s", db_path, exc)
+
+    connect_kwargs = dict(kwargs)
+    connect_kwargs["check_same_thread"] = check_same_thread
+    if isolation_level != "":
+        connect_kwargs["isolation_level"] = isolation_level
+
+    conn = sqlite3.connect(db_path, **connect_kwargs)
+    try:
+        apply_wal_with_fallback(
+            conn,
+            str(db_path),
+            busy_timeout_ms=busy_timeout_ms,
+            synchronous=synchronous,
+        )
+    except Exception as exc:  # never let hardening break connection creation
+        logger.debug("Journal-mode hardening skipped for %s: %s", db_path, exc)
+    return conn