Files
2e35730637 feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Slack connectors turn authenticated messages into company work.
> - Direct callbacks need an awake tenant.
> - Cloud can retain authenticated asynchronous events until the tenant
wakes.
> - Replay must preserve the original callback proof and current
endpoint authority.
> - Configuration changes must retain registration responsibility
through failures.
> - This PR enrolls verified Slack identities and drains sealed events
through the normal SDK owner.

## Linked Issues or Issue Description

**Subsystem affected**

Server chat connectors and signed Cloud connector delivery.

**Problem or motivation**

Direct Slack Events API delivery depends on an awake tenant. Broker
outages, lost responses and configuration changes must preserve delivery
and registration responsibility. Enrollment failures must also leave
other connector events able to drain.

**Proposed solution**

Stage nonsecret enrollment or retirement intent inside the local
configuration transaction. Let the existing connection-event owner
perform broker requests and retries. Replay sealed events through the
current Slack SDK with their original authentication evidence.

**Alternatives considered**

An early acknowledgment for every callback changes command and modal
semantics. A second delivery ledger duplicates existing ownership.
Reusing a new stack identity to retire old registrations crosses an
authorization boundary.

A companion Cloud broker change is required. Related setup fixes: Refs
#18837, Refs #18878. Kickoff searches found no competing async
enrollment consumer. The ROADMAP's connector setup and access controls
stay in place.

## What Changed

- Persist enrollment and credential-free retirement intent before local
configuration commits. Broker failure does not make a completed pause or
removal look unsuccessful.
- Bind company, endpoint, connection, public ID, generation and
authorized actor. Preserve lost-response responsibility and recover only
the supported same-instance configuration outage.
- Permit an explicit move to a new authorized stack only after confirmed
old retirement. Keep the old identity until the new broker receipt
matches the current intent and generation.
- Show permanent signing-credential failures as attention. Preserve
transient retries and reject uncertain inbound authentication before SDK
admission.
- Drain events and acknowledgments before enrollment reconciliation.
Persist a bounded retry on the exact busy intent so other due endpoints
can enroll or retire. Preserve current custody under concurrent changes.
- Stage retirement in the Apps removal transaction too. Archive the
endpoint and connection, retain the original subject and instance, then
revoke credentials. Retirement does not need the deleted Slack
credential.
- Acknowledge a malformed sealed Slack receipt as invalid once. Keep
actual broker outages on the retry path.
- Run the real configuration, SDK ingress, generation fences and public
Apps removal paths in the default SQL suite. Keep Vitest transport
helpers outside production services.
- Preserve original callback bytes, signature, time and public URL
evidence. Use the existing transactional chat-delivery ledger for
redelivery.
- Provide a checked-in prewire guard and launcher using the current Node
executable. Add broker, worker, SDK, SQL and controlled-process
acceptance.
- Stabilize AI account summaries by ID and grant ID. This fixes a CI
migration-repeat equality failure without changing SQL, authorization or
reconnect behavior.
- Open the four configured database connections before the enrollment
test freezes its clock. Keep all 35 original assertions and all
timeouts.
- Measure status-card dispatch entry before its mock does database work.
Keep the original strict 500 ms bound and check any still-armed earlier
deadline. Bound that fixture's pool to four connections.

## Verification

- Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges
preserve master `47027e7` and all owned changes. A read-only merge
assessment against fetched master `cb5d7e44` is conflict-free. The only
overlapping file has separate AI test additions. The branch has not
merged that later master; fresh remote checks must validate the current
PR test tree.
- At this head, normal server production typechecking and declaration
emission passed in 35.24s. The affected actual CI test shard passed in
6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard
controls retain their source-matched results.
- All **15 default real-PostgreSQL cases passed in 34.66s**, with no
skips or filtering. They cover normal configure and SDK ingress,
rollback, stale generations, malformed receipts, busy-lease fairness,
concurrent intent custody, unknown errors, and public Apps removal.
Removal tests check actual credential revocation, retained retirement
through outage and repeat removal, one ordinary retry and confirmed
retirement, transaction rollback and foreign-company refusal.
- The public removal counterfactual failed before the fix: archival and
credential revocation passed, but the inactive intent was absent. The
same test source passed after the staging call. Separate counterfactuals
also reproduced busy-endpoint starvation and malformed-receipt retries.
Original failures remain retained.
- The paired broker/SDK/process journey passed in 31.98s at ancestor
`0324c011`. It preserves callback proof, lost-ACK redelivery, exactly
one comment, one delivery and one successful controlled process. Its
test, producer, SDK, guard and runtime bytes are unchanged. The new
public Apps archive branch is not called by that journey; its evidence
carries on that checked basis.
- The new SQL cases need no companion Cloud checkout. Normal CI runs
them with the repository fixture. For an existing dedicated PostgreSQL
fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node
server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional
paired launcher still requires the normally built companion Cloud
checkout.
- Every pool was bounded to four or fewer connections. The final 15-case
run sampled three runtime clients plus one observer. The independent
final audit found zero clients and all five recorded processes absent.
Earlier closing-socket observations are kept separate from final
cleanup.
- Earlier unrelated CI failures are documented: [enrollment
timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754)
and [status-card observation
bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843).
The enrollment test retains all 35 assertions and its original timeouts.
A controlled connection probe established the frozen-clock mechanism;
the original remote access history remains unknown. The status-card
fixture measures dispatch entry with the original strict 500ms bound. A
real delayed-dispatch control fails that bound.
- Full local workspace typecheck, build and test inventory was not run
because its embedded-cluster and wider resource paths exceed this local
lease. Exact-head remote CI now passes: 54 successful checks, three
skips and Ponytail's neutral check, with aggregate verification green
and the legacy security context independently confirmed successful. Both
complete bot reviews are 5/5, with all 13 paginated review threads
resolved. Seven execution jobs originally reported runner shutdown; one
diagnostic rerun recovered them and the dependent aggregate check.
Original logs remain retained, and the underlying shutdown cause is
unknown. This is recovery evidence, not a product-fix claim.
- Provider responses are controlled. This proves local SQL, broker, SDK
and process behavior. It does not prove live Slack, agent tools,
synchronous callbacks or deployed suspension.

## Risks

The companion broker is required. Outage or conflicting authority can
leave explicit pending or attention responsibility. A move between
stacks requires confirmation from the old authorized owner first.
Synchronous commands, actions, modal validation, options and challenges
retain their deadlines and endpoint blocker. This slice grants no full
Slack endpoint sleep exemption. No deployed suspension or savings claim
is made.

## Model Used

OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository
tools and code execution. The served backend revision and context limit
are not exposed. A GPT-6-family coordinator performed technical review
and integration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <[email protected]>
Co-authored-by: Codex <[email protected]>
2026-11-06 15:22:51 -06:00

18 KiB

Slack app ownership and setup

Slack has one chat runtime and two provisioning paths. A saved connection's slackSetupMethod selects provisioning; it does not change tools, scopes, conversation routing, people permissions, or channel access.

Method App creation and installation Default
managed A personal Slack manager grant creates a child app; managed installation supplies its bot token. Slack consent/admin approval remains a fallback. New drafts on qualified Cloud deployments only
automatic A temporary app configuration access token creates a customer-owned app; the user approves its OAuth installation. New self-hosted drafts and Cloud without managed availability
manual User creates an app from the canonical manifest and enters credentials. Legacy drafts with no saved method
existing User supplies credentials for an existing app. Explicit selection

Existing drafts and active bots retain their method. Enabling managed setup never adopts existing apps. Choose agent offers Use your own app under Advanced, below Slash command, when the instance supports managed setup and creation has not started. The own-app flow retains Create manually and Use an existing app under Advanced. After creation dispatch, a managed draft cannot switch paths.

Managed setup has two visible stops: Choose agent and Connect. Continue starts manager authorization or installation in the saved workspace automatically. Multiple workspaces still require a selection. Recovery actions remain explicit; opening a failed or uncertain registration never retries creation automatically. Creation, installation, message waiting, and success use the same production components and server evidence as the own-app flow. A click or OAuth return does not constitute successful configuration.

Reopening a completed Slack setup retains its Slack handoff. Done opens the saved connection settings without repeating activation or reactivating a paused bot. Unfinished setups still complete their normal server verification first.

Cloud deployment prerequisites

Managed setup is disabled by default. Qualification requires all of:

  • The instance chat-connector rollout gate and connection-management permission.
  • Verified Cloud runtime identity, connector broker signing/sealing configuration, and valid independently configured public board and webhook HTTPS origins.
  • PAPERCLIP_SLACK_MANAGED_SETUP_ENABLED=true on the approved tenant deployment.
  • The companion Cloud broker deployment with the explicitly enabled slack.manager profile, approved Slack manager app, token rotation, and its client secret in the broker's secret store. No manager client secret belongs in the tenant.

The broker must advertise its manager app identity as well as the enabled profile. Previously saved grants from a different manager app are not reused. A broker without this capability leaves managed setup unavailable. Do not enable the tenant gate until real Slack qualification has passed; fixture coverage does not prove Marketplace eligibility or a workspace's approval policy.

Self-hosted instances do not need a Paperclip account, broker enrollment, manager credentials, or hosted-service calls. They continue using the own-app flow.

Personal workspace authorization

Manager grants belong to a company and one Paperclip user, and bind a verified Slack user, workspace, and manager app. Another company member must authorize their own account, even for the same workspace. Access and rotating refresh tokens are user-scoped vault entries. Refreshes serialize across connections, persist the new refresh token, and require fresh authorization after an ambiguous exchange. Managed provisioning resolves the submitted grant from the Slack manager-grant registry. Both provisioning and revocation check its company and current user; these IDs are separate from tool-connection grants.

Revoking a grant requests provider revocation, removes its local credentials, invalidates pending manager authorization, and prevents future provisioning with it. The local activity records whether provider revocation succeeded. Provider errors never restore local access. It does not uninstall apps or disable already-installed bots. Removing a bot deletes endpoint-specific secrets and pending setup attempts; a reusable personal manager grant survives. Provider-side app removal remains explicit through the saved Slack management URL.

Recovery

  • A dispatched creation with an unknown result is uncertain. Never automatically create again. The user must inspect Slack and explicitly confirm that no app was created before a new request identity is accepted.
  • Denied or pending installation retains the app. Resume with the same grant or complete the child app's state-bound OAuth approval; the installer must match the grant's Slack identity before personal linking.
  • Returned bot credentials are vaulted before inventory or runtime configuration. Retry connecting reuses those credentials after network or inventory failures. Rejected or under-scoped bot credentials instead reinstall the saved app; they never trigger another app creation.
  • Manifest updates finish before managed installation. Changes requiring updated permissions therefore pass through installation again before activation.
  • Expired or revoked manager grants require reconnection. Losing managed availability pauses setup; it never changes the saved ownership method. A saved draft that needs workspace reauthorization shows Reconnect Slack workspace on Connect, even before its app has been created. This button opens Slack authorization and returns to the same saved agent setup.
  • Slack rate limits are grouped by manager/workspace/method. Provider Retry-After is persisted across matching grants in the company, survives restarts, and blocks early retries. Other tenant instances must observe Slack's limits independently; no background loop retries app creation.

All OAuth attempts bind company, actor/session, endpoint, registration revision, and configured origins, expire after ten minutes, and are claimed once before exchange. HTTP diagnostics redact manager tokens, refresh tokens, codes, and sealed handoffs. Local activity records contain only IDs and safe outcome codes. This adds no first-party telemetry.

Code and verification

Server provisioning lives in server/src/services/connectors/slack/setup/; chat-slack-registration.ts remains the compatibility facade. Shared completion owns credential binding, personal linking, welcome messages, and verification. Removal cleanup stays independently callable. Slack wizard components and stage selection live in ui/src/pages/apps/chat/slack/.

Inbound workspace checks and authenticated setup evidence resolve only the current signing secret. They do not read bot or app tokens for signature checks. Runtime setup still resolves its full credential set. Each check uses the saved company and connection, current secret version, and normal secret access audit path.

Production Storybook journeys: Connections / Slack / Managed setup and Connections / Slack / Automatic setup (the persisted own-app automatic path). Server integration cases cover personal grants, duplicate requests, uncertain creation, approval fallback, refresh, revocation, removal, saved-credential recovery, rate limits, and secret canaries. Browser cases are in tests/e2e/chat-adapters-ui-providers.spec.ts; companion broker tests cover manager user OAuth and the authenticated tenant handoff.

Before enabling a deployment, create two test agents in the authorized workspace. The second must reuse the personal grant without another configuration token or manager authorization. Verify avatars, identity linking, signed inbound events, and same-thread replies. Record real approval behavior separately from fixtures.

Managed setup during broker outages

Cloud deployments configured for managed Slack keep the managed default when the broker profile is temporarily unavailable. New drafts remain on Connect and offer Retry. Use your own app remains available under Advanced on Choose agent before creation. Discovery loading or failure never selects the configuration-token page. Self-hosted instances without managed configuration still use their independent own-app flow; saved draft methods do not change automatically. Provisioning continues to require current broker availability.

Local managed-bot qualification

Maintainers with an approved Slack manager app can exercise the real managed journey from an authenticated local test drive exposed through HTTPS (including TryCloudflare). This development transport talks directly to Slack and reuses the production manager-grant vault, state claims, provisioning, completion, and runtime. It does not exercise the Cloud broker's sealed handoff; qualify that separately on staging afterward.

Set NODE_ENV=development, PAPERCLIP_SLACK_MANAGED_SETUP_ENABLED=true, and PAPERCLIP_SLACK_MANAGER_DEVELOPMENT_CONFIG to an absolute path outside the checkout containing a mode-0600 JSON file with appId, clientId, and clientSecret. Use only an authorized qualification manager app. No credentials are shipped with Paperclip. Production ignores this development configuration, and Cloud runtime identity continues to select the Cloud broker.

Configure the local instance for authenticated public access, with its public URL and chat webhook URL pointing to the HTTPS tunnel. Add the exact public URL plus /api/chat-slack/managed/oauth/callback to the manager app's OAuth redirect URLs, preserving any existing redirects. Start a new Slack setup: saved own-app drafts retain their selected method. Authorize the manager app normally; do not copy user grants from another instance. The second bot should reuse the same local user's vaulted grant. A changed tunnel requires updating public origins and the manager redirect before starting fresh authorization.

Remove the temporary redirect when retiring the test drive and securely remove its development credential file. Removing a bot does not revoke a manager grant used by other bots.

Test messages from Settings

Send me a test message sends a DM to the signed-in connection manager, using their linked identity in this workspace. If this bot is not linked yet, Paperclip can reuse that user's verified manager grant or another bot's linked identity in the same company and workspace. Otherwise, the user selects their Slack account and receives a private connection invitation in Slack. They must open that invitation, sign in, and confirm before Paperclip links the account and sends the test DM. The account picker loads further workspace pages on request. Existing revoked or conflicting identity links are not overwritten. Direct messages must be enabled, and the user must be an active company operator.

The action records a durable request before sending. Repeating the same request returns its saved result; an uncertain delivery is never automatically resent. Open Slack configuration opens the saved app's OAuth & Permissions page. Legacy connections without an app ID open Slack's app list instead. Open Slack remains available after automatic setup completion and on the saved connection page. It opens the current user's bot DM when its workspace and channel are known and its linked owner matches the signed-in user. Another manager or an unknown identity uses the saved provider link or Slack instead.

An explicit managed reconnect reinstalls the saved app and verifies the replacement bot credentials. Its request ID is saved before dispatch, so a duplicate completed request does not reinstall again. Credential-configuration recovery still resumes the saved token after a downstream failure. No reconnect creates another Slack app.

Durable Cloud Events API delivery

A broker that advertises slackEvents.version=1 can accept a signed Events API registration after ordinary configuration settles. The producer uses the saved child app ID, current workspace, signing key, endpoint/connection and runtime generation. It retains the existing public callback URL. Missing verified app identity or broker support leaves the existing setup in place; it does not guess an identity or substitute the Slack manager's signing key.

Cloud stores the child signing key through its existing encrypted stack vault. Registration metadata and receipts contain no plaintext credentials. An unchanged credential/endpoint identity reuses its immutable vault ref across generations; rotation and current registration authority remain separately fenced. Removal retires exact queue membership and prevents an old generation from reenrolling. The local configuration transaction first saves nonsecret enrollment or retirement intent and signals the existing connection-events owner. Broker work runs through that owner after commit; configuration APIs do not await an optional Cloud request. Configure, pause and removal can complete locally while Cloud is unavailable. The owner retries only retained due intent and reconstructs it after restart, including removal for archived endpoints after credential cleanup. Confirmation replaces only the same current intent and generation; a completed remote call with a lost response remains pending. Current authorization errors retain visible attention debt instead of claiming registration succeeded. Event draining and acknowledgments run before enrollment reconciliation. The owner then admits one due enrollment operation and schedules the earliest retained responsibility. A busy credential lease or enrollment failure retains a bounded retry without aborting that event drain. Rollback leaves no committed intent to run.

Retirement sends the original responsible subject and the exact instance, company, endpoint, public ID, connection and monotonic generation. Cloud can publish a nonreceiving generation floor without a signing key, live descriptor or previously returned binding ID. This fences delayed enrollment after local removal. Normal current broker authority still applies; the subject alone does not authorize an operation. An instance that never enrolled with Cloud creates no new intent or background retry when Cloud is absent. Previously configured retirement responsibility retains its original instance identity as visible attention if that identity becomes unavailable. Unsupported capability or disabled new enrollment retains bounded retry responsibility; a closed generation or identity conflict requires attention. Current authorized confirmation is required before clearing the exact pending intent.

Ordinary redeploys retain the same stack identity. To move enrollment to another stack, first confirm retirement under the old stack's authority, with no pending old intent. Then explicitly reconfigure under the new authorized stack at a newer runtime generation. The old stored identity remains until the new broker receipt matches the current intent and generation. An environment change cannot transfer live responsibility. Unavailable old authority remains visible attention.

A missing or disabled signing credential also requires attention. Enrollment does not retry it as a broker outage. Incoming receipts retain typed attention before SDK admission; uncertain authentication does not acknowledge the receipt.

Only authenticated event_callback messages use this durable lane. The sealed queue retains original bytes, signature, timestamp and acceptance time. The app checks current endpoint/generation authority before SDK handling. It verifies original freshness at Cloud acceptance; ordinary external requests still use current time. The credential snapshot lease releases before normal lifecycle callbacks reacquire their own leases.

Private replay provenance prevents internal replay URLs from replacing saved external callback verification evidence. The normal transactional chat delivery ledger remains the message admission authority. An ambiguous callback or ACK failure retains its receipt; malformed receipts rejected before useful admission can be explicitly ignored with bounded diagnostics. Credential-authentication uncertainty requires attention and is not silently acknowledged. The broker's existing retention horizon remains the disclosed recovery boundary.

This does not qualify a complete Slack endpoint for sleep. Slash commands, interactive actions, modal validation and challenges still need their full synchronous response semantics. The endpoint blocker remains. Qualification must cover normal signed enrollment, durable retry after removal, sealed drain, the actual event worker and SDK, lost-ACK redelivery and one domain effect, generation changes, and callback evidence. Synthetic protocol coverage does not prove live Slack, complete configuration UI or a deployed stop/restart/wake journey.

Reproduce the paired Cloud acceptance

Use the repository's supported Node version and installed dependencies. Compile the companion Cloud checkout first. Set PAPERCLIP_SLACK_TEST_DATABASE_URL and SLACK_EVENT_TEST_DATABASE_URL to two distinct, owned loopback PostgreSQL *_test databases. Set SLACK_TEST_CLOUD_DIST to its absolute compiled dist directory. This test applies actual migrations and uses only those two databases. It does not create a PostgreSQL cluster or use a live Slack provider.

From the app repository root, run:

node server/scripts/test-slack-cloud-postgres.mjs

The launcher uses the current Node executable and one Vitest worker. It loads the checked-in prewire guard in that actual worker before client construction. The controlled process child explicitly loads the same guard before its script. The guard rejects off-host and unowned socket targets before transport. Caught unexpected attempts still fail the fixture. No private preload path or manual SLACK_GATE_GUARD_PID setting is needed. The fixture remains opt-in during the normal test suite; its default owner and guard regressions run without databases.