dify/knowledge-fs/docs/production-deployment.md

KnowledgeFS Production Deployment

KnowledgeFS has one supported production topology: an internal backend service deployed as part of Dify. It has no independent SaaS, private-cloud, or single-host deployment mode.

The existing Dify knowledge-base feature and KnowledgeFS intentionally coexist during rollout. This deployment does not migrate, replace, or delete existing Dataset/Document data.

Dependency ownership

Capability	Owner	KnowledgeFS access
Model configuration and credentials	Dify model manager / Plugin Daemon	Dify inner model API
Datasource configuration, OAuth, and credentials	Dify datasource plugins	Dify inner datasource API
Physical object storage	Dify STORAGE_TYPE implementation	Dify inner storage API
KnowledgeFS relational state	KnowledgeFS database	DATABASE_URL
Complex document parsing	Unstructured-compatible service	UNSTRUCTURED_API_URL
PDF image rasterization	KnowledgeFS API image	Bundled Poppler pdftoppm executable
Capability signing	Dify	Public JWKS only in KnowledgeFS

KnowledgeFS must never receive model-provider keys, datasource secrets, direct Plugin Daemon credentials, or object-storage provider credentials.

Dify Compose

The canonical service definitions are:

• docker/docker-compose.yaml
• docker/docker-compose-template.yaml
• docker/envs/core-services/knowledge-fs.env.example

The Compose service:

• starts by default with the rest of Dify;
• builds knowledge-fs/apps/api/Dockerfile when a prebuilt image is unavailable;
• remains on the internal default network and exposes only port 8787 to peer services;
• receives DIFY_INNER_API_URL=http://api:5001;
• receives the same inner API key used by Dify's plugin boundary;
• waits for the Dify API and its database dependency;
• uses /health for liveness and /ready for traffic readiness.

KNOWLEDGE_INTEGRATED_MODE_ENABLED controls Workspace provisioning/cutover behavior only. Whether it is false or true, model, datasource, and object-storage calls always go through Dify.

Operator-owned environment

The tracked KnowledgeFS environment example intentionally contains only settings that belong to the service:

Variable	Purpose
DATABASE_URL	KnowledgeFS PostgreSQL connection string.
KNOWLEDGE_DOCUMENT_COMPILATION_RUNTIME	Durable document worker rollout.
KNOWLEDGE_PDF_RASTERIZER	PDF image rasterizer. The production image defaults to poppler; set off as a kill switch.
KNOWLEDGE_PDF_RASTERIZER_DPI	Main PDF image resolution; the bounded deployment default is 144.
KNOWLEDGE_PDF_RASTERIZER_THUMBNAIL_DPI	Thumbnail resolution; the bounded deployment default is 48.
KNOWLEDGE_PDF_RASTERIZER_TIMEOUT_MS	Poppler subprocess timeout; the deployment default is 30000.
KNOWLEDGE_PDF_RASTERIZER_MAX_ASSETS	Maximum PDF assets rasterized for one document; the deployment default is 500.
KNOWLEDGE_PDF_RASTERIZER_MAX_CONCURRENCY	Maximum concurrent Poppler page batches per API replica; defaults to 2 and accepts 1..8.
KNOWLEDGE_FS_CAPABILITY_V2_ENABLED	Capability-v2 verifier rollout.
KNOWLEDGE_FS_CAPABILITY_V2_PUBLIC_JWKS	Public verification key set issued by Dify.
KNOWLEDGE_QUERY_IMAGE_RETRIEVAL_ENABLED	Opt in to query-image visual retrieval; requires an enabled visual-embedding provider/index and a query mode other than off.
KNOWLEDGE_QUERY_IMAGE_EXPANSION_TIMEOUT_MS	Timeout for the single Deep/Research vision expansion call; defaults to 8000 ms.
UNSTRUCTURED_API_URL	Parser endpoint for complex formats.
UNSTRUCTURED_API_KEY	Optional parser authentication.
UNSTRUCTURED_MAX_CONCURRENCY	Process-wide parser request limit; defaults to 2.
UNSTRUCTURED_REQUEST_TIMEOUT_MS	Total timeout for one parser request and response body; defaults to 120000.
UNSTRUCTURED_MAX_RESPONSE_BYTES	Maximum parser response body; defaults to 33554432 (32 MiB).

Compose injects DIFY_INNER_API_URL and DIFY_INNER_API_KEY; do not duplicate them in the operator-owned env file. Do not add MINIO_*, cloud object-storage credentials, provider API keys, PLUGIN_DAEMON_*, datasource tokens, or OAuth client secrets.

DIFY_OBJECT_STORAGE_REQUEST_TIMEOUT_MS bounds each authenticated inner object-storage request, including response consumption. It defaults to 60000; transport failures and 408/409/425/429/5xx responses remain retryable at the durable compilation layer.

PDF image rasterization

The production API image installs Poppler and verifies pdftoppm during the image build. Its image defaults enable rasterization at 144 DPI, generate 48 DPI thumbnails, stop an individual Poppler operation after 30 seconds, cap one document at 500 rasterized assets, and run at most two Poppler page batches concurrently per API replica. The process still runs as the unprivileged node user.

The Dify Compose service keeps canonical values in docker/envs/core-services/knowledge-fs.env, where they override the image defaults. It maps only whitelisted DIFY_ROOT_*_OVERRIDE proxies, so an explicitly set PDF rasterizer value in docker/.env takes precedence without exposing the rest of the root environment. An unset or empty root value leaves the service env (or image default when that file is absent) in control. Set KNOWLEDGE_PDF_RASTERIZER=off in either operator env during an incident or on a deliberately constrained deployment. Do not set KNOWLEDGE_PDF_RASTERIZER_COMMAND for the published image; its bundled command is on PATH.

Rasterization supplies durable image objects for PDF image elements when the parser returns layout coordinates without image bytes. It does not repair already-published parse artifacts. Re-run the document ingestion after deploying the corrected image to repopulate images that were previously stored without an asset reference.

Image-query rollout

The public Dify API accepts query images as actor-owned Dify UploadFile references. KnowledgeFS does not receive storage credentials or persist a second copy of the bytes: Dify validates tenant and account ownership, MIME, size, count, and aggregate size, then KnowledgeFS resolves each file through the authenticated inner API for the lifetime of one query run.

Image-to-visual retrieval is independently disabled by default. Enable KNOWLEDGE_QUERY_IMAGE_RETRIEVAL_ENABLED=true only when all of the existing KNOWLEDGE_VISUAL_EMBEDDING_* settings select the same multimodal embedding space used to build the published visual_vector projections, and KNOWLEDGE_VISUAL_EMBEDDING_QUERY_MODE is fallback or primary. An explicit query mode of off remains authoritative.

Fast performs no vision-LLM expansion. Deep and Research perform at most one bounded image-to-text expansion through Dify's selected reasoning model; durable Research persists the derived text so retry/replay does not repeat that call. Research then uses the derived text for document selection and level-by-level PageIndex navigation before final synthesis. Model calls are included in the Research dry-run estimate and durable budget accounting.

The request bounds are four images, 10 MiB per image, 32 MiB in aggregate, with MIME restricted to PNG, JPEG, WebP, and GIF. Operational traces and terminal metadata use these stable degradation reasons: query-image-visual-leg-unavailable, query-image-ignored-no-vision-model, and query-image-expansion-timeout.

Database release

Apply checked-in KnowledgeFS migrations through the controlled migration runner before scaling a new binary:

pnpm db:migrations:check
pnpm local:db:migrate

Use the environment's normal migration job in production rather than running the local command from an application container. The KnowledgeFS migration runner owns only KnowledgeFS tables. It must not mutate existing Dify Dataset/Document tables or perform a production data migration.

Keep destructive legacy-removal flags disabled until the separately approved zero-traffic, backup/restore, DBA, and CAB gates are complete.

Readiness contract

Production /ready fails closed unless all enabled capabilities are assembled. The base checks include:

• an authentication verifier;
• Dify model-runtime configuration;
• Dify datasource-runtime configuration;
• Dify object-storage configuration;
• durable database repositories required by enabled workers and product routes.

/health is liveness and component diagnostics; it is not permission to receive production traffic. A service with /health=200 and /ready=503 must remain out of rotation.

Direct upload remains disabled because the Dify storage bridge deliberately does not expose provider-specific presign or multipart primitives. Upload bytes pass through the bounded KnowledgeFS API and Dify inner storage API.

Release validation

Before publishing an image:

pnpm install --frozen-lockfile
pnpm typecheck
pnpm test
pnpm lint:backend
pnpm openapi:export:test
pnpm db:migrations:check
pnpm dify:compose:config
pnpm docker:api:build
pnpm docker:api:bundle-smoke
git diff --check

The isolated bundle smoke is not a production dependency test. In a Dify-connected environment, also verify:

1. /health and /ready.
2. A tenant-scoped KnowledgeSpace create/read.
3. A bounded document upload and object read through Dify storage.
4. Embedding, rerank, LLM, and model-catalog calls through Dify model instances.
5. Datasource validation/browse through a Dify-managed credentialId.
6. No model, datasource, OAuth, Plugin Daemon, or storage credentials appear in KnowledgeFS environment variables, requests, logs, or database rows.
7. Existing Dify knowledge-base flows remain unchanged.

Workspace rollout

Roll out Workspace by Workspace. Keep the integrated-mode/capability flags disabled by default, then enable only after the selected Workspace has:

• durable KnowledgeFS provisioning state;
• capability verification;
• successful model, datasource, and storage smoke;
• rollback evidence and monitoring ownership.

The rollout flag changes admission and provisioning behavior. It does not switch transports and does not authorize a fallback runtime.

Kubernetes

infra/kubernetes/dify-integration-baseline.yaml is an inert reference with zero replicas, an internal ClusterIP, fail-closed probes, and no public ingress. A downstream Dify deployment may adopt it only while preserving the same ownership boundaries.

Rollback

Rollback the KnowledgeFS image or disable the affected Workspace cutover. Preserve KnowledgeFS database rows and Dify-owned objects unless a reviewed recovery procedure says otherwise. Do not rotate the Dify inner key, delete existing knowledge-base data, or redirect KnowledgeFS to a direct storage/plugin endpoint as a rollback shortcut.

After rollback, rerun Dify-connected health and tenant smoke, confirm the existing knowledge-base feature is unaffected, and record the release and rollback evidence.