langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #28986] feat: Add Aliyun SLS (Simple Log Service) integration for workflow execution logging #32266

New Issue
Closed
opened 2026-02-21 20:51:04 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:51:04 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/28986

State: closed
Merged: Yes

Summary

This PR implements Aliyun Simple Log Service (SLS) integration as an alternative storage backend for workflow execution logs, addressing the challenge described in #27117 where PostgreSQL-based execution records consume significant database space and create stability/performance risks.

🚀 Motivation: Why Move Logs to Aliyun SLS?

As Dify deployments scale, storing high-volume workflow runtime logs in the primary PostgreSQL database presents significant architectural challenges:

  • Performance Bottlenecks: Heavy write throughput from workflow executions consumes valuable database connections and IOPS, often competing with critical business transactions (e.g., app configuration, knowledge base indexing).
  • Scalability Limits: Rapidly growing log tables degrade query performance over time, causing issues like increased latency for the "first token" and necessitating frequent manual maintenance (vacuuming, partitioning).
  • Limited Observability: The relational database model is ill-suited for the multi-dimensional, full-text analysis required for deep debugging and LLMOps auditing.

✨ Key Benefits of SLS Integration

This PR introduces Aliyun Log Service (SLS) as a specialized, serverless storage backend for runtime logs, offering a robust Storage-Compute Decoupling solution:

  • 🛡️ Stability & Isolation
    Offloads the write pressure of massive execution logs from PostgreSQL to SLS. This Read/Write Separation ensures that spikes in AI inference traffic do not exhaust DB connection pools or lock resources, effectively protecting the stability of the core Dify application.

  • 📊 Advanced LLMOps Analytics
    Unlocks native, real-time SQL analysis capabilities. Unlike simple keyword searches in DBs, SLS allows users to build custom Dashboards for:

    • Cost Analysis: Track token usage by model or user.
    • Performance Profiling: Monitor P95/P99 latency of workflow nodes to identify bottlenecks.
    • Alerting: Set up automatic alerts for abnormal failure rates.

  • ⚡ Zero-Latency Impact
    Implements an asynchronous batching mechanism. Log data is buffered and pushed to SLS in the background via a worker, ensuring that the logging process adds zero blocking latency to the main workflow inference path.

  • 📉 Cost & Lifecycle Efficiency
    Leverages a storage engine optimized for append-only logs with lower costs than SSD-backed databases. Native support for TTL (Time-To-Live) and cold storage archiving eliminates the need for manual data cleanup scripts.

Key Features:

1. Aliyun SLS LogStore Integration

  • New extension module (ext_logstore) that initializes SLS projects, logstores, and indexes on application startup
  • Automatic index schema synchronization from SQLAlchemy models, ensuring consistency between database schema and LogStore indexes
  • Support for storing workflow execution and node execution logs in Aliyun SLS
  • Idempotent initialization that safely handles existing projects and logstores

2. Dual-Write Support for Safe Migration

  • Configurable dual-write mode (LOGSTORE_DUAL_WRITE_ENABLED) that writes to both SLS and SQL database simultaneously
  • Enables safe migration from SQL-based logging to SLS without data loss

3. Repository Pattern Implementation

  • Four new repository classes implementing the workflow execution repository interface:
    • LogstoreWorkflowExecutionRepository - for workflow runs
    • LogstoreWorkflowNodeExecutionRepository - for workflow node executions
    • LogstoreApiWorkflowRunRepository - for API workflow runs
    • LogstoreApiWorkflowNodeExecutionRepository - for API workflow node executions
  • Automatic type mapping from SQLAlchemy column types to LogStore index types (text, long, double, json)
  • Full compatibility with existing domain models (WorkflowExecution & WorkflowNodeExecution)

4. Production-Ready Configuration

  • Environment variables for Aliyun SLS authentication and configuration
  • Configurable TTL (Time-To-Live) for log retention with automatic cleanup
  • Debug logging support when SQLALCHEMY_ECHO=true

Configuration Example:

# Enable Aliyun SLS LogStore
ALIYUN_SLS_ACCESS_KEY_ID=your_access_key_id
ALIYUN_SLS_ACCESS_KEY_SECRET=your_access_key_secret
ALIYUN_SLS_ENDPOINT=cn-hangzhou.log.aliyuncs.com
ALIYUN_SLS_REGION=cn-hangzhou
ALIYUN_SLS_PROJECT_NAME=dify-workflow-logs

# Optional: Configure TTL and dual-write behavior
ALIYUN_SLS_LOGSTORE_TTL=365  # default: 365 days, 3650 for permanent storage
LOGSTORE_DUAL_READ_ENABLED=false  # Set to true during migration phase

Architecture Notes:

This implementation follows the repository pattern established in #23966 and aligns with the domain model architecture improvements referenced in #19850 and #19428. The LogStore repositories implement the same interfaces as the existing SQLAlchemy repositories, ensuring seamless integration with the workflow execution system.

Testing Notes:

  • ✅ All backend linting checks passed (make lint)
  • ✅ Type checking passed with BasedPyright (make type-check)
  • ✅ Import contracts validated (8 contracts kept)
  • ✅ Code formatted with Ruff
  • Manual testing recommended with Aliyun SLS credentials

Screenshots

Not applicable - backend infrastructure change with no UI changes.

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/28986 **State:** closed **Merged:** Yes --- ## Summary This PR implements Aliyun Simple Log Service (SLS) integration as an alternative storage backend for workflow execution logs, addressing the challenge described in #27117 where PostgreSQL-based execution records consume significant database space and create stability/performance risks. * Resolves #27117 ### 🚀 Motivation: Why Move Logs to Aliyun SLS? As Dify deployments scale, storing high-volume workflow runtime logs in the primary PostgreSQL database presents significant architectural challenges: * **Performance Bottlenecks:** Heavy write throughput from workflow executions consumes valuable database connections and IOPS, often competing with critical business transactions (e.g., app configuration, knowledge base indexing). * **Scalability Limits:** Rapidly growing log tables degrade query performance over time, causing issues like increased latency for the "first token" and necessitating frequent manual maintenance (vacuuming, partitioning). * **Limited Observability:** The relational database model is ill-suited for the multi-dimensional, full-text analysis required for deep debugging and LLMOps auditing. ### ✨ Key Benefits of SLS Integration This PR introduces **Aliyun Log Service (SLS)** as a specialized, serverless storage backend for runtime logs, offering a robust **Storage-Compute Decoupling** solution: * **🛡️ Stability & Isolation** Offloads the write pressure of massive execution logs from PostgreSQL to SLS. This **Read/Write Separation** ensures that spikes in AI inference traffic do not exhaust DB connection pools or lock resources, effectively protecting the stability of the core Dify application. * **📊 Advanced LLMOps Analytics** Unlocks native, real-time SQL analysis capabilities. Unlike simple keyword searches in DBs, SLS allows users to build custom **Dashboards** for: * **Cost Analysis:** Track token usage by model or user. * **Performance Profiling:** Monitor P95/P99 latency of workflow nodes to identify bottlenecks. * **Alerting:** Set up automatic alerts for abnormal failure rates. * **⚡ Zero-Latency Impact** Implements an **asynchronous batching mechanism**. Log data is buffered and pushed to SLS in the background via a worker, ensuring that the logging process adds **zero blocking latency** to the main workflow inference path. * **📉 Cost & Lifecycle Efficiency** Leverages a storage engine optimized for append-only logs with lower costs than SSD-backed databases. Native support for **TTL (Time-To-Live)** and cold storage archiving eliminates the need for manual data cleanup scripts. ### Key Features: **1. Aliyun SLS LogStore Integration** - New extension module (`ext_logstore`) that initializes SLS projects, logstores, and indexes on application startup - Automatic index schema synchronization from SQLAlchemy models, ensuring consistency between database schema and LogStore indexes - Support for storing workflow execution and node execution logs in Aliyun SLS - Idempotent initialization that safely handles existing projects and logstores **2. Dual-Write Support for Safe Migration** - Configurable dual-write mode (`LOGSTORE_DUAL_WRITE_ENABLED`) that writes to both SLS and SQL database simultaneously - Enables safe migration from SQL-based logging to SLS without data loss **3. Repository Pattern Implementation** - Four new repository classes implementing the workflow execution repository interface: - `LogstoreWorkflowExecutionRepository` - for workflow runs - `LogstoreWorkflowNodeExecutionRepository` - for workflow node executions - `LogstoreApiWorkflowRunRepository` - for API workflow runs - `LogstoreApiWorkflowNodeExecutionRepository` - for API workflow node executions - Automatic type mapping from SQLAlchemy column types to LogStore index types (text, long, double, json) - Full compatibility with existing domain models (`WorkflowExecution` & `WorkflowNodeExecution`) **4. Production-Ready Configuration** - Environment variables for Aliyun SLS authentication and configuration - Configurable TTL (Time-To-Live) for log retention with automatic cleanup - Debug logging support when `SQLALCHEMY_ECHO=true` ### Configuration Example: ```bash # Enable Aliyun SLS LogStore ALIYUN_SLS_ACCESS_KEY_ID=your_access_key_id ALIYUN_SLS_ACCESS_KEY_SECRET=your_access_key_secret ALIYUN_SLS_ENDPOINT=cn-hangzhou.log.aliyuncs.com ALIYUN_SLS_REGION=cn-hangzhou ALIYUN_SLS_PROJECT_NAME=dify-workflow-logs # Optional: Configure TTL and dual-write behavior ALIYUN_SLS_LOGSTORE_TTL=365 # default: 365 days, 3650 for permanent storage LOGSTORE_DUAL_READ_ENABLED=false # Set to true during migration phase ``` ### Architecture Notes: This implementation follows the repository pattern established in #23966 and aligns with the domain model architecture improvements referenced in #19850 and #19428. The LogStore repositories implement the same interfaces as the existing SQLAlchemy repositories, ensuring seamless integration with the workflow execution system. ### Testing Notes: - ✅ All backend linting checks passed (`make lint`) - ✅ Type checking passed with BasedPyright (`make type-check`) - ✅ Import contracts validated (8 contracts kept) - ✅ Code formatted with Ruff - Manual testing recommended with Aliyun SLS credentials ## Screenshots Not applicable - backend infrastructure change with no UI changes. ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:51:04 -05:00
yindo closed this issue 2026-02-21 20:51:04 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32266
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?