langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #29480] feat: Add InterSystems IRIS vector database support #32430

New Issue
Closed
opened 2026-02-21 20:51:23 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:51:23 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/29480

State: closed
Merged: Yes

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Fixes #28778

This PR adds support for InterSystems IRIS as a vector database backend for Dify. IRIS is an enterprise-grade multi-model database with native VECTOR data type and HNSW (Hierarchical Navigable Small World) indexing for efficient similarity search.

Key Features Implemented:

  • Native VECTOR Support: Uses IRIS native VECTOR(DOUBLE, dimension) data type with HNSW indexing for fast cosine similarity search
  • Connection Pooling: Thread-safe singleton connection pool to minimize license usage and improve performance
  • Schema Auto-Creation: Idempotent schema creation with in-memory caching to avoid redundant database queries
  • Full-Text Search: Optional iFind index support with configurable language analyzers
  • Docker Integration: Complete Docker Compose setup with initialization scripts for easy deployment
  • Comprehensive Testing: Integration tests following Dify's AbstractVectorTest pattern

Architecture:

├── __init__.py
└── iris_vector.py          # Main implementation with IrisVector class

api/configs/middleware/vdb/
└── iris_config.py          # Configuration management

api/tests/integration_tests/vdb/iris/
├── __init__.py
└── test_iris.py            # Integration tests

docker/iris/
├── docker-entrypoint.sh    # Container initialization
└── iris-init.script        # Database setup script

Technical Implementation:

  1. Backend (api/core/rag/datasource/vdb/iris/iris_vector.py):

    • IrisConnectionPool: Thread-safe connection pool with health checks
    • IrisVector: BaseVector implementation with CRUD operations
    • Distributed locking using Redis for concurrent schema creation

  2. Configuration (api/configs/middleware/vdb/iris_config.py):

    • Comprehensive validation for connection parameters
    • Support for both individual parameters and connection URL
    • Configurable pool sizes (min: 1-10, max: 1-20)

  3. Docker Setup:

    • IRIS Community Edition service in Docker Compose
    • Custom initialization scripts for database setup
    • Profile-based optional deployment

  4. Tests (api/tests/integration_tests/vdb/iris/test_iris.py):

    • Integration tests using AbstractVectorTest pattern
    • DevContainer-compatible configuration

How to Use IRIS Vector Database:

**1. Configure Environment Variables:**

Edit your `.env` file (or `docker/.env`) to set IRIS as the vector store:

bash
# Set IRIS as the vector database
VECTOR_STORE=iris

# IRIS connection settings
IRIS_HOST=iris
IRIS_SUPER_SERVER_PORT=1972
IRIS_USER=_SYSTEM
IRIS_PASSWORD=YourSecurePassword
IRIS_DATABASE=USER
IRIS_SCHEMA=dify
IRIS_MIN_CONNECTION=1
IRIS_MAX_CONNECTION=3

# Optional: Enable full-text search
IRIS_TEXT_INDEX=true
IRIS_TEXT_INDEX_LANGUAGE=en

**2. Start Docker Compose with IRIS Profile:**
# Start Dify with IRIS container
docker compose up -d

**3. Verify IRIS is Running:**
# Check IRIS container status
docker compose ps iris

# View IRIS logs
docker compose logs iris

Files Changed:

Core Implementation:

  • api/core/rag/datasource/vdb/iris/__init__.py - Package initialization
  • api/core/rag/datasource/vdb/iris/iris_vector.py - Main IRIS vector database implementation
  • api/configs/middleware/vdb/iris_config.py - Configuration class with validation

Integration:

  • api/core/rag/datasource/vdb/vector_factory.py - Added IRIS factory registration
  • api/core/rag/datasource/vdb/vector_type.py - Added VectorType.IRIS enum
  • api/configs/middleware/__init__.py - Added IRIS configuration imports

Testing:

  • api/tests/integration_tests/vdb/iris/__init__.py - Test package initialization
  • api/tests/integration_tests/vdb/iris/test_iris.py - Integration test suite
  • api/tests/integration_tests/.env.example - Added IRIS test configuration

Docker & Deployment:

  • docker/iris/docker-entrypoint.sh - Container entrypoint script
  • docker/iris/iris-init.script - IRIS database initialization script
  • docker/docker-compose.yaml - Added IRIS service with profile support
  • docker/docker-compose-template.yaml - Template with IRIS configuration
  • docker/.env.example - Added IRIS environment variables documentation

Dependencies:

  • api/pyproject.toml - Added intersystems-irispython>=5.1.0 to vdb extras
  • api/uv.lock - Updated dependency lock file

Documentation:

  • .gitignore - Added IRIS-specific ignores

Additional Notes:

Deployment Requirements:

  • ⚠️ Container Image Update Required: The dify-api container image must be rebuilt to include the new intersystems-irispython dependency. Users deploying with Docker Compose will need to rebuild the API service after merging this PR:

IRIS Container Initialization:

  • 🔐 Default Password Configuration: The IRIS container automatically changes the default _SYSTEM user password to Dify@1234 during initialization. This is configured in the docker/iris/iris-init.script file. For production deployments, it is strongly recommended to change this password by modifying the IRIS_PASSWORD environment variable in your .env file.

Connection Pooling Strategy:

  • Singleton pattern used to minimize IRIS license consumption across multiple API instances
  • Connection health checks ensure stale connections are replaced automatically
  • Configurable min/max pool sizes (default: 1-3 connections)

Schema Management:

  • Double-checked locking pattern ensures thread-safe schema creation
  • In-memory cache avoids redundant database queries for schema existence
  • Redis distributed lock prevents race conditions in multi-instance deployments

Performance Optimizations:

  • HNSW index provides O(log n) similarity search performance
  • Optional iFind full-text index for hybrid search capabilities
  • Batch operations supported for efficient bulk insertions

DevContainer Compatibility:

  • Tests use host.docker.internal for connecting from DevContainer to host Docker
  • Fallback to localhost when running on host OS directly

Screenshots

N/A - Backend feature only

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/29480 **State:** closed **Merged:** Yes --- > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary Fixes #28778 This PR adds support for **InterSystems IRIS** as a vector database backend for Dify. IRIS is an enterprise-grade multi-model database with native VECTOR data type and HNSW (Hierarchical Navigable Small World) indexing for efficient similarity search. ### Key Features Implemented: - **Native VECTOR Support**: Uses IRIS native `VECTOR(DOUBLE, dimension)` data type with HNSW indexing for fast cosine similarity search - **Connection Pooling**: Thread-safe singleton connection pool to minimize license usage and improve performance - **Schema Auto-Creation**: Idempotent schema creation with in-memory caching to avoid redundant database queries - **Full-Text Search**: Optional iFind index support with configurable language analyzers - **Docker Integration**: Complete Docker Compose setup with initialization scripts for easy deployment - **Comprehensive Testing**: Integration tests following Dify's AbstractVectorTest pattern ### Architecture: ```api/core/rag/datasource/vdb/iris/ ├── __init__.py └── iris_vector.py # Main implementation with IrisVector class api/configs/middleware/vdb/ └── iris_config.py # Configuration management api/tests/integration_tests/vdb/iris/ ├── __init__.py └── test_iris.py # Integration tests docker/iris/ ├── docker-entrypoint.sh # Container initialization └── iris-init.script # Database setup script ``` ### Technical Implementation: 1. **Backend** ([api/core/rag/datasource/vdb/iris/iris_vector.py](api/core/rag/datasource/vdb/iris/iris_vector.py)): - `IrisConnectionPool`: Thread-safe connection pool with health checks - `IrisVector`: BaseVector implementation with CRUD operations - Distributed locking using Redis for concurrent schema creation 2. **Configuration** ([api/configs/middleware/vdb/iris_config.py](api/configs/middleware/vdb/iris_config.py)): - Comprehensive validation for connection parameters - Support for both individual parameters and connection URL - Configurable pool sizes (min: 1-10, max: 1-20) 3. **Docker Setup**: - IRIS Community Edition service in Docker Compose - Custom initialization scripts for database setup - Profile-based optional deployment 4. **Tests** ([api/tests/integration_tests/vdb/iris/test_iris.py](api/tests/integration_tests/vdb/iris/test_iris.py)): - Integration tests using AbstractVectorTest pattern - DevContainer-compatible configuration ### How to Use IRIS Vector Database: ``` **1. Configure Environment Variables:** Edit your `.env` file (or `docker/.env`) to set IRIS as the vector store: bash # Set IRIS as the vector database VECTOR_STORE=iris # IRIS connection settings IRIS_HOST=iris IRIS_SUPER_SERVER_PORT=1972 IRIS_USER=_SYSTEM IRIS_PASSWORD=YourSecurePassword IRIS_DATABASE=USER IRIS_SCHEMA=dify IRIS_MIN_CONNECTION=1 IRIS_MAX_CONNECTION=3 # Optional: Enable full-text search IRIS_TEXT_INDEX=true IRIS_TEXT_INDEX_LANGUAGE=en **2. Start Docker Compose with IRIS Profile:** # Start Dify with IRIS container docker compose up -d **3. Verify IRIS is Running:** # Check IRIS container status docker compose ps iris # View IRIS logs docker compose logs iris ``` ### Files Changed: **Core Implementation:** - `api/core/rag/datasource/vdb/iris/__init__.py` - Package initialization - `api/core/rag/datasource/vdb/iris/iris_vector.py` - Main IRIS vector database implementation - `api/configs/middleware/vdb/iris_config.py` - Configuration class with validation **Integration:** - `api/core/rag/datasource/vdb/vector_factory.py` - Added IRIS factory registration - `api/core/rag/datasource/vdb/vector_type.py` - Added `VectorType.IRIS` enum - `api/configs/middleware/__init__.py` - Added IRIS configuration imports **Testing:** - `api/tests/integration_tests/vdb/iris/__init__.py` - Test package initialization - `api/tests/integration_tests/vdb/iris/test_iris.py` - Integration test suite - `api/tests/integration_tests/.env.example` - Added IRIS test configuration **Docker & Deployment:** - `docker/iris/docker-entrypoint.sh` - Container entrypoint script - `docker/iris/iris-init.script` - IRIS database initialization script - `docker/docker-compose.yaml` - Added IRIS service with profile support - `docker/docker-compose-template.yaml` - Template with IRIS configuration - `docker/.env.example` - Added IRIS environment variables documentation **Dependencies:** - `api/pyproject.toml` - Added `intersystems-irispython>=5.1.0` to vdb extras - `api/uv.lock` - Updated dependency lock file **Documentation:** - `.gitignore` - Added IRIS-specific ignores ### Additional Notes: **Deployment Requirements:** - ⚠️ **Container Image Update Required**: The `dify-api` container image must be rebuilt to include the new `intersystems-irispython` dependency. Users deploying with Docker Compose will need to rebuild the API service after merging this PR: **IRIS Container Initialization:** - 🔐 Default Password Configuration: The IRIS container automatically changes the default _SYSTEM user password to Dify@1234 during initialization. This is configured in the docker/iris/iris-init.script file. For production deployments, it is strongly recommended to change this password by modifying the IRIS_PASSWORD environment variable in your .env file. **Connection Pooling Strategy:** - Singleton pattern used to minimize IRIS license consumption across multiple API instances - Connection health checks ensure stale connections are replaced automatically - Configurable min/max pool sizes (default: 1-3 connections) **Schema Management:** - Double-checked locking pattern ensures thread-safe schema creation - In-memory cache avoids redundant database queries for schema existence - Redis distributed lock prevents race conditions in multi-instance deployments **Performance Optimizations:** - HNSW index provides O(log n) similarity search performance - Optional iFind full-text index for hybrid search capabilities - Batch operations supported for efficient bulk insertions **DevContainer Compatibility:** - Tests use `host.docker.internal` for connecting from DevContainer to host Docker - Fallback to `localhost` when running on host OS directly ## Screenshots N/A - Backend feature only ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [ ] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:51:23 -05:00
yindo closed this issue 2026-02-21 20:51:23 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32430
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?