langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-21 17:45:23 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

[PR #1585] [MERGED] feat: add JSON and YAML extractors to dify_extractor #1970

New Issue
Closed
opened 2026-02-16 11:15:44 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 11:15:44 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-official-plugins/pull/1585
Author: @brianxiadong
Created: 8/26/2025
Status: ✅ Merged
Merged: 8/27/2025
Merged by: @crazywoola

Base: feat/datasource ← Head: feat/datasource

📝 Commits (2)

  • c62ded8 feat: add JSON and YAML extractors to dify_extractor
  • d5eaab3 bump the version

📊 Changes

5 files changed (+157 additions, -5 deletions)

View changed files

📝 tools/dify_extractor/manifest.yaml (+1 -1)
📝 tools/dify_extractor/tools/dify_extractor.py (+6 -0)
📝 tools/dify_extractor/tools/dify_extractor.yaml (+4 -4)
➕ tools/dify_extractor/tools/json_extractor.py (+73 -0)
➕ tools/dify_extractor/tools/yaml_extractor.py (+73 -0)

📄 Description

Add JSON and YAML Extractors to Dify Extractor

Overview

This PR adds support for JSON and YAML file extraction to the dify_extractor tool, expanding its capabilities to handle structured data formats commonly used in configuration files, APIs, and data exchange.

close https://github.com/langgenius/dify/issues/24530
close https://github.com/langgenius/dify/issues/24529

Changes Made

New Files Added

  • tools/json_extractor.py: Implements JSONExtractor class for parsing JSON files
  • tools/yaml_extractor.py: Implements YAMLExtractor class for parsing YAML files

Modified Files

  • tools/dify_extractor.py: Updated to integrate the new extractors
  • tools/dify_extractor.yaml: Updated configuration to support .json and .yaml/.yml file extensions

Features

JSON Extractor

  • Parses JSON files and converts them to readable Markdown format
  • Handles nested objects and arrays with proper indentation
  • Provides graceful error handling for malformed JSON files
  • Maintains original structure while making content searchable

YAML Extractor

  • Parses YAML files and converts them to readable Markdown format
  • Supports both .yaml and .yml file extensions
  • Handles complex YAML structures including nested objects and lists
  • Provides graceful error handling for invalid YAML syntax
  • Preserves data types and structure in the output

Technical Implementation

Architecture

Both extractors follow the established pattern:

  • Inherit from BaseExtractor abstract class
  • Implement the required extract() method
  • Return ExtractorResult with formatted content and metadata
  • Handle file encoding detection automatically

Error Handling

  • JSON parsing errors are caught and reported with helpful error messages
  • YAML parsing errors include line numbers and specific error details
  • Invalid file formats return structured error information
  • All errors are logged appropriately for debugging

Output Format

Both extractors convert structured data to Markdown format:

  • JSON objects become formatted code blocks with syntax highlighting
  • YAML content is preserved with proper indentation
  • Nested structures are clearly represented
  • File metadata is included in the document structure

Testing

The implementation includes comprehensive error handling and has been tested with:

  • Valid JSON and YAML files with various structures
  • Malformed files to verify error handling
  • Edge cases like empty files and complex nested structures
  • Different file encodings

Benefits

  • Enhanced Data Processing: Users can now extract and process structured data from JSON and YAML files
  • Configuration File Support: Better support for modern application configuration files
  • API Response Processing: Ability to process JSON API responses and configuration data
  • DevOps Integration: Support for YAML-based CI/CD configurations and Kubernetes manifests
  • Consistent Interface: Maintains the same API as existing extractors for seamless integration

Backward Compatibility

This change is fully backward compatible:

  • No existing functionality is modified
  • All existing extractors continue to work as before
  • New file types are added without affecting existing ones
  • Configuration changes are additive only

Usage Examples

JSON Files

{
  "name": "example",
  "version": "1.0.0",
  "dependencies": {
    "library": "^2.0.0"
  }
}

YAML Files

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_url: "postgresql://localhost:5432/mydb"
  debug: true

Both file types will be converted to searchable Markdown format while preserving their structure and readability.

Future Enhancements

This foundation enables future support for:

  • TOML files
  • XML files with structured parsing
  • Custom JSON schema validation
  • YAML schema validation

Type: Feature Enhancement
Impact: Low Risk - Additive functionality only
Testing: Manual testing completed for various file formats and edge cases

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-official-plugins/pull/1585 **Author:** [@brianxiadong](https://github.com/brianxiadong) **Created:** 8/26/2025 **Status:** ✅ Merged **Merged:** 8/27/2025 **Merged by:** [@crazywoola](https://github.com/crazywoola) **Base:** `feat/datasource` ← **Head:** `feat/datasource` --- ### 📝 Commits (2) - [`c62ded8`](https://github.com/langgenius/dify-official-plugins/commit/c62ded83a6bc2fd7ac4af385bdf932b94ab49efd) feat: add JSON and YAML extractors to dify_extractor - [`d5eaab3`](https://github.com/langgenius/dify-official-plugins/commit/d5eaab3b2d7c0d2e6faef9165d8a93367ef90a14) bump the version ### 📊 Changes **5 files changed** (+157 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `tools/dify_extractor/manifest.yaml` (+1 -1) 📝 `tools/dify_extractor/tools/dify_extractor.py` (+6 -0) 📝 `tools/dify_extractor/tools/dify_extractor.yaml` (+4 -4) ➕ `tools/dify_extractor/tools/json_extractor.py` (+73 -0) ➕ `tools/dify_extractor/tools/yaml_extractor.py` (+73 -0) </details> ### 📄 Description # Add JSON and YAML Extractors to Dify Extractor ## Overview This PR adds support for JSON and YAML file extraction to the `dify_extractor` tool, expanding its capabilities to handle structured data formats commonly used in configuration files, APIs, and data exchange. close https://github.com/langgenius/dify/issues/24530 close https://github.com/langgenius/dify/issues/24529 ## Changes Made ### New Files Added - **`tools/json_extractor.py`**: Implements `JSONExtractor` class for parsing JSON files - **`tools/yaml_extractor.py`**: Implements `YAMLExtractor` class for parsing YAML files ### Modified Files - **`tools/dify_extractor.py`**: Updated to integrate the new extractors - **`tools/dify_extractor.yaml`**: Updated configuration to support `.json` and `.yaml`/`.yml` file extensions ## Features ### JSON Extractor - Parses JSON files and converts them to readable Markdown format - Handles nested objects and arrays with proper indentation - Provides graceful error handling for malformed JSON files - Maintains original structure while making content searchable ### YAML Extractor - Parses YAML files and converts them to readable Markdown format - Supports both `.yaml` and `.yml` file extensions - Handles complex YAML structures including nested objects and lists - Provides graceful error handling for invalid YAML syntax - Preserves data types and structure in the output ## Technical Implementation ### Architecture Both extractors follow the established pattern: - Inherit from `BaseExtractor` abstract class - Implement the required `extract()` method - Return `ExtractorResult` with formatted content and metadata - Handle file encoding detection automatically ### Error Handling - JSON parsing errors are caught and reported with helpful error messages - YAML parsing errors include line numbers and specific error details - Invalid file formats return structured error information - All errors are logged appropriately for debugging ### Output Format Both extractors convert structured data to Markdown format: - JSON objects become formatted code blocks with syntax highlighting - YAML content is preserved with proper indentation - Nested structures are clearly represented - File metadata is included in the document structure ## Testing The implementation includes comprehensive error handling and has been tested with: - Valid JSON and YAML files with various structures - Malformed files to verify error handling - Edge cases like empty files and complex nested structures - Different file encodings ## Benefits - **Enhanced Data Processing**: Users can now extract and process structured data from JSON and YAML files - **Configuration File Support**: Better support for modern application configuration files - **API Response Processing**: Ability to process JSON API responses and configuration data - **DevOps Integration**: Support for YAML-based CI/CD configurations and Kubernetes manifests - **Consistent Interface**: Maintains the same API as existing extractors for seamless integration ## Backward Compatibility This change is fully backward compatible: - No existing functionality is modified - All existing extractors continue to work as before - New file types are added without affecting existing ones - Configuration changes are additive only ## Usage Examples ### JSON Files ```json { "name": "example", "version": "1.0.0", "dependencies": { "library": "^2.0.0" } } ``` ### YAML Files ```yaml apiVersion: v1 kind: ConfigMap metadata: name: app-config data: database_url: "postgresql://localhost:5432/mydb" debug: true ``` Both file types will be converted to searchable Markdown format while preserving their structure and readability. ## Future Enhancements This foundation enables future support for: - TOML files - XML files with structured parsing - Custom JSON schema validation - YAML schema validation --- **Type**: Feature Enhancement **Impact**: Low Risk - Additive functionality only **Testing**: Manual testing completed for various file formats and edge cases --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 11:15:44 -05:00
yindo closed this issue 2026-02-16 11:15:44 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#1970
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?