langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 13:05:55 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #24521] feat: Add JsonExtractor for structured JSON file parsing in RAG #30674

New Issue
Closed
opened 2026-02-21 20:48:00 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:48:00 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/24521

State: closed
Merged: No

fixes #24529

This PR introduces a new JsonExtractor class that enables structured parsing of JSON files in the RAG (Retrieval-Augmented Generation) system. Previously, JSON files were processed as plain text by the TextExtractor , which resulted in loss of structural information and poor retrieval performance.

Key Features:

  • Structured JSON Parsing : Converts JSON objects and arrays into individual documents while preserving key-value relationships

  • Nested Structure Support : Handles complex nested JSON structures with proper flattening

  • Rich Metadata : Includes source path, object index, type information, key lists, and searchable text

  • Error Handling : Robust error handling for malformed JSON files

  • Comprehensive Testing : Full test coverage including edge cases and error scenarios
    Files Changed:

  • ✅ Added : api/core/rag/extractor/json_extractor.py - Main JsonExtractor implementation

  • ✅ Added : api/tests/unit_tests/core/rag/extractor/test_json_extractor.py - Comprehensive test suite

  • ✅ Modified : api/core/rag/extractor/extract_processor.py - Added .json extension support
    Benefits:

  • Improved Retrieval : Better semantic understanding of JSON content for RAG applications

  • Preserved Structure : Maintains JSON hierarchy and relationships in extracted documents

  • Enhanced Metadata : Rich metadata enables more precise document filtering and retrieval

  • Better User Experience : More accurate responses when working with JSON data sources
    Testing:

  • ✅ Unit tests cover JSON objects, arrays, nested structures, and error cases

  • ✅ All existing tests pass

  • ✅ Code follows project conventions and includes proper English documentation
    Technical Implementation:

  • Follows existing extractor patterns (similar to CSVExtractor)

  • Uses Python's built-in json module for parsing

  • Implements proper encoding detection and error handling

  • Generates individual Document objects for each top-level JSON element

  • Includes comprehensive metadata for enhanced retrieval capabilities

**Original Pull Request:** https://github.com/langgenius/dify/pull/24521 **State:** closed **Merged:** No --- fixes #24529 This PR introduces a new JsonExtractor class that enables structured parsing of JSON files in the RAG (Retrieval-Augmented Generation) system. Previously, JSON files were processed as plain text by the TextExtractor , which resulted in loss of structural information and poor retrieval performance. Key Features: - Structured JSON Parsing : Converts JSON objects and arrays into individual documents while preserving key-value relationships - Nested Structure Support : Handles complex nested JSON structures with proper flattening - Rich Metadata : Includes source path, object index, type information, key lists, and searchable text - Error Handling : Robust error handling for malformed JSON files - Comprehensive Testing : Full test coverage including edge cases and error scenarios Files Changed: - ✅ Added : api/core/rag/extractor/json_extractor.py - Main JsonExtractor implementation - ✅ Added : api/tests/unit_tests/core/rag/extractor/test_json_extractor.py - Comprehensive test suite - ✅ Modified : api/core/rag/extractor/extract_processor.py - Added .json extension support Benefits: - Improved Retrieval : Better semantic understanding of JSON content for RAG applications - Preserved Structure : Maintains JSON hierarchy and relationships in extracted documents - Enhanced Metadata : Rich metadata enables more precise document filtering and retrieval - Better User Experience : More accurate responses when working with JSON data sources Testing: - ✅ Unit tests cover JSON objects, arrays, nested structures, and error cases - ✅ All existing tests pass - ✅ Code follows project conventions and includes proper English documentation Technical Implementation: - Follows existing extractor patterns (similar to CSVExtractor) - Uses Python's built-in json module for parsing - Implements proper encoding detection and error handling - Generates individual Document objects for each top-level JSON element - Includes comprehensive metadata for enhanced retrieval capabilities
yindo added the pull-request label 2026-02-21 20:48:00 -05:00
yindo closed this issue 2026-02-21 20:48:00 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#30674
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?