langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-22 01:55:27 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

[PR #1477] [MERGED] fix(bedrock): Comprehensive inference profile parameter support and response format fix v0.0.30 #1914

New Issue
Closed
opened 2026-02-16 11:15:13 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 11:15:13 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-official-plugins/pull/1477
Author: @akshaysinghas
Created: 8/11/2025
Status: ✅ Merged
Merged: 8/11/2025
Merged by: @crazywoola

Base: main ← Head: fix-response-format-inference-profile-clean

📝 Commits (2)

  • b51eddc fix(bedrock): Comprehensive inference profile parameter support and response format fix
  • 505f137 refactor(bedrock): Implement schema-driven parameter extraction for inference profiles

📊 Changes

2 files changed (+78 additions, -9 deletions)

View changed files

📝 models/bedrock/manifest.yaml (+1 -1)
📝 models/bedrock/models/llm/llm.py (+77 -8)

📄 Description

🎯 Overview

This PR comprehensively fixes critical issues with Bedrock inference profiles that were causing parameter limitations and missing
functionality compared to direct model access.

🐛 Issues Fixed

  1. Missing Response Format Support for Anthropic Models
  • ❌ Before: Response Format parameter missing from Anthropic inference profiles
  • ✅ After: Response Format available and functional for Anthropic-based inference profiles only
  1. Limited Parameters for Nova/Other Models
  • ❌ Before: Nova inference profiles showed only basic descriptions, missing cache settings, max tokens slider, cross-region options
  • ✅ After: All model parameters automatically available based on model schema
  1. Hard-coded Parameter Management
  • ❌ Before: Parameter lists scattered throughout code, prone to inconsistencies
  • ✅ After: Schema-driven approach automatically extracts parameters from model definitions

🔧 Technical Changes

Core Implementation

  • Schema-driven parameter extraction: New _get_inference_profile_parameter_rules() method
  • Response format processing: Only applied to Anthropic models with proper validation
  • Max tokens fix: Added support for max_new_tokens parameter used by Nova models
  • Model-specific handling: Automatic parameter inclusion based on underlying model type

Parameter Support by Model Type

📋 Anthropic (Claude): response_format, max_tokens, temperature, top_p, top_k, reasoning_type, reasoning_budget
📋 Nova: max_new_tokens, cross-region, system_cache_checkpoint, latest_two_messages_cache_checkpoint, temperature, top_p, top_k
📋 Llama: max_gen_len, temperature, top_p, top_k, reasoning_type, reasoning_budget
📋 Other models: All parameters from their respective schemas (excluding model_name)

Files Modified

  • models/bedrock/manifest.yaml - Version bump to 0.0.30
  • models/bedrock/models/llm/llm.py - Core parameter extraction and response format logic

🧪 Test Scenarios

Response Format Testing

  • Anthropic inference profiles show Response Format option
  • Nova/Llama inference profiles hide Response Format option
  • Response Format values save and generate structured outputs correctly
  • Test with various response formats (JSON, XML, etc.)

Parameter Completeness Testing

  • Nova inference profiles show max tokens slider, cache options, cross-region
  • Anthropic inference profiles show reasoning parameters, response format
  • Llama inference profiles show max_gen_len and reasoning options
  • Verify all parameters function correctly during inference

🔄 Before/After Comparison

Nova Inference Profile UI

Before: Only showed basic parameter descriptions
After: Shows complete parameter set matching direct Nova model access

Anthropic Inference Profile

Before: Missing Response Format option
After: Response Format available with proper JSON/structured output support

🏗️ Architecture Improvements

Maintainability

  • ✅ Eliminated hard-coded parameter lists
  • ✅ Single source of truth (model YAML schemas)
  • ✅ Automatic parameter discovery for new models
  • ✅ Reduced code duplication

Robustness

  • ✅ Model-specific parameter validation
  • ✅ Graceful fallback handling
  • ✅ Response format limited to compatible models only

📚 Implementation Details

Schema-Driven Parameter Extraction

def _get_inference_profile_parameter_rules(self, model_schema, underlying_model_id: str = None) -> list:
# Automatically extracts parameters from model schema
# Excludes model_name (determined by inference profile)
# Applies model-specific filtering (e.g., response_format for Anthropic only)

Response Format Processing

Only processes response_format for Anthropic models

if is_anthropic and model_parameters.get('response_format'):
# Apply ANTHROPIC_BLOCK_MODE_PROMPT transformation

⚡ Performance Impact

  • Minimal: Parameter extraction happens once during model schema loading
  • Positive: Eliminates redundant parameter filtering logic
  • Scalable: New models automatically inherit correct parameter handling

🔒 Backward Compatibility

  • ✅ Existing inference profile configurations continue to work
  • ✅ No breaking changes to API or UI behavior
  • ✅ Graceful handling of missing or invalid parameters

📋 Checklist

  • Version bumped to 0.0.30
  • Response format works only with Anthropic models
  • All model types show appropriate parameters
  • Schema-driven parameter extraction implemented
  • Max tokens fixed for Nova models (max_new_tokens support)
  • Fallback parameter handling improved
  • Code duplication eliminated
  • Full integration testing in production environment
    Screenshot 2025-08-11 at 11 23 42 AM
    Screenshot 2025-08-11 at 11 23 58 AM
    Screenshot 2025-08-11 at 11 24 04 AM
    Screenshot 2025-08-11 at 11 24 38 AM
    Screenshot 2025-08-11 at 11 24 57 AM
    Screenshot 2025-08-11 at 11 25 00 AM
    Screenshot 2025-08-11 at 11 25 06 AM

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-official-plugins/pull/1477 **Author:** [@akshaysinghas](https://github.com/akshaysinghas) **Created:** 8/11/2025 **Status:** ✅ Merged **Merged:** 8/11/2025 **Merged by:** [@crazywoola](https://github.com/crazywoola) **Base:** `main` ← **Head:** `fix-response-format-inference-profile-clean` --- ### 📝 Commits (2) - [`b51eddc`](https://github.com/langgenius/dify-official-plugins/commit/b51eddc5e0e01915aee19fe537be2a85852d997a) fix(bedrock): Comprehensive inference profile parameter support and response format fix - [`505f137`](https://github.com/langgenius/dify-official-plugins/commit/505f137d302f01aee3ba061720d47787d535af88) refactor(bedrock): Implement schema-driven parameter extraction for inference profiles ### 📊 Changes **2 files changed** (+78 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `models/bedrock/manifest.yaml` (+1 -1) 📝 `models/bedrock/models/llm/llm.py` (+77 -8) </details> ### 📄 Description 🎯 Overview This PR comprehensively fixes critical issues with Bedrock inference profiles that were causing parameter limitations and missing functionality compared to direct model access. 🐛 Issues Fixed 1. Missing Response Format Support for Anthropic Models - ❌ Before: Response Format parameter missing from Anthropic inference profiles - ✅ After: Response Format available and functional for Anthropic-based inference profiles only 2. Limited Parameters for Nova/Other Models - ❌ Before: Nova inference profiles showed only basic descriptions, missing cache settings, max tokens slider, cross-region options - ✅ After: All model parameters automatically available based on model schema 3. Hard-coded Parameter Management - ❌ Before: Parameter lists scattered throughout code, prone to inconsistencies - ✅ After: Schema-driven approach automatically extracts parameters from model definitions 🔧 Technical Changes Core Implementation - Schema-driven parameter extraction: New _get_inference_profile_parameter_rules() method - Response format processing: Only applied to Anthropic models with proper validation - Max tokens fix: Added support for max_new_tokens parameter used by Nova models - Model-specific handling: Automatic parameter inclusion based on underlying model type Parameter Support by Model Type 📋 Anthropic (Claude): response_format, max_tokens, temperature, top_p, top_k, reasoning_type, reasoning_budget 📋 Nova: max_new_tokens, cross-region, system_cache_checkpoint, latest_two_messages_cache_checkpoint, temperature, top_p, top_k 📋 Llama: max_gen_len, temperature, top_p, top_k, reasoning_type, reasoning_budget 📋 Other models: All parameters from their respective schemas (excluding model_name) Files Modified - models/bedrock/manifest.yaml - Version bump to 0.0.30 - models/bedrock/models/llm/llm.py - Core parameter extraction and response format logic 🧪 Test Scenarios Response Format Testing - Anthropic inference profiles show Response Format option - Nova/Llama inference profiles hide Response Format option - Response Format values save and generate structured outputs correctly - Test with various response formats (JSON, XML, etc.) Parameter Completeness Testing - Nova inference profiles show max tokens slider, cache options, cross-region - Anthropic inference profiles show reasoning parameters, response format - Llama inference profiles show max_gen_len and reasoning options - Verify all parameters function correctly during inference 🔄 Before/After Comparison Nova Inference Profile UI Before: Only showed basic parameter descriptions After: Shows complete parameter set matching direct Nova model access Anthropic Inference Profile Before: Missing Response Format option After: Response Format available with proper JSON/structured output support 🏗️ Architecture Improvements Maintainability - ✅ Eliminated hard-coded parameter lists - ✅ Single source of truth (model YAML schemas) - ✅ Automatic parameter discovery for new models - ✅ Reduced code duplication Robustness - ✅ Model-specific parameter validation - ✅ Graceful fallback handling - ✅ Response format limited to compatible models only 📚 Implementation Details Schema-Driven Parameter Extraction def _get_inference_profile_parameter_rules(self, model_schema, underlying_model_id: str = None) -> list: # Automatically extracts parameters from model schema # Excludes model_name (determined by inference profile) # Applies model-specific filtering (e.g., response_format for Anthropic only) Response Format Processing # Only processes response_format for Anthropic models if is_anthropic and model_parameters.get('response_format'): # Apply ANTHROPIC_BLOCK_MODE_PROMPT transformation ⚡ Performance Impact - Minimal: Parameter extraction happens once during model schema loading - Positive: Eliminates redundant parameter filtering logic - Scalable: New models automatically inherit correct parameter handling 🔒 Backward Compatibility - ✅ Existing inference profile configurations continue to work - ✅ No breaking changes to API or UI behavior - ✅ Graceful handling of missing or invalid parameters 📋 Checklist - Version bumped to 0.0.30 - Response format works only with Anthropic models - All model types show appropriate parameters - Schema-driven parameter extraction implemented - Max tokens fixed for Nova models (max_new_tokens support) - Fallback parameter handling improved - Code duplication eliminated - Full integration testing in production environment <img width="497" height="874" alt="Screenshot 2025-08-11 at 11 23 42 AM" src="https://github.com/user-attachments/assets/c408096f-72d7-4c68-8733-7cc64c037af6" /> <img width="338" height="345" alt="Screenshot 2025-08-11 at 11 23 58 AM" src="https://github.com/user-attachments/assets/8cb60176-a95b-46a7-a921-73b446e2c26d" /> <img width="353" height="298" alt="Screenshot 2025-08-11 at 11 24 04 AM" src="https://github.com/user-attachments/assets/f0e3a99f-cbbe-4890-97da-57845d58c5d6" /> <img width="431" height="456" alt="Screenshot 2025-08-11 at 11 24 38 AM" src="https://github.com/user-attachments/assets/2b3ba207-e49b-4a18-b2af-e7f2afbc7bd9" /> <img width="428" height="573" alt="Screenshot 2025-08-11 at 11 24 57 AM" src="https://github.com/user-attachments/assets/ad1683c7-cdf2-4d86-b5c1-08939b8252d8" /> <img width="79" height="202" alt="Screenshot 2025-08-11 at 11 25 00 AM" src="https://github.com/user-attachments/assets/6a67f7b2-7e7f-4580-946d-a1f765e0da57" /> <img width="415" height="879" alt="Screenshot 2025-08-11 at 11 25 06 AM" src="https://github.com/user-attachments/assets/001379f6-59bb-4110-bc81-cd356c8b140a" /> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 11:15:13 -05:00
yindo closed this issue 2026-02-16 11:15:13 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#1914
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?