Update development.mdx

2026-07-19 14:43:33 -04:00 · 2025-09-25 21:58:54 +02:00
parent 7a7a238286
commit 6906343e9f
1 changed files with 100 additions and 0 deletions
@@ -289,6 +289,106 @@ async def test_function(
 ```
 </details>

+### Rich UI Element Embedding
+
+Both External and Built-In Tools now support rich UI element embedding, allowing tools to return HTML content and interactive iframes that display directly within chat conversations. This feature enables tools to provide sophisticated visual interfaces, interactive widgets, charts, dashboards, and other rich web content.
+
+When a tool returns an `HTMLResponse` with the appropriate headers, the content will be embedded as an interactive iframe in the chat interface rather than displayed as plain text.
+
+#### Basic Usage
+
+To embed HTML content, your tool should return an `HTMLResponse` with the `Content-Disposition: inline` header:
+
+```python
+from fastapi.responses import HTMLResponse
+
+def create_visualization_tool(self, data: str) -> HTMLResponse:
+    """
+    Creates an interactive data visualization that embeds in the chat.
+    
+    :param data: The data to visualize
+    """
+    html_content = """
+    <!DOCTYPE html>
+    <html>
+    <head>
+        <title>Data Visualization</title>
+        <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
+    </head>
+    <body>
+        <div id="chart" style="width:100%;height:400px;"></div>
+        <script>
+            // Your interactive chart code here
+            Plotly.newPlot('chart', [{
+                y: [1, 2, 3, 4],
+                type: 'scatter'
+            }]);
+        </script>
+    </body>
+    </html>
+    """
+    
+    headers = {"Content-Disposition": "inline"}
+    return HTMLResponse(content=html_content, headers=headers)
+```
+
+#### Advanced Features
+
+The embedded iframes support auto-resizing and include configurable security settings. The system automatically handles:
+
+- **Auto-resizing**: Embedded content automatically adjusts height based on its content
+- **Cross-origin communication**: Safe message passing between the iframe and parent window
+- **Security sandbox**: Configurable security restrictions for embedded content
+
+#### Security Considerations
+
+When embedding external content, several security options can be configured through the UI settings:
+
+- `iframeSandboxAllowForms`: Allow form submissions within embedded content
+- `iframeSandboxAllowSameOrigin`: Allow same-origin requests (use with caution)
+- `iframeSandboxAllowPopups`: Allow popup windows from embedded content
+
+#### Use Cases
+
+Rich UI embedding is perfect for:
+
+- **Interactive dashboards**: Real-time data visualization and controls
+- **Form interfaces**: Complex input forms with validation and dynamic behavior  
+- **Charts and graphs**: Interactive plotting with libraries like Plotly, D3.js, or Chart.js
+- **Media players**: Video, audio, or interactive media content
+- **Custom widgets**: Specialized UI components for specific tool functionality
+- **External integrations**: Embedding content from external services or APIs
+
+#### External Tool Example
+
+For external tools served via HTTP endpoints:
+
+```python
+@app.post("/tools/dashboard")
+async def create_dashboard():
+    html = """
+    <div style="padding: 20px;">
+        <h2>System Dashboard</h2>
+        <canvas id="myChart" width="400" height="200"></canvas>
+        <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
+        <script>
+            const ctx = document.getElementById('myChart').getContext('2d');
+            new Chart(ctx, {
+                type: 'line',
+                data: { /* your chart data */ }
+            });
+        </script>
+    </div>
+    """
+    
+    return HTMLResponse(
+        content=html,
+        headers={"Content-Disposition": "inline"}
+    )
+```
+
+The embedded content automatically inherits responsive design and integrates seamlessly with the chat interface, providing a native-feeling experience for users interacting with your tools.
+
 ## External packages

 In the Tools definition metadata you can specify custom packages. When you click `Save` the line will be parsed and `pip install` will be run on all requirements at once.