langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Workflows and Chatflows can not be traced by Phoenix #15431

New Issue
Closed
opened 2026-02-21 19:21:42 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 19:21:42 -05:00
Owner
Copy Link
Copy Source

Originally created by @kurokobo on GitHub (Jul 16, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.6.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Prepare following docker-compose.override.yaml: 
    services:
  phoenix:
    image: arizephoenix/phoenix:version-11
    ports:
      - 6006:6006
    environment:
      PHOENIX_WORKING_DIR: /mnt/data
      PHOENIX_SECRET: phoenix-secret-00000000000000000
      PHOENIX_ADMIN_SECRET: phoenix-admin-secret-00000000000
    volumes:
      - phoenix_data:/mnt/data

volumes:
  phoenix_data:
    driver: local
  2. Start fresh Dify instance with above override file and setup model provider
  3. Create and publish new minimal workflow or chatflow which contains LLM nodes
  4. Configure tracing on the app with following params: - API Key: phoenix-admin-secret-00000000000
    • Project: Demo
    • Endpoint: http://phoenix:6006
  5. Ensure new project Demo has been created on Phoenix, and there is an api_check trace in the Demo project.
  6. Run the workflow or chatflow which created in step 3.

✔️ Expected Behavior

  • New spans and traces are recorded in Demo project on the Phoenix
  • LLM trace has input and output

❌ Actual Behavior

In the worker logs, there is a log for the tracing:

worker-1  | 2025-07-16 14:36:23.387 INFO [MainThread] [strategy.py:161] - Task tasks.ops_trace_task.process_trace_tasks[1a36c08e-bdac-4709-962f-23c0bd219d38] received
worker-1  | 2025-07-16 14:36:23.398 DEBUG [Dummy-3] [opendal_storage.py:52] - file ops_trace/34aeab0b-306e-43a7-a58e-915c275d69cd/8006532e977f4ca586c903809ac5f330.json loaded
worker-1  | 2025-07-16 14:36:23.441 INFO [Dummy-3] [arize_phoenix_trace.py:123] - [Arize/Phoenix] Trace: message_id='608c5201-8c59-4089-97a1-a778264969b1' message_data=None inputs=None outputs=None start_time=datetime.datetime(2025, 7, 16, 14, 36, 16, 929573) end_time=datetime.datetime(2025, 7, 16, 14, 36, 18, 336963) metadata={'workflow_id': '5336ac72-1c63-4f70-8cc3-56db21410abe', 'conversation_id': '3646aa5e-1949-40e3-9ef6-0de47928a71b', 'workflow_run_id': 'a8093be2-160b-4720-9e8d-8aac78fd52e4', 'tenant_id': '6919c28a-2426-4076-bc02-069e1b6d4f8c', 'elapsed_time': 1.40739, 'status': 'succeeded', 'version': 'draft', 'total_tokens': 19, 'file_list': [], 'triggered_from': 'debugging', 'user_id': '04f9154e-634f-472a-9849-3acd38bea269', 'app_id': '34aeab0b-306e-43a7-a58e-915c275d69cd'} workflow_data=<models.workflow.WorkflowRun object at 0x7f482066ff80> conversation_id='3646aa5e-1949-40e3-9ef6-0de47928a71b' workflow_app_log_id=None workflow_id='5336ac72-1c63-4f70-8cc3-56db21410abe' tenant_id='6919c28a-2426-4076-bc02-069e1b6d4f8c' workflow_run_id='a8093be2-160b-4720-9e8d-8aac78fd52e4' workflow_run_elapsed_time=1.40739 workflow_run_status='succeeded' workflow_run_inputs={'sys.query': 'Hi', 'sys.files': [], 'sys.conversation_id': '3646aa5e-1949-40e3-9ef6-0de47928a71b', 'sys.user_id': '04f9154e-634f-472a-9849-3acd38bea269', 'sys.dialogue_count': 0, 'sys.app_id': '34aeab0b-306e-43a7-a58e-915c275d69cd', 'sys.workflow_id': '5336ac72-1c63-4f70-8cc3-56db21410abe', 'sys.workflow_run_id': 'a8093be2-160b-4720-9e8d-8aac78fd52e4'} workflow_run_outputs={'answer': 'Hello! How can I assist you today?'} workflow_run_version='draft' error='' total_tokens=19 file_list=[] query='Hi'
worker-1  | 2025-07-16 14:36:23.441 INFO [Dummy-3] [ops_trace_task.py:46] - Processing trace tasks success, app_id: 34aeab0b-306e-43a7-a58e-915c275d69cd
worker-1  | 2025-07-16 14:36:23.447 DEBUG [Dummy-3] [opendal_storage.py:80] - file ops_trace/34aeab0b-306e-43a7-a58e-915c275d69cd/8006532e977f4ca586c903809ac5f330.json deleted
worker-1  | 2025-07-16 14:36:23.504 INFO [Dummy-3] [trace.py:128] - Task tasks.ops_trace_task.process_trace_tasks[1a36c08e-bdac-4709-962f-23c0bd219d38] succeeded in 0.1159144959965488s: None

However, nothing has been recorded in Demo project on the Phoenix.

Image Image Image Image
Originally created by @kurokobo on GitHub (Jul 16, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.6.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Prepare following `docker-compose.override.yaml`: ```yaml services: phoenix: image: arizephoenix/phoenix:version-11 ports: - 6006:6006 environment: PHOENIX_WORKING_DIR: /mnt/data PHOENIX_SECRET: phoenix-secret-00000000000000000 PHOENIX_ADMIN_SECRET: phoenix-admin-secret-00000000000 volumes: - phoenix_data:/mnt/data volumes: phoenix_data: driver: local ``` 2. Start fresh Dify instance with above override file and setup model provider 3. Create and publish new minimal workflow or chatflow which contains LLM nodes - Example chatflow: [tracing-bug-demo-chatflow.txt](https://github.com/user-attachments/files/21260501/tracing-bug-demo-chatflow.txt) - Example workflow: [tracing-bug-demo-workflow.txt](https://github.com/user-attachments/files/21260500/tracing-bug-demo-workflow.txt) 5. Configure tracing on the app with following params: - API Key: `phoenix-admin-secret-00000000000` - Project: `Demo` - Endpoint: `http://phoenix:6006` 5. Ensure new project Demo has been created on Phoenix, and there is an `api_check` trace in the Demo project. 6. Run the workflow or chatflow which created in step 3. ### ✔️ Expected Behavior - New spans and traces are recorded in Demo project on the Phoenix - LLM trace has input and output ### ❌ Actual Behavior In the `worker` logs, there is a log for the tracing: ```bash worker-1 | 2025-07-16 14:36:23.387 INFO [MainThread] [strategy.py:161] - Task tasks.ops_trace_task.process_trace_tasks[1a36c08e-bdac-4709-962f-23c0bd219d38] received worker-1 | 2025-07-16 14:36:23.398 DEBUG [Dummy-3] [opendal_storage.py:52] - file ops_trace/34aeab0b-306e-43a7-a58e-915c275d69cd/8006532e977f4ca586c903809ac5f330.json loaded worker-1 | 2025-07-16 14:36:23.441 INFO [Dummy-3] [arize_phoenix_trace.py:123] - [Arize/Phoenix] Trace: message_id='608c5201-8c59-4089-97a1-a778264969b1' message_data=None inputs=None outputs=None start_time=datetime.datetime(2025, 7, 16, 14, 36, 16, 929573) end_time=datetime.datetime(2025, 7, 16, 14, 36, 18, 336963) metadata={'workflow_id': '5336ac72-1c63-4f70-8cc3-56db21410abe', 'conversation_id': '3646aa5e-1949-40e3-9ef6-0de47928a71b', 'workflow_run_id': 'a8093be2-160b-4720-9e8d-8aac78fd52e4', 'tenant_id': '6919c28a-2426-4076-bc02-069e1b6d4f8c', 'elapsed_time': 1.40739, 'status': 'succeeded', 'version': 'draft', 'total_tokens': 19, 'file_list': [], 'triggered_from': 'debugging', 'user_id': '04f9154e-634f-472a-9849-3acd38bea269', 'app_id': '34aeab0b-306e-43a7-a58e-915c275d69cd'} workflow_data=<models.workflow.WorkflowRun object at 0x7f482066ff80> conversation_id='3646aa5e-1949-40e3-9ef6-0de47928a71b' workflow_app_log_id=None workflow_id='5336ac72-1c63-4f70-8cc3-56db21410abe' tenant_id='6919c28a-2426-4076-bc02-069e1b6d4f8c' workflow_run_id='a8093be2-160b-4720-9e8d-8aac78fd52e4' workflow_run_elapsed_time=1.40739 workflow_run_status='succeeded' workflow_run_inputs={'sys.query': 'Hi', 'sys.files': [], 'sys.conversation_id': '3646aa5e-1949-40e3-9ef6-0de47928a71b', 'sys.user_id': '04f9154e-634f-472a-9849-3acd38bea269', 'sys.dialogue_count': 0, 'sys.app_id': '34aeab0b-306e-43a7-a58e-915c275d69cd', 'sys.workflow_id': '5336ac72-1c63-4f70-8cc3-56db21410abe', 'sys.workflow_run_id': 'a8093be2-160b-4720-9e8d-8aac78fd52e4'} workflow_run_outputs={'answer': 'Hello! How can I assist you today?'} workflow_run_version='draft' error='' total_tokens=19 file_list=[] query='Hi' worker-1 | 2025-07-16 14:36:23.441 INFO [Dummy-3] [ops_trace_task.py:46] - Processing trace tasks success, app_id: 34aeab0b-306e-43a7-a58e-915c275d69cd worker-1 | 2025-07-16 14:36:23.447 DEBUG [Dummy-3] [opendal_storage.py:80] - file ops_trace/34aeab0b-306e-43a7-a58e-915c275d69cd/8006532e977f4ca586c903809ac5f330.json deleted worker-1 | 2025-07-16 14:36:23.504 INFO [Dummy-3] [trace.py:128] - Task tasks.ops_trace_task.process_trace_tasks[1a36c08e-bdac-4709-962f-23c0bd219d38] succeeded in 0.1159144959965488s: None ``` However, nothing has been recorded in Demo project on the Phoenix. <img width="591" height="402" alt="Image" src="https://github.com/user-attachments/assets/6adabc9a-a4ea-48b3-adae-3efcfe58b95a" /> <img width="386" height="243" alt="Image" src="https://github.com/user-attachments/assets/4bc250fc-a528-4b29-bb46-0d246d6a3b42" /> <img width="774" height="253" alt="Image" src="https://github.com/user-attachments/assets/8b69f935-fdea-4597-8786-34a3374e52ae" /> <img width="579" height="425" alt="Image" src="https://github.com/user-attachments/assets/fa42516c-4d10-4707-b6c2-e666bbdc3c07" />
yindo added the 🐞 bug label 2026-02-21 19:21:42 -05:00
yindo closed this issue 2026-02-21 19:21:42 -05:00
yindo commented 2026-02-21 19:21:42 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 16, 2025):

Based on the codebase, Dify fully supports tracing workflows and chatflows to Phoenix using the OpenTelemetry OTLP HTTP exporter, and there is no logic that disables or excludes these traces for Phoenix. The tracing logic creates both workflow-level and node-level spans with detailed metadata and submits them to Phoenix as expected. If traces are not appearing in Phoenix, the most likely causes are integration or deployment issues—such as configuration problems (API key, endpoint, project, or permissions), data format mismatches, or network/transport issues between Dify and Phoenix. There are also comprehensive logging and error paths in the tracing code, so checking both Dify and Phoenix logs for errors or warnings during trace submission may help pinpoint the problem. Troubleshooting should focus on runtime configuration, network connectivity, and confirming that the trace data matches what Phoenix expects in terms of OTLP protocol and required fields.
Relevant code details

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 16, 2025): <!-- Answer --> Based on the codebase, Dify fully supports tracing workflows and chatflows to Phoenix using the OpenTelemetry OTLP HTTP exporter, and there is no logic that disables or excludes these traces for Phoenix. The tracing logic creates both workflow-level and node-level spans with detailed metadata and submits them to Phoenix as expected. If traces are not appearing in Phoenix, the most likely causes are integration or deployment issues—such as configuration problems (API key, endpoint, project, or permissions), data format mismatches, or network/transport issues between Dify and Phoenix. There are also comprehensive logging and error paths in the tracing code, so checking both Dify and Phoenix logs for errors or warnings during trace submission may help pinpoint the problem. Troubleshooting should focus on runtime configuration, network connectivity, and confirming that the trace data matches what Phoenix expects in terms of OTLP protocol and required fields. [Relevant code details](https://github.com/langgenius/dify/blob/3aecceff27c6b712628ad463c6e6ac15b8527ebe/api/core/ops/arize_phoenix_trace/arize_phoenix_trace.py) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/44a6d89f-98f9-4b59-92e9-43ae7ec596e3?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22504)
yindo commented 2026-02-21 19:21:42 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Jul 16, 2025):

@ialisaleh @mikeldking @crazywoola
Thank you for contributing to the Phoenix integration in https://github.com/langgenius/dify/pull/19840!
With the minimal setup, it can run with just a single container, so it's been super handy as a lightweight observability solution in my self-hosted environment.

I'm facing above issue, and I've made a quick investigation.

https://github.com/langgenius/dify/blob/3aecceff27c6b712628ad463c6e6ac15b8527ebe/api/core/ops/arize_phoenix_trace/arize_phoenix_trace.py#L144-L146

The workflow_trace method returns immediately if trace_info does not contain message_data. However, based on the logs in my environment, the message_data for workflow (or chatflow) invocations is always None.

Is this guard logic with message_data appropriate? For example, when I looked at the workflow_trace for Langfuse, I didn't see any guard logic like this.

Also, it seems that the input information for the LLM node is included in node_execution.process_data.prompts rather than node_execution.inputs, so with the current implementation, even if we remove the guard logic mentioned earlier, I don't think the prompts for the LLM node will be traced.

I'm not a professional in Phoenix or LLM observability, so I don't fully understand what kind of spans and traces would be ideal in the end.
I'm thinking of creating a simple PR sometime soon, but if any of you are able to make the appropriate fixes, it would be great if you could handle it. Alternatively, I'd really appreciate it if you could review my PR when it's ready.
Thanks in advance :)

@kurokobo commented on GitHub (Jul 16, 2025): @ialisaleh @mikeldking @crazywoola Thank you for contributing to the Phoenix integration in https://github.com/langgenius/dify/pull/19840! With the minimal setup, it can run with just a single container, so it's been super handy as a lightweight observability solution in my self-hosted environment. I'm facing above issue, and I've made a quick investigation. https://github.com/langgenius/dify/blob/3aecceff27c6b712628ad463c6e6ac15b8527ebe/api/core/ops/arize_phoenix_trace/arize_phoenix_trace.py#L144-L146 The `workflow_trace` method returns immediately if `trace_info` does not contain `message_data`. However, based on the logs in my environment, the `message_data` for workflow (or chatflow) invocations is always `None`. Is this guard logic with `message_data` appropriate? For example, when I looked at the `workflow_trace` for Langfuse, I didn't see any guard logic like this. Also, it seems that the input information for the LLM node is included in `node_execution.process_data.prompts` rather than `node_execution.inputs`, so with the current implementation, even if we remove the guard logic mentioned earlier, I don't think the prompts for the LLM node will be traced. I'm not a professional in Phoenix or LLM observability, so I don't fully understand what kind of spans and traces would be ideal in the end. I'm thinking of creating a simple PR sometime soon, but if any of you are able to make the appropriate fixes, it would be great if you could handle it. Alternatively, I'd really appreciate it if you could review my PR when it's ready. Thanks in advance :)
yindo commented 2026-02-21 19:21:42 -05:00
Author
Owner
Copy Link
Copy Source

@ialisaleh commented on GitHub (Jul 16, 2025):

Hi @kurokobo - You're right about the workflow_trace guard clause. Currently, it skips tracing if message_data is None, which can unintentionally block useful spans from being recorded. In my initial testing with both local & hosted Phoenix instances (including Arize), I didn’t encounter this issue.

Your observation about node_execution.process_data.prompts vs node_execution.inputs is also spot on. It appears that the LLM prompt data may not be captured properly with the current logic, which is worth revisiting.

I’ll investigate this issue in depth and test this using a proper setup to better replicate your environment and see what improvements we can make. I appreciate you calling this out, and I’ll follow up soon with what I find. Thanks!

cc: @mikeldking @crazywoola

@ialisaleh commented on GitHub (Jul 16, 2025): Hi @kurokobo - You're right about the `workflow_trace` guard clause. Currently, it skips tracing if `message_data` is `None`, which can unintentionally block useful spans from being recorded. In my initial testing with both local & hosted Phoenix instances (including Arize), I didn’t encounter this issue. Your observation about `node_execution.process_data.prompts` vs `node_execution.inputs` is also spot on. It appears that the LLM prompt data may not be captured properly with the current logic, which is worth revisiting. I’ll investigate this issue in depth and test this using a proper setup to better replicate your environment and see what improvements we can make. I appreciate you calling this out, and I’ll follow up soon with what I find. Thanks! cc: @mikeldking @crazywoola
yindo commented 2026-02-21 19:21:42 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Jul 17, 2025):

Sent PR for this, if you have some time and are interested, I’d really appreciate it if you could take a look and review it. Thanks!
https://github.com/langgenius/dify/pull/22547

@kurokobo commented on GitHub (Jul 17, 2025): Sent PR for this, if you have some time and are interested, I’d really appreciate it if you could take a look and review it. Thanks! https://github.com/langgenius/dify/pull/22547
yindo commented 2026-02-21 19:21:42 -05:00
Author
Owner
Copy Link
Copy Source

@lyzno1 commented on GitHub (Jul 17, 2025):

+1

@lyzno1 commented on GitHub (Jul 17, 2025): +1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15431
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?