[DOCS]: Robots.txt is killing us #249

New Issue

2026-02-16T05:19:35-05:00

yindo commented

2026-02-16 05:19:35 -05:00

Originally created by @kcsf on GitHub (Oct 11, 2025).

Type of Documentation Change

Other

Documentation Page URL or Path

https://docs.dify.ai/robots.txt

Current Content

Robots.txt is currently set to disallow EVERYTHING:
User-agent: *
Disallow: /

This effectively prevents ALL RESPONSIBLE agents from reading the docs and helping end-users build with your tools.
ie. Claude "Failed to fetch https://docs.dify.ai/en/guides/workflow/node/llm"

I know you have a "Dify Docs MCP" service - but that's broken on windows. I don't have time to figure out why. And it wouldn't really be necessary if we could just fetch the doc pages that we need, when we need them. I also know bandwidth can get expensive. Blocking "everything" is not the answer. Use your own tools to create a more refined robots.txt that blocks 90% of the garbage but still allows your active customers to read docs and build responsibly.

At the very least, I would appreciate it if you would allow reading of the docs we use every day:

User-agent: *
Disallow: /
Allow: /en/guides/

Suggested Changes

User-agent: *
Disallow: /
Allow: /en/guides/

Reason for Change

Claude Desktop can't fetch necessary docs without this change.

Code of Conduct

I agree to follow Dify's documentation contribution guidelines
I have searched for existing issues search for existing issues, including closed ones.
I confirm that I am using English to submit this report, otherwise it will be closed.
【中文用户 & Non English User】请使用英语提交，否则会被关闭：）
Please do not modify this template :) and fill in all the required fields.

Originally created by @kcsf on GitHub (Oct 11, 2025). ### Type of Documentation Change Other ### Documentation Page URL or Path https://docs.dify.ai/robots.txt ### Current Content Robots.txt is currently set to disallow EVERYTHING: User-agent: * Disallow: / This effectively prevents ALL RESPONSIBLE agents from reading the docs and helping end-users build with your tools. ie. Claude "Failed to fetch https://docs.dify.ai/en/guides/workflow/node/llm" I know you have a "Dify Docs MCP" service - but that's broken on windows. I don't have time to figure out why. And it wouldn't really be necessary if we could just fetch the doc pages that we need, when we need them. I also know bandwidth can get expensive. Blocking "everything" is not the answer. Use your own tools to create a more refined robots.txt that blocks 90% of the garbage but still allows your active customers to read docs and build responsibly. At the very least, I would appreciate it if you would allow reading of the docs we use every day: User-agent: * Disallow: / Allow: /en/guides/ ### Suggested Changes User-agent: * Disallow: / Allow: /en/guides/ ### Reason for Change Claude Desktop can't fetch necessary docs without this change. ### Code of Conduct - [x] I agree to follow Dify's documentation [contribution guidelines](https://github.com/langgenius/dify/blob/0277a37fcad5ad86aeb239485c27fffd5cd90043/CONTRIBUTING.md) - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭：） - [x] Please do not modify this template :) and fill in all the required fields.

yindo added the documentation bug labels 2026-02-16 05:19:35 -05:00

yindo closed this issue

2026-02-16 05:19:35 -05:00

yindo commented

2026-02-16 05:19:35 -05:00

@kcsf commented on GitHub (Oct 11, 2025):

Obviously with access to your full site-map and traffic analytics, you could have Claude write a 100 line robots.txt that serves the purpose of conserving bandwidth/etc. while providing access to responsible agents.

@kcsf commented on GitHub (Oct 11, 2025): Obviously with access to your full site-map and traffic analytics, you could have Claude write a 100 line robots.txt that serves the purpose of conserving bandwidth/etc. while providing access to responsible agents.

yindo commented

2026-02-16 05:19:35 -05:00

@dosubot[bot] commented on GitHub (Oct 23, 2025):

Hi, @kcsf. I'm Dosu, and I'm helping the dify-docs team manage their backlog and am marking this issue as stale.

Issue Summary:

You pointed out that the current robots.txt blocks all user agents, preventing tools like Claude Desktop from accessing documentation.
You proposed updating robots.txt to allow access to the /en/guides/ path to balance accessibility and bandwidth.
You later suggested using site analytics to create a more nuanced robots.txt that permits responsible agents while managing traffic.
There has been no response from the maintainers on this issue so far.

Next Steps:

Please let me know if this issue is still relevant to the latest version of the dify-docs repository by commenting here.
If I do not hear back within 5 days, I will automatically close this issue.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Oct 23, 2025): Hi, @kcsf. I'm [Dosu](https://dosu.dev), and I'm helping the dify-docs team manage their backlog and am marking this issue as stale. **Issue Summary:** - You pointed out that the current robots.txt blocks all user agents, preventing tools like Claude Desktop from accessing documentation. - You proposed updating robots.txt to allow access to the /en/guides/ path to balance accessibility and bandwidth. - You later suggested using site analytics to create a more nuanced robots.txt that permits responsible agents while managing traffic. - There has been no response from the maintainers on this issue so far. **Next Steps:** - Please let me know if this issue is still relevant to the latest version of the dify-docs repository by commenting here. - If I do not hear back within 5 days, I will automatically close this issue. Thank you for your understanding and contribution!

yindo referenced this issue

2026-02-16 06:15:30 -05:00

[PR #249] [CLOSED] Revert "Docs: update step by step run" #430

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-docs#249