issue: No docs on web loaders #66

New Issue

2026-02-15T17:05:48-05:00

yindo commented

2026-02-15 17:05:48 -05:00

Originally created by @rragundez on GitHub (Mar 12, 2025).

I am making tests using the web search functionality and I see quite different behaviour from the different loaders, unfortunately there is no documentation about what their difference in terms of accuracy and speed for example.

SafeWebBaseLoader
SafePlaywrightURLLoader
SafeFireCrawlLoader

I do see they are based on some langchain loaders but they also contain significant custom logic in open webui.

In particular SafeWebBaseLoader sometimes kinda hangs, and I would suspect it is because one of the sites is giving it trouble and it keep retrying with backoff but the logs are not really telling anything about that and I do not know how to find it out or change the bahaviour.

Hope someone can help clarifying the differences, even if it is just some simple bullet points would help.

As a general comment, it appears that the documentation about features kinda stopped at some point and now new features are either not documented or poorly documented, like things as simple as available environment variables to control web search behaviour, I had to go into the source code to find that out. IMO as I have seen in other open source projects if this continues it limits the usability of the product for non-experts that are not contributors to the project.

Originally created by @rragundez on GitHub (Mar 12, 2025). I am making tests using the web search functionality and I see quite different behaviour from the different loaders, unfortunately there is no documentation about what their difference in terms of accuracy and speed for example. - SafeWebBaseLoader - SafePlaywrightURLLoader - SafeFireCrawlLoader I do see they are based on some langchain loaders but they also contain significant custom logic in open webui. In particular SafeWebBaseLoader sometimes kinda hangs, and I would suspect it is because one of the sites is giving it trouble and it keep retrying with backoff but the logs are not really telling anything about that and I do not know how to find it out or change the bahaviour. Hope someone can help clarifying the differences, even if it is just some simple bullet points would help. As a general comment, it appears that the documentation about features kinda stopped at some point and now new features are either not documented or poorly documented, like things as simple as available environment variables to control web search behaviour, I had to go into the source code to find that out. IMO as I have seen in other open source projects if this continues it limits the usability of the product for non-experts that are not contributors to the project.

yindo added the documentation help wanted labels 2026-02-15 17:05:48 -05:00

yindo commented

2026-02-15 17:05:48 -05:00

@roryeckel commented on GitHub (Mar 15, 2025):

I wrote the docs when the feature was introduced, it was just that the docs PR wasn't merged in at the same time.

https://github.com/open-webui/docs/pull/411

I have not written about FireCrawl as I am not familiar with it, I only wrote the Playwright part but I am interested in learning about it.

@roryeckel commented on GitHub (Mar 15, 2025): I wrote the docs when the feature was introduced, it was just that the docs PR wasn't merged in at the same time. https://github.com/open-webui/docs/pull/411 I have not written about FireCrawl as I am not familiar with it, I only wrote the Playwright part but I am interested in learning about it.

yindo commented

2026-02-15 17:05:49 -05:00