langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-19 08:25:29 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[PR #3673] [MERGED] Surface rerankers and modernize Hugging Face integration pages #3706

New Issue
Closed
opened 2026-06-05 18:52:53 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 18:52:53 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/docs/pull/3673
Author: @tomaarsen
Created: 4/22/2026
Status: ✅ Merged
Merged: 5/12/2026
Merged by: @ccurme

Base: main ← Head: docs/hf-docs-and-rerankers

📝 Commits (10+)

  • fdee08a Surface rerankers and modernize Hugging Face integration pages
  • cb9b00b LateOn is the ColBERT model
  • 249e2e4 Attempt to fix the broken links
  • f4d5b07 Merge branch 'main' into docs/hf-docs-and-rerankers
  • 37d06db Move "choosing embedding models" to python/integrations/embeddings
  • ab1da5f Merge branch 'main' into docs/hf-docs-and-rerankers
  • c97a624 revert changes to sidebars
  • e83e15d delete document_transformers landing page
  • 6078228 delete cachebackedembeddings reference
  • a87262a delete section from tutorial

📊 Changes

10 files changed (+349 additions, -225 deletions)

View changed files

📝 src/oss/python/integrations/document_transformers/cross_encoder_reranker.mdx (+34 -85)
📝 src/oss/python/integrations/document_transformers/infinity_rerank.mdx (+7 -3)
📝 src/oss/python/integrations/document_transformers/volcengine_rerank.mdx (+7 -3)
📝 src/oss/python/integrations/embeddings/bge_huggingface.mdx (+56 -19)
📝 src/oss/python/integrations/embeddings/huggingfacehub.mdx (+37 -46)
📝 src/oss/python/integrations/embeddings/index.mdx (+105 -0)
📝 src/oss/python/integrations/embeddings/instruct_embeddings.mdx (+18 -18)
📝 src/oss/python/integrations/embeddings/sentence_transformers.mdx (+71 -27)
📝 src/oss/python/integrations/providers/huggingface.mdx (+10 -23)
📝 src/snippets/embeddings-tabs-py.mdx (+4 -1)

📄 Description

Overview

  • Modernize the Sentence Transformers and Hugging Face embedding and cross-encoder reranker integration pages, fixing deprecated langchain_community imports and stale 2023-era model defaults
  • Surface HuggingFace local models where they're currently under-represented: add HuggingFaceEmbeddings to the Python "Top integrations" table, add a new shared reranker-tabs snippet, and add a brand-new document_transformers/index.mdx with a reranker comparison table (previously that entire directory had no index page and was completely absent from the sidebar navigation)
  • Add a new "Choose an embedding model" conceptual guide and a reranking section in the flagship RAG tutorial

Type of change

Type: New and updated documentation page

Related issues/PRs

N/A

Checklist

  • I have read the contributing guidelines, including the language policy
  • I have tested my changes locally using docs dev
  • All code examples have been tested and work correctly
  • I have used root relative paths for internal links
  • I have updated navigation in src/docs.json if needed

I couldn't run make broken-links locally, but did run make lint_prose.

Additional notes

I'm the maintainer of Sentence Transformers, and I'm pitching this PR with two main goals.

The first is to highlight rerankers more across the docs. The flagship RAG tutorial never mentioned them, even though reranking is probably the cheapest retrieval-quality win available on top of vector search. The entire document_transformers/ directory had no index page and was completely absent from the sidebar navigation, so pages like cross_encoder_reranker.mdx were only reachable via direct URL or cross-link. And cross_encoder_reranker.mdx itself was using a 2023-era model default.

The second is to update the Sentence Transformers / Hugging Face integration pages. Several still imported from deprecated langchain_community paths (including HuggingFaceInferenceAPIEmbeddings, which has been explicitly deprecated since langchain-community==0.2.2), the default model IDs were a couple of years stale, and the main sentence_transformers.mdx page opened with a red "experienced users only" warning that the current pip install langchain-huggingface has made obsolete.

For context: the prose and code changes here were drafted with AI assistance and then critically edited by me before opening the PR. Every claim about the current state of the code, every recommended model, and every cross-link was checked against the repo and against the live PyPI packages (langchain-huggingface, langchain-community, langchain-classic), and the model recommendations are based on my own experience with the Sentence Transformers ecosystem and MTEB (e.g. the newly released lightonai/DenseOn model).

Concrete changes:

  • New src/oss/python/integrations/document_transformers/index.mdx with a reranker comparison table (HuggingFace first, with a "local" column), plus a content-transformation section and a card grid of all 21 pages. Registered in the "Integrations by component" sidebar group.
  • New "Improve retrieval with reranking" section in rag.mdx, backed by a new reranker-tabs-py.mdx snippet covering HuggingFace (local, no API key), Cohere, Jina, Voyage AI, and FlashRank.
  • cross_encoder_reranker.mdx: default bumped to BAAI/bge-reranker-v2-m3, plus a model-selection table covering mixedbread-ai/mxbai-rerank-large-v2, Alibaba-NLP/gte-multilingual-reranker-base, Qwen/Qwen3-Reranker-0.6B, and the classic cross-encoder/ms-marco-* family.
  • sentence_transformers.mdx: warning removed; now covers device/throughput (with ST's auto-selection noted), normalization, batch sizes, query/document prompts for instruction-aware models, and Text Embeddings Inference for production.
  • bge_huggingface.mdx restructured by model generation: BAAI/bge-m3 via HuggingFaceEmbeddings as the recommended path, bge-*-en-v1.5 via HuggingFaceBgeEmbeddings as a zero-config alternative. Also caught that the legacy class defaults to "Represent this question..." while the v1.5 model card actually recommends "Represent this sentence..."; the generic-class example uses the model-card wording.
  • Migrated deprecated imports on providers/huggingface.mdx, instruct_embeddings.mdx, and huggingfacehub.mdx, with short deprecation notes (including the explicit HuggingFaceInferenceAPIEmbeddings deprecation).
  • New choosing-embeddings.mdx: provider-neutral conceptual guide covering deployment patterns, MTEB, cost/latency/dimensionality/context-length trade-offs, multilingual support, prompts, licensing, and rerankers / hybrid / late-interaction (including lightonai/DenseOn).
  • Hygiene: added HuggingFaceEmbeddings to the Python embeddings/index.mdx "Top integrations" table; bumped infinity_rerank.mdx and volcengine_rerank.mdx embedding examples from all-MiniLM-L6-v2 to BAAI/bge-m3.

I also ran the dev server and browsed each changed page end-to-end before opening this. Happy to pare back or split if you'd prefer. Let me know. I think it'll be nice to display the rerankers a tad more prominently in the docs via this.

  • Tom Aarsen

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/docs/pull/3673 **Author:** [@tomaarsen](https://github.com/tomaarsen) **Created:** 4/22/2026 **Status:** ✅ Merged **Merged:** 5/12/2026 **Merged by:** [@ccurme](https://github.com/ccurme) **Base:** `main` ← **Head:** `docs/hf-docs-and-rerankers` --- ### 📝 Commits (10+) - [`fdee08a`](https://github.com/langchain-ai/docs/commit/fdee08ad68f254a7d31f55f711de86d976fcf52e) Surface rerankers and modernize Hugging Face integration pages - [`cb9b00b`](https://github.com/langchain-ai/docs/commit/cb9b00bd0a63e9cbb2e277739bec339fcc351193) LateOn is the ColBERT model - [`249e2e4`](https://github.com/langchain-ai/docs/commit/249e2e47ec10e8dcf84e07665fde792a4f6654ef) Attempt to fix the broken links - [`f4d5b07`](https://github.com/langchain-ai/docs/commit/f4d5b07ca7ddd9af76f556f7083d84cd67384fba) Merge branch 'main' into docs/hf-docs-and-rerankers - [`37d06db`](https://github.com/langchain-ai/docs/commit/37d06db81c983f858fa886b05952bab5717ea008) Move "choosing embedding models" to python/integrations/embeddings - [`ab1da5f`](https://github.com/langchain-ai/docs/commit/ab1da5f3563788b6a1c6f976be050e0b6620ac8f) Merge branch 'main' into docs/hf-docs-and-rerankers - [`c97a624`](https://github.com/langchain-ai/docs/commit/c97a624164104d4a53c199b405df5b257c05dae7) revert changes to sidebars - [`e83e15d`](https://github.com/langchain-ai/docs/commit/e83e15d3934e29cea50d76b02243aebda2ac7f22) delete document_transformers landing page - [`6078228`](https://github.com/langchain-ai/docs/commit/60782285e7f9a086f878cae5db5d6987bd9de62b) delete cachebackedembeddings reference - [`a87262a`](https://github.com/langchain-ai/docs/commit/a87262a359627d5772418bc20ff72fc5fd909823) delete section from tutorial ### 📊 Changes **10 files changed** (+349 additions, -225 deletions) <details> <summary>View changed files</summary> 📝 `src/oss/python/integrations/document_transformers/cross_encoder_reranker.mdx` (+34 -85) 📝 `src/oss/python/integrations/document_transformers/infinity_rerank.mdx` (+7 -3) 📝 `src/oss/python/integrations/document_transformers/volcengine_rerank.mdx` (+7 -3) 📝 `src/oss/python/integrations/embeddings/bge_huggingface.mdx` (+56 -19) 📝 `src/oss/python/integrations/embeddings/huggingfacehub.mdx` (+37 -46) 📝 `src/oss/python/integrations/embeddings/index.mdx` (+105 -0) 📝 `src/oss/python/integrations/embeddings/instruct_embeddings.mdx` (+18 -18) 📝 `src/oss/python/integrations/embeddings/sentence_transformers.mdx` (+71 -27) 📝 `src/oss/python/integrations/providers/huggingface.mdx` (+10 -23) 📝 `src/snippets/embeddings-tabs-py.mdx` (+4 -1) </details> ### 📄 Description ## Overview * Modernize the Sentence Transformers and Hugging Face embedding and cross-encoder reranker integration pages, fixing deprecated `langchain_community` imports and stale 2023-era model defaults * Surface HuggingFace local models where they're currently under-represented: add `HuggingFaceEmbeddings` to the Python "Top integrations" table, add a new shared reranker-tabs snippet, and add a brand-new `document_transformers/index.mdx` with a reranker comparison table (previously that entire directory had no index page and was completely absent from the sidebar navigation) * Add a new "Choose an embedding model" conceptual guide and a reranking section in the flagship RAG tutorial ## Type of change **Type:** New and updated documentation page ## Related issues/PRs N/A ## Checklist <!-- Put an 'x' in all boxes that apply --> - [x] I have read the [contributing guidelines](README.md), including the [language policy](https://docs.langchain.com/oss/python/contributing/overview#language-policy) - [x] I have tested my changes locally using `docs dev` - [x] All code examples have been tested and work correctly - [x] I have used **root relative** paths for internal links - [x] I have updated navigation in `src/docs.json` if needed I couldn't run `make broken-links` locally, but did run `make lint_prose`. ## Additional notes I'm the maintainer of Sentence Transformers, and I'm pitching this PR with two main goals. The first is to highlight rerankers more across the docs. The flagship RAG tutorial never mentioned them, even though reranking is probably the cheapest retrieval-quality win available on top of vector search. The entire `document_transformers/` directory had no index page and was completely absent from the sidebar navigation, so pages like `cross_encoder_reranker.mdx` were only reachable via direct URL or cross-link. And `cross_encoder_reranker.mdx` itself was using a 2023-era model default. The second is to update the Sentence Transformers / Hugging Face integration pages. Several still imported from deprecated `langchain_community` paths (including `HuggingFaceInferenceAPIEmbeddings`, which has been explicitly deprecated since `langchain-community==0.2.2`), the default model IDs were a couple of years stale, and the main `sentence_transformers.mdx` page opened with a red "experienced users only" warning that the current `pip install langchain-huggingface` has made obsolete. For context: the prose and code changes here were drafted with AI assistance and then critically edited by me before opening the PR. Every claim about the current state of the code, every recommended model, and every cross-link was checked against the repo and against the live PyPI packages (`langchain-huggingface`, `langchain-community`, `langchain-classic`), and the model recommendations are based on my own experience with the Sentence Transformers ecosystem and MTEB (e.g. the newly released [lightonai/DenseOn](https://huggingface.co/lightonai/DenseOn) model). Concrete changes: - New `src/oss/python/integrations/document_transformers/index.mdx` with a reranker comparison table (HuggingFace first, with a "local" column), plus a content-transformation section and a card grid of all 21 pages. Registered in the "Integrations by component" sidebar group. - New "Improve retrieval with reranking" section in `rag.mdx`, backed by a new `reranker-tabs-py.mdx` snippet covering HuggingFace (local, no API key), Cohere, Jina, Voyage AI, and FlashRank. - `cross_encoder_reranker.mdx`: default bumped to `BAAI/bge-reranker-v2-m3`, plus a model-selection table covering `mixedbread-ai/mxbai-rerank-large-v2`, `Alibaba-NLP/gte-multilingual-reranker-base`, `Qwen/Qwen3-Reranker-0.6B`, and the classic `cross-encoder/ms-marco-*` family. - `sentence_transformers.mdx`: warning removed; now covers device/throughput (with ST's auto-selection noted), normalization, batch sizes, query/document prompts for instruction-aware models, and Text Embeddings Inference for production. - `bge_huggingface.mdx` restructured by model generation: `BAAI/bge-m3` via `HuggingFaceEmbeddings` as the recommended path, `bge-*-en-v1.5` via `HuggingFaceBgeEmbeddings` as a zero-config alternative. Also caught that the legacy class defaults to `"Represent this question..."` while the v1.5 model card actually recommends `"Represent this sentence..."`; the generic-class example uses the model-card wording. - Migrated deprecated imports on `providers/huggingface.mdx`, `instruct_embeddings.mdx`, and `huggingfacehub.mdx`, with short deprecation notes (including the explicit `HuggingFaceInferenceAPIEmbeddings` deprecation). - New `choosing-embeddings.mdx`: provider-neutral conceptual guide covering deployment patterns, MTEB, cost/latency/dimensionality/context-length trade-offs, multilingual support, prompts, licensing, and rerankers / hybrid / late-interaction (including `lightonai/DenseOn`). - Hygiene: added `HuggingFaceEmbeddings` to the Python `embeddings/index.mdx` "Top integrations" table; bumped `infinity_rerank.mdx` and `volcengine_rerank.mdx` embedding examples from `all-MiniLM-L6-v2` to `BAAI/bge-m3`. I also ran the dev server and browsed each changed page end-to-end before opening this. Happy to pare back or split if you'd prefer. Let me know. I think it'll be nice to display the rerankers a tad more prominently in the docs via this. - Tom Aarsen --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 18:52:53 -04:00
yindo closed this issue 2026-06-05 18:52:53 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#3706
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?