langchain-ai/langsmith-docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/langsmith-docs.git synced 2026-08-27 01:41:19 -04:00
Code Issues 16 Packages Projects Releases Wiki Activity

Fix cookbook converter escaping: preserve inline code and correctly convert multiple HTML tables #86

New Issue
Closed
opened 2026-02-21 17:17:09 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 17:17:09 -05:00
Owner
Copy Link
Copy Source

Originally created by @0x-Professor on GitHub (Aug 9, 2025).

Summary

The cookbook conversion script currently over-escapes characters outside fenced code blocks but does not respect inline code (single backticks), and the HTML table conversion only handles a single table with limited fidelity. This can corrupt inline code samples and mangle pandas-generated tables in converted content.

Affected code:

  • replace_brackets does not skip inline code and notes it can break pandas tables.
  • html_table_to_markdown converts only the first table and loses alignment/formatting.

References:

Impact

  • Inline code like `a < b` becomes a &lt; b, altering rendered content.
  • Pandas tables embedded as HTML may be partially or incorrectly converted, leading to unreadable docs.
  • These issues reduce trust in code examples and degrade documentation quality.

Steps to Reproduce

  1. Run the cookbook build script on markdown that includes:
    • Inline code with <, >, {, } (e.g., Use `a < b && c > d` in filters).
    • A pandas DataFrame rendered to HTML with multiple <table> elements (e.g., with a header table and data table).
  2. Observe that:
    • Inline code segments have their characters escaped instead of preserved.
    • Only the first table is converted to Markdown; subsequent tables are ignored, and cell formatting/alignment/code spans are lost.

Proposed Solution

  • Inline code preservation:
    • Enhance replace_brackets to detect and skip transformations inside single-backtick code spans (`...`), in addition to fenced code blocks.
    • Strategy: tokenize lines by backtick pairs and only escape characters in non-code segments; keep existing fenced-code tracking.
  • Robust table conversion:
    • Update html_table_to_markdown to:
      • Iterate over all <table> elements within the document.
      • Preserve inline code within cells (do not escape backtick-enclosed spans).
      • Support basic alignment by detecting <th align>/<td align> or style attributes and mapping them to Markdown alignment (:---, :---:, ---:).
      • Gracefully handle empty headers and colspan/rowspan by flattening where possible or leaving the original HTML when fidelity would be lost.
  • Testing:
    • Add unit tests covering:
      • Inline code with <, >, {, }.
      • Documents with 2+ tables, including cells containing inline code.
      • Edge cases: unmatched backticks, nested code spans within tables, and no-table content (no-op).

Acceptance Criteria

  • Inline code inside single backticks renders unchanged after conversion; only non-code text is escaped.
  • All tables in a document are converted to Markdown or, if conversion would be lossy, original HTML is retained for those specific tables without breaking the rest of the page.
  • Alignment is preserved where specified; otherwise defaults are reasonable.
  • New tests pass and cover the scenarios above.
  • Running the cookbook build on existing samples with pandas tables no longer corrupts content.

Implementation Notes

  • For inline code detection, consider a lightweight state machine per line:
    • Toggle an in_inline_code flag when encountering backticks not within fenced blocks.
    • Perform escaping only when in_code_block == false and in_inline_code == false.
  • For tables, consider iterating with BeautifulSoup:
    • For each <table>, build a Markdown representation; replace the HTML table with Markdown, leaving surrounding content intact.
    • If a table uses complex rowspan/colspan that cannot be represented, skip conversion for that table and log a warning.

Additional Context

  • Current TODOs in code explicitly acknowledge these gaps and their effect on pandas tables:
    • “TODO: Handle single backticks … this will break some rendering of pandas tables … this is a quick fix for now”
  • Addressing this will improve the reliability of the docs build pipeline and reduce manual cleanup.
Originally created by @0x-Professor on GitHub (Aug 9, 2025). ## Summary The cookbook conversion script currently over-escapes characters outside fenced code blocks but does not respect inline code (single backticks), and the HTML table conversion only handles a single table with limited fidelity. This can corrupt inline code samples and mangle pandas-generated tables in converted content. Affected code: - `replace_brackets` does not skip inline code and notes it can break pandas tables. - `html_table_to_markdown` converts only the first table and loses alignment/formatting. References: - replace_brackets/html_table_to_markdown: https://github.com/langchain-ai/langsmith-docs/blob/614162ac3e51268c2795f93eb85b44045dedf202/subdirectories/scripts/build_cookbook.py#L256-L341 ## Impact - Inline code like `` `a < b` `` becomes `a &lt; b`, altering rendered content. - Pandas tables embedded as HTML may be partially or incorrectly converted, leading to unreadable docs. - These issues reduce trust in code examples and degrade documentation quality. ## Steps to Reproduce 1. Run the cookbook build script on markdown that includes: - Inline code with `<`, `>`, `{`, `}` (e.g., ``Use `a < b && c > d` in filters``). - A pandas DataFrame rendered to HTML with multiple `<table>` elements (e.g., with a header table and data table). 2. Observe that: - Inline code segments have their characters escaped instead of preserved. - Only the first table is converted to Markdown; subsequent tables are ignored, and cell formatting/alignment/code spans are lost. ## Proposed Solution - Inline code preservation: - Enhance `replace_brackets` to detect and skip transformations inside single-backtick code spans (`` `...` ``), in addition to fenced code blocks. - Strategy: tokenize lines by backtick pairs and only escape characters in non-code segments; keep existing fenced-code tracking. - Robust table conversion: - Update `html_table_to_markdown` to: - Iterate over all `<table>` elements within the document. - Preserve inline code within cells (do not escape backtick-enclosed spans). - Support basic alignment by detecting `<th align>`/`<td align>` or style attributes and mapping them to Markdown alignment (`:---`, `:---:`, `---:`). - Gracefully handle empty headers and colspan/rowspan by flattening where possible or leaving the original HTML when fidelity would be lost. - Testing: - Add unit tests covering: - Inline code with `<`, `>`, `{`, `}`. - Documents with 2+ tables, including cells containing inline code. - Edge cases: unmatched backticks, nested code spans within tables, and no-table content (no-op). ## Acceptance Criteria - Inline code inside single backticks renders unchanged after conversion; only non-code text is escaped. - All tables in a document are converted to Markdown or, if conversion would be lossy, original HTML is retained for those specific tables without breaking the rest of the page. - Alignment is preserved where specified; otherwise defaults are reasonable. - New tests pass and cover the scenarios above. - Running the cookbook build on existing samples with pandas tables no longer corrupts content. ## Implementation Notes - For inline code detection, consider a lightweight state machine per line: - Toggle an `in_inline_code` flag when encountering backticks not within fenced blocks. - Perform escaping only when `in_code_block == false` and `in_inline_code == false`. - For tables, consider iterating with BeautifulSoup: - For each `<table>`, build a Markdown representation; replace the HTML table with Markdown, leaving surrounding content intact. - If a table uses complex rowspan/colspan that cannot be represented, skip conversion for that table and log a warning. ## Additional Context - Current TODOs in code explicitly acknowledge these gaps and their effect on pandas tables: - “TODO: Handle single backticks … this will break some rendering of pandas tables … this is a quick fix for now” - Addressing this will improve the reliability of the docs build pipeline and reduce manual cleanup.
yindo closed this issue 2026-02-21 17:17:09 -05:00
yindo commented 2026-02-21 17:17:09 -05:00
Author
Owner
Copy Link
Copy Source

@katmayb commented on GitHub (Nov 18, 2025):

Closing this issue, as we've moved to a new repository and this is not relevant to the new setup. Thank you!

@katmayb commented on GitHub (Nov 18, 2025): Closing this issue, as we've moved to a [new repository](https://github.com/langchain-ai/docs/issues) and this is not relevant to the new setup. Thank you!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langsmith-docs#86
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?