langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-23 12:55:40 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[GH-ISSUE #1672] [langchain]: Gaps in multimodal documentation + samples (image / file / ....) #223

New Issue
Open
opened 2026-02-17 17:19:26 -05:00 by yindo · 0 comments
yindo commented 2026-02-17 17:19:26 -05:00
Owner
Copy Link
Copy Source

Originally created by @ddewaele on GitHub (Nov 30, 2025).
Original GitHub issue: https://github.com/langchain-ai/docs/issues/1672

Type of issue

issue / bug

Language

JavaScript

Description

Some remarks on the MultiModal in the Messages section of the documentation :

Code Samples

The following code samples don't work

Image input :

Code Sample

// From base64 data
const message = new HumanMessage({
  content: [
    { type: "text", text: "Describe the content of this image." },
    {
      type: "image",
      source_type: "base64",
      data: "AAAAIGZ0eXBtcDQyAAAAAGlzb21tcDQyAAACAGlzb2...",
    },
  ],
});

Errors

On both Claude and OpenAI the image content will fail

  • Claude : BadRequestError: 400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'"},"request_id":"req_011CVeFwdRwbUZXG1oMB1jY3"}
  • OpenAI : BadRequestError: 400 Invalid MIME type. Only image types are supported.

Fixes :

  • On both models you need to provide a mime_type field in the content block
mime_type:"image/jpeg"

File input

Code Sample

// From base64 data
const message = new HumanMessage({
  content: [
    { type: "text", text: "Describe the content of this document." },
    {
      type: "file",
      source_type: "base64",
      data: "AAAAIGZ0eXBtcDQyAAAAAGlzb21tcDQyAAACAGlzb2...",
    },
  ],
});

Errors

On OpenAI the pdf content will fail. On Claude it works out of the box

  • Claude : works fine
  • OpenAI : BadRequestError: 400 Missing required parameter: 'messages[0].content[1].file.file_id'.

On OpenAI if you need to add

mime_type: "application/pdf",
metadata: { filename: "file.pdf" },

This is not at all clear from the docs.
There are some "hints" in the documentation that are a bit fuzzy :

Extra keys can be included top-level in the content block or nested in "extras": {"key": value}.
OpenAI and AWS Bedrock Converse, for example, require a filename for PDFs. See the provider page for your chosen model for specifics.

But nowhere in the integration docs for both providers

does it mention these subtleties (the mime_type requirement + the filename requirement). The #pdf anchor also seems to be missing.

I would be more than happy to write up a PR for this to make the docs more explicit and give some better samples. But it's unclear what we want to achieve with the code samples.

  • Should the content sections of the input messages work on all (multi-)model providers ?
  • Is there some guidance on what developers can put in these "standard" LangChain content sections. (what source_type values do we support, what source values do we support, ... ?)
  • Do you want to guarantee they work on a particular model provider (and make that explicit in the docs).
  • What do you mean with this extras field that you can provide (it doesn't seem to work when I tried to add the mime-type)
  • Should all these provider specific details be mentioned in the integration docs ?
  • Do we want to promote the use of contentBlock vs content fields when creating messages ?
Originally created by @ddewaele on GitHub (Nov 30, 2025). Original GitHub issue: https://github.com/langchain-ai/docs/issues/1672 ### Type of issue issue / bug ### Language JavaScript ### Description Some remarks on the [MultiModal in the Messages section](https://docs.langchain.com/oss/javascript/langchain/messages#multimodal) of the documentation : ## Code Samples The following code samples don't work ### Image input : #### Code Sample ``` // From base64 data const message = new HumanMessage({ content: [ { type: "text", text: "Describe the content of this image." }, { type: "image", source_type: "base64", data: "AAAAIGZ0eXBtcDQyAAAAAGlzb21tcDQyAAACAGlzb2...", }, ], }); ``` #### Errors On both Claude and OpenAI the image content will fail - Claude : `BadRequestError: 400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'"},"request_id":"req_011CVeFwdRwbUZXG1oMB1jY3"} ` - OpenAI : `BadRequestError: 400 Invalid MIME type. Only image types are supported.` #### Fixes : - On both models you need to provide a `mime_type` field in the content block ``` mime_type:"image/jpeg" ``` ### File input #### Code Sample ``` // From base64 data const message = new HumanMessage({ content: [ { type: "text", text: "Describe the content of this document." }, { type: "file", source_type: "base64", data: "AAAAIGZ0eXBtcDQyAAAAAGlzb21tcDQyAAACAGlzb2...", }, ], }); ``` #### Errors On OpenAI the pdf content will fail. On Claude it works out of the box - Claude : works fine - OpenAI : `BadRequestError: 400 Missing required parameter: 'messages[0].content[1].file.file_id'.` On OpenAI if you need to add ``` mime_type: "application/pdf", metadata: { filename: "file.pdf" }, ``` This is not at all clear from the docs. There are some "hints" in the documentation that are a bit fuzzy : _Extra keys can be included top-level in the content block or nested in "extras": {"key": value}. [OpenAI](https://docs.langchain.com/oss/javascript/integrations/chat/openai#pdfs) and [AWS Bedrock Converse](https://docs.langchain.com/oss/javascript/integrations/chat/bedrock), for example, require a filename for PDFs. See the [provider page](https://docs.langchain.com/oss/javascript/integrations/providers/overview) for your chosen model for specifics._ But nowhere in the integration docs for both providers - https://docs.langchain.com/oss/javascript/integrations/chat/anthropic - https://docs.langchain.com/oss/javascript/integrations/chat/openai does it mention these subtleties (the mime_type requirement + the filename requirement). The `#pdf` anchor also seems to be missing. I would be more than happy to write up a PR for this to make the docs more explicit and give some better samples. But it's unclear what we want to achieve with the code samples. - Should the content sections of the input messages work on all (multi-)model providers ? - Is there some guidance on what developers can put in these "standard" LangChain `content` sections. (what `source_type` values do we support, what `source` values do we support, ... ?) - Do you want to guarantee they work on a particular model provider (and make that explicit in the docs). - What do you mean with this `extras` field that you can provide (it doesn't seem to work when I tried to add the mime-type) - Should all these provider specific details be mentioned in the integration docs ? - Do we want to promote the use of `contentBlock` vs `content` fields when creating messages ?
yindo added the langchainexternal labels 2026-02-17 17:19:26 -05:00
yindo changed title from [langchain]: Gaps in multimodal documentation + samples (image / file / ....) to [GH-ISSUE #1672] [langchain]: Gaps in multimodal documentation + samples (image / file / ....) 2026-06-05 17:25:43 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label external langchain
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#223
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?