run-llama/llama_cloud_services
1
0
Fork 0
mirror of https://github.com/run-llama/llama_cloud_services.git synced 2026-08-26 16:56:36 -04:00
Code Issues 352 Packages Projects Releases 10 Wiki Activity

MultiModalVectorStoreIndex retriever index out of bounds #496

New Issue
Open
opened 2026-02-16 00:18:01 -05:00 by yindo · 3 comments
yindo commented 2026-02-16 00:18:01 -05:00
Owner
Copy Link
Copy Source

Originally created by @abusch472 on GitHub (Jul 3, 2025).

Hello,

I am using the Llamaparse API with pinecone as my vector database to parse a few pdf documents. I have been using the MultiModalVectorStoreIndex with BAAI/bge-large-en-v1.5 as my text encoder and llamaindex/vdr-2b-multi-v1 as my image encoder. (using this notebook as reference)

When I run retrieval through text_retrieve() it works as expected, but when I run text_to_image_retrieve() it fails with error:

File "/usr/local/lib/python3.10/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py", line 161, in __call__
    num_image_tokens = image_grid_thw[index].prod() // merge_length
IndexError: index 1 is out of bounds for dimension 0 with size 1.

I read that the qwen model throws this error when the number of image tokens isn't the same as the number of images, but I'm not familiar enough with the LlamaIndex to know exactly why this would be happening.

My next step will be to try using a different embeddings model. Happy to provide any additional context as needed.

Thank you for your help!

Originally created by @abusch472 on GitHub (Jul 3, 2025). Hello, I am using the Llamaparse API with pinecone as my vector database to parse a few pdf documents. I have been using the MultiModalVectorStoreIndex with BAAI/bge-large-en-v1.5 as my text encoder and llamaindex/vdr-2b-multi-v1 as my image encoder. (using [this](https://github.com/run-llama/llama_cloud_services/blob/main/examples/parse/demo_advanced.ipynb) notebook as reference) When I run retrieval through text_retrieve() it works as expected, but when I run text_to_image_retrieve() it fails with error: ``` File "/usr/local/lib/python3.10/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py", line 161, in __call__ num_image_tokens = image_grid_thw[index].prod() // merge_length IndexError: index 1 is out of bounds for dimension 0 with size 1. ``` I [read ](https://github.com/huggingface/transformers/issues/35254) that the qwen model throws this error when the number of image tokens isn't the same as the number of images, but I'm not familiar enough with the LlamaIndex to know exactly why this would be happening. My next step will be to try using a different embeddings model. Happy to provide any additional context as needed. Thank you for your help!
yindo commented 2026-02-16 00:18:01 -05:00
Author
Owner
Copy Link
Copy Source

@logan-markewich commented on GitHub (Jul 3, 2025):

Hmm spooky. If you can reproduce this it would be very helpful!

Specifically, you should be to reproduce this using some code like this

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

image_embed_model = HuggingFaceEmbedding(
    model_name="llamaindex/vdr-2b-multi-v1",
    embed_batch_size=2,
    ...
)

text_embedding = image_embed_model.get_text_embedding("some text")
image_embed = image_embed_model.get_image_embedding("./file.png")

if this reproduces, if you can share the file you used, thatd be super helpful

@logan-markewich commented on GitHub (Jul 3, 2025): Hmm spooky. If you can reproduce this it would be very helpful! Specifically, you should be to reproduce this using some code like this ``` from llama_index.embeddings.huggingface import HuggingFaceEmbedding image_embed_model = HuggingFaceEmbedding( model_name="llamaindex/vdr-2b-multi-v1", embed_batch_size=2, ... ) text_embedding = image_embed_model.get_text_embedding("some text") image_embed = image_embed_model.get_image_embedding("./file.png") ``` if this reproduces, if you can share the file you used, thatd be super helpful
yindo commented 2026-02-16 00:18:02 -05:00
Author
Owner
Copy Link
Copy Source

@shuaib7860 commented on GitHub (Jul 31, 2025):

Hi I'm running into the same problem and the issue does not reproduce when I follow the above code. The issue only seems to manifest when utilising a multimodal query engine. My code is below as well as the error I see.

mm_model = OllamaMultiModal(model="gemma3:27b", request_timeout=100.0)
image_embed_model = HuggingFaceEmbedding(model_name="llamaindex/vdr-2b-v1", device='cpu', trust_remote_code=True)
Settings.embed_model = image_embed_model

query_engine = index.as_query_engine(llm=mm_model, text_qa_template=qa_tmpl)
query_str = "What does a base station consist of?"
response = query_engine.query(query_str)

The error manifests as:

File ~/Documents/MultiModal/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py:139, in Qwen2VLProcessor.call(self, images, text, videos, **kwargs)
136 for i in range(len(text)):
137 while self.image_token in text[i]:
138 text[i] = text[i].replace(
--> 139 self.image_token, "<|placeholder|>" * (image_grid_thw[index].prod() // merge_length), 1
140 )
141 index += 1
142 text[i] = text[i].replace("<|placeholder|>", self.image_token)

IndexError: index 1 is out of bounds for dimension 0 with size 1

Weirdly it seems to be assuming an image has been provided into the query as default even though I have not provided any image in my query and it just a simple text query. Any help would be greatly appreciated.

@shuaib7860 commented on GitHub (Jul 31, 2025): Hi I'm running into the same problem and the issue does not reproduce when I follow the above code. The issue only seems to manifest when utilising a multimodal query engine. My code is below as well as the error I see. mm_model = OllamaMultiModal(model="gemma3:27b", request_timeout=100.0) image_embed_model = HuggingFaceEmbedding(model_name="llamaindex/vdr-2b-v1", device='cpu', trust_remote_code=True) Settings.embed_model = image_embed_model query_engine = index.as_query_engine(llm=mm_model, text_qa_template=qa_tmpl) query_str = "What does a base station consist of?" response = query_engine.query(query_str) The error manifests as: File ~/Documents/MultiModal/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py:139, in Qwen2VLProcessor.__call__(self, images, text, videos, **kwargs) 136 for i in range(len(text)): 137 while self.image_token in text[i]: 138 text[i] = text[i].replace( --> [139](https://file+.vscode-resource.vscode-cdn.net/Users/shuaib.choudhry/Documents/MultiModal/~/Documents/MultiModal/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/processing_qwen2_vl.py:139) self.image_token, "<|placeholder|>" * (image_grid_thw[index].prod() // merge_length), 1 140 ) 141 index += 1 142 text[i] = text[i].replace("<|placeholder|>", self.image_token) IndexError: index 1 is out of bounds for dimension 0 with size 1 Weirdly it seems to be assuming an image has been provided into the query as default even though I have not provided any image in my query and it just a simple text query. Any help would be greatly appreciated.
yindo commented 2026-02-16 00:18:02 -05:00
Author
Owner
Copy Link
Copy Source

@abusch472 commented on GitHub (Jul 31, 2025):

Hey,

I ended up just keeping a separate index for text and image, as I couldn't figure out the multimodalvector store. Then, during retrieval I used the following lines of code to retrieve from the vector stores.

        # generate your embeddings according to your specific model interface

        text_query = VectorStoreQuery(
            query_embedding = text_query_embedding,
            similarity_top_k=self.text_top_k
        )

        image_query = VectorStoreQuery(
            query_embedding=image_query_embedding,
            similarity_top_k=self.img_top_k
        )

        text_results = self.text_vector_store.query(text_query)

        image_results = self.image_vector_store.query(image_query)

Hope this helps!

@abusch472 commented on GitHub (Jul 31, 2025): Hey, I ended up just keeping a separate index for text and image, as I couldn't figure out the multimodalvector store. Then, during retrieval I used the following lines of code to retrieve from the vector stores. ``` # generate your embeddings according to your specific model interface text_query = VectorStoreQuery( query_embedding = text_query_embedding, similarity_top_k=self.text_top_k ) image_query = VectorStoreQuery( query_embedding=image_query_embedding, similarity_top_k=self.img_top_k ) text_results = self.text_vector_store.query(text_query) image_results = self.image_vector_store.query(image_query) ``` Hope this helps!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
LlamaCloud
LlamaExtract
LlamaParse
bug
client
complex layout
documentation
duplicate
enhancement
figure
font
header footer
help wanted
ocr
parser
pdf_debug
pdf_debug
pull-request
Mirrored from GitHub Pull Request
 python-lib
question
server_issue
server_issue
ui
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/llama_cloud_services#496
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?