ollama/ollama-python
1
0
Fork 0
mirror of https://github.com/ollama/ollama-python.git synced 2026-07-21 09:05:23 -04:00
Code Issues 101 Packages Projects Releases 10 Wiki Activity

Batching #9

New Issue
Closed
opened 2026-02-15 16:27:25 -05:00 by yindo · 3 comments
yindo commented 2026-02-15 16:27:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @varunshenoy on GitHub (Jan 25, 2024).

Is there any way to batch prompts with local models using this package? Thank you!

Originally created by @varunshenoy on GitHub (Jan 25, 2024). Is there any way to batch prompts with local models using this package? Thank you!
yindo closed this issue 2026-02-15 16:27:25 -05:00
yindo commented 2026-02-15 16:27:25 -05:00
Author
Owner
Copy Link
Copy Source

@mxyng commented on GitHub (Jan 25, 2024):

Can you describe what you're trying to do? I'm not sure I understand

@mxyng commented on GitHub (Jan 25, 2024): Can you describe what you're trying to do? I'm not sure I understand
yindo commented 2026-02-15 16:27:25 -05:00
Author
Owner
Copy Link
Copy Source

@varunshenoy commented on GitHub (Jan 25, 2024):

Suppose I have a list of prompts: ["A list of colors: red, blue", "Geckos eat", "The capital of France is"]

You can run these all through an LLM in a single forward pass with a little to no latency hit due to the inherent parallelism in transformers. Mainly used for inference servers with multiple people hitting it at the same time, but I want to see if I can speed up some local workflows.

Here's some documentation from Hugging Face's transformers library that explains how they support batching:
SCR-20240125-lbxs

@varunshenoy commented on GitHub (Jan 25, 2024): Suppose I have a list of prompts: ["A list of colors: red, blue", "Geckos eat", "The capital of France is"] You can run these all through an LLM in a single forward pass with a little to no latency hit due to the inherent parallelism in transformers. Mainly used for inference servers with multiple people hitting it at the same time, but I want to see if I can speed up some local workflows. Here's some documentation from Hugging Face's `transformers` library that explains how they support batching: <img width="905" alt="SCR-20240125-lbxs" src="https://github.com/ollama/ollama-python/assets/10859091/c82b6f01-73bb-4acd-ae66-1a3d70945dce">
yindo commented 2026-02-15 16:27:25 -05:00
Author
Owner
Copy Link
Copy Source

@mxyng commented on GitHub (Jan 26, 2024):

This issue would be more appropriate for ollama since this repo is only the python interface to the ollama API.

@mxyng commented on GitHub (Jan 26, 2024): This issue would be more appropriate for [ollama](https://github.com/ollama/ollama) since this repo is only the python interface to the ollama API.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
pull-request
Mirrored from GitHub Pull Request
 thinking
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ollama/ollama-python#9
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?