Taqi Jaffri b7290f01d8 Batching for hf_pipeline (#10795) ...

The huggingface pipeline in langchain (used for locally hosted models)
does not support batching. If you send in a batch of prompts, it just
processes them serially using the base implementation of _generate:
https://github.com/docugami/langchain/blob/master/libs/langchain/langchain/llms/base.py#L1004C2-L1004C29

This PR adds support for batching in this pipeline, so that GPUs can be
fully saturated. I updated the accompanying notebook to show GPU batch
inference.

---------

Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>

2023-09-25 18:23:11 +01:00

..

_scripts

llm feat table revision (#10947)

2023-09-22 10:29:12 -07:00

api_reference

add model feat table (#10921)

2023-09-22 01:10:27 -07:00

docs_skeleton

redirect mrkl (#10979)

2023-09-23 16:09:13 -07:00

extras

Batching for hf_pipeline (#10795)

2023-09-25 18:23:11 +01:00

snippets

JSONLoader Documentation Fix (#10505)

2023-09-21 11:37:40 -07:00

.local_build.sh

Update local script for docs build (#8377)

2023-07-27 13:13:59 -07:00

vercel_requirements.txt

Add api cross ref linking (#8275)

2023-07-26 12:38:58 -07:00