hsuyuming 630ae24b28 implement get_num_tokens to use google's count_tokens function (#10565) ...

can get the correct token count instead of using gpt-2 model

**Description:** 
Implement get_num_tokens within VertexLLM to use google's count_tokens
function.
(https://cloud.google.com/vertex-ai/docs/generative-ai/get-token-count).
So we don't need to download gpt-2 model from huggingface, also when we
do the mapreduce chain we can get correct token count.

**Tag maintainer:** 
@lkuligin 
**Twitter handle:** 
My twitter: @abehsu1992626

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>

2023-10-30 15:10:05 -07:00

..

cli

change project template (#12493)

2023-10-30 14:06:30 -07:00

experimental

bump to 36 (#12487)

2023-10-28 08:51:23 -07:00

langchain

implement get_num_tokens to use google's count_tokens function (#10565)

2023-10-30 15:10:05 -07:00