Files
David Duong d60eae5995 Use a JS based tokenizer for token counting (#1239)
* Use a rewritten tiktoken for token counting

* Fix lint, add test

* Fix type import, typo

* Remove @dqbd/tiktoken as "weak" dependency

* Move JS implementation of tiktoken into a separate package

* Upgrade to js-tiktoken@1.0.2

* Replace all occurrences of @dqbd/tiktoken

* Avoid bundling ranks, fix test with CRA

* Use AsyncCaller, cache promises

* Remove WASM configuration changes

* Upgrade to 1.0.6 for CRA fixes, mark js-tiktoken/lite as an external dependency

* Remove next.config.js
2023-05-17 14:44:29 +01:00

20 lines
343 B
JSON

{
"name": "test-exports-vite",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview",
"test": "tsc"
},
"dependencies": {
"langchain": "workspace:*"
},
"devDependencies": {
"typescript": "^5.0.0",
"vite": "^4.2.0"
}
}