Mintplex-Labs/tiktoken
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/tiktoken.git synced 2026-07-19 13:54:01 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 586d205b4f Fix README.md Tat Dat Duong 2023-03-11 02:20:35 +01:00
  • 2c7e0e43a6 Create a lite build which defers loading of weights to consumers Tat Dat Duong 2023-03-11 02:06:57 +01:00
  • 15dd0f254e Add caveats for CFW Tat Dat Duong 2023-03-10 20:00:21 +01:00
  • 9f9ad0d476 Merge pull request #13 from dqbd/gpt-3.5-turbo David Duong 2023-03-08 02:53:51 +01:00
  • 268bc5c7ee Add README.md Tat Dat Duong 2023-03-08 02:03:47 +01:00
  • c1d11fb191 Add gpt-3.5-turbo support in types and matchers Tat Dat Duong 2023-03-08 01:59:52 +01:00
  • 7c75b0463f Merge remote-tracking branch 'upstream/main' Tat Dat Duong 2023-03-08 01:53:57 +01:00
  • 680fbc5562 Update README.md Tat Dat Duong 2023-03-08 01:49:26 +01:00
  • 9ea421b1b7 Merge pull request #12 from dqbd/async-init-bundler David Duong 2023-03-08 01:46:22 +01:00
  • 945d4f2124 Reverse order of default Tat Dat Duong 2023-03-08 01:38:31 +01:00
  • fcb52cf097 Rename TiktokenEmbedding to TiktokenEncoding Tat Dat Duong 2023-03-08 01:34:00 +01:00
  • 7c8cd78300 Run WASM build sequentially in CI for now Tat Dat Duong 2023-03-08 01:33:13 +01:00
  • 03cdbb3ad1 Flatten the structure Tat Dat Duong 2023-03-08 01:25:03 +01:00
  • 33bb13d3f3 Add custom initialisation, bundler changes Tat Dat Duong 2023-03-07 21:56:47 +01:00
  • 4866cf9fa1 Optimize performace, enable/disable features of core Tat Dat Duong 2023-03-06 00:38:44 +01:00
  • 8e4068201b Revert "debug ci" Tat Dat Duong 2023-03-05 19:22:51 +01:00
  • dabb296e61 invalid -y command Tat Dat Duong 2023-03-05 19:08:35 +01:00
  • ae30c13eba Fix CI again Tat Dat Duong 2023-03-05 19:01:24 +01:00
  • 12805e3724 Try CI again Tat Dat Duong 2023-03-05 18:58:09 +01:00
  • 08403a1445 debug ci Tat Dat Duong 2023-03-05 18:50:15 +01:00
  • 7db26cba18 Try yum once again Tat Dat Duong 2023-03-05 17:59:47 +01:00
  • 82cd4139ee Replace yum with apt Tat Dat Duong 2023-03-05 17:54:51 +01:00
  • 01bf979f58 Fix Python CI Tat Dat Duong 2023-03-05 17:47:05 +01:00
  • d989c22e4d CI: install initialize wasm-pack Tat Dat Duong 2023-03-05 16:57:22 +01:00
  • 98ac9530ea Add CI step to build and test Tat Dat Duong 2023-03-05 16:52:26 +01:00
  • 2370284c5c Fix wrong result for None Tat Dat Duong 2023-03-05 16:52:10 +01:00
  • ec7c121e38 Bump version, sync codebase Shantanu Jain 2023-03-02 11:27:55 -08:00
  • 33207e6c4f Update to newer build wheels Tat Dat Duong 2023-03-02 15:29:01 +01:00
  • d1c4af2777 Remove the _js suffix Tat Dat Duong 2023-03-02 15:23:37 +01:00
  • bbcb5914f7 Match sdist Tat Dat Duong 2023-03-02 15:14:59 +01:00
  • b04f0cf786 Attempt to fix sdist Tat Dat Duong 2023-03-02 15:10:32 +01:00
  • ef77b1a2ea Make sure JS builds Tat Dat Duong 2023-03-01 23:36:33 +01:00
  • 4d8f9afc83 Merge branch 'eisber/main' Tat Dat Duong 2023-03-01 23:16:33 +01:00
  • 01d4f9e88b Move to separate js folder Tat Dat Duong 2023-03-01 23:13:42 +01:00
  • f1560bd3b5 remove comments Markus Cozowicz 2023-02-28 17:26:38 +01:00
  • 381d32b69d update groupid Markus Cozowicz 2023-02-28 17:19:20 +01:00
  • 4e70da002e cleanup Markus Cozowicz 2023-02-28 17:17:16 +01:00
  • 3b39f6b1c5 Merge remote-tracking branch 'upstream/main' into main Markus Cozowicz 2023-02-27 14:44:48 +01:00
  • 98070f1374 fix java Markus Cozowicz 2023-02-27 13:28:26 +01:00
  • b883d0c5ea fix path add matrix java test Markus Cozowicz 2023-02-27 13:21:33 +01:00
  • 6c45bf698a build jar Markus Cozowicz 2023-02-27 13:13:20 +01:00
  • 484f15a7a7 fix rust build Markus Cozowicz 2023-02-27 12:59:32 +01:00
  • 42efde424e add jar build Markus Cozowicz 2023-02-27 12:58:16 +01:00
  • 22584d4eb3 add github action Markus Cozowicz 2023-02-27 12:54:59 +01:00
  • e3ab3f643e moved config into json Markus Cozowicz 2023-02-27 12:17:37 +01:00
  • 42548d8353 Merge pull request #1 from eisber/error_handling Markus Cozowicz 2023-02-27 08:51:11 +01:00
  • f5fbc9c5e9 Expose p50k_edit encoding (#32) Fritz Obermeyer 2023-02-26 14:36:53 -05:00
  • fbaa86e0f0 Sync codebase Shantanu Jain 2023-02-25 21:01:46 -08:00
  • c4b8770184 Improve performance of byte_pair_merge (#31) Nick Stathas 2023-02-25 23:37:35 -05:00
  • 87603e9813 Bump version to 0.4.0 Tat Dat Duong 2023-02-25 00:21:23 +01:00
  • 53628a47a2 Improve error handling in JNI functions Jack Gerrits 2023-02-24 10:41:35 -05:00
  • d38c936f17 Validate the values properly Tat Dat Duong 2023-02-24 14:24:17 +01:00
  • f7fe7174b0 Update README.md, add tests, fix disallowed special bug Tat Dat Duong 2023-02-24 14:07:07 +01:00
  • 90ee9f69fb Fix any in TS files, add core tests Tat Dat Duong 2023-02-24 00:43:29 +01:00
  • d0dc32bb6d Improve error handling, add support for parameters Tat Dat Duong 2023-02-23 23:31:08 +01:00
  • 3a39f24764 Bump version, update README Tat Dat Duong 2023-02-23 23:28:19 +01:00
  • 46668dfabe Merge pull request #2 from dqbd/extendability David Duong 2023-02-23 16:46:29 +01:00
  • 02d132e941 feat: add option to extend special tokens and to provide custom bfe Tat Dat Duong 2023-02-23 12:28:47 +01:00
  • 3139d04e12 chore: add README Tat Dat Duong 2023-02-23 04:13:01 +01:00
  • 97d8dea88f improve error handling Markus Cozowicz 2023-02-22 16:19:50 +01:00
  • 5a32b88981 Update README.md, polish API Tat Dat Duong 2023-02-19 15:30:38 +01:00
  • 799df7fe86 v0.2.1 Tat Dat Duong 2023-02-19 15:28:14 +01:00
  • 95aef95cb3 Add wasm-bindgen, inline ranks Tat Dat Duong 2023-02-19 14:32:49 +01:00
  • 1d3f707a27 Initial VERY rough impl of JNI layer Jack Gerrits 2023-02-10 22:09:29 -05:00
  • 2f371f60f7 more TODO Markus Cozowicz 2023-02-10 23:05:29 +01:00
  • f248694fa9 java bindings... Markus Cozowicz 2023-02-10 22:50:13 +01:00
  • c98f8240ed Fix python setup.py for package in workspace Jack Gerrits 2023-02-10 14:34:53 -05:00
  • ba69bd915d fixed core Markus Cozowicz 2023-02-10 20:30:23 +01:00
  • 25306a5a5b split into multiple libs Markus Cozowicz 2023-02-10 20:21:27 +01:00
  • 6dcfa91364 return ref Jack Gerrits 2023-02-09 14:52:59 -05:00
  • c685d9982c moving registry into rust Markus Cozowicz 2023-02-09 18:19:25 +01:00
  • 502a6ec00e Propagate errors from split whitespace in bpe_merges Jack Gerrits 2023-02-09 10:21:57 -05:00
  • 4c6afcb314 fixed? Markus Cozowicz 2023-02-09 16:12:16 +01:00
  • 09dada409a ported loading to Rust Markus Cozowicz 2023-02-09 15:44:02 +01:00
  • 6c49a4129c refactor Markus Cozowicz 2023-02-08 11:32:33 +01:00
  • 7830ed537b Bump version, sync codebase Shantanu Jain 2023-02-03 11:46:09 -08:00
  • 156eff92d2 Add link to OpenAI Cookbook example (#20) Ted Sanders 2023-01-23 11:48:52 -08:00
  • cf385cada0 Fix docstring, type annotation for private method (#19) Arvid Lunnemark 2023-01-19 14:51:15 -08:00
  • 40d9b1f14e Update codebase Shantanu Jain 2023-01-03 13:20:35 -08:00
  • 0f8ec705e2 Remove redundant word in docstring (#7) Nguyen-Khanh Vu 2022-12-17 03:41:08 +01:00
  • 4226a6c008 Remove duplicate words in docstring (#3) Henrik Torget 2022-12-16 22:55:30 +01:00
  • ab3688a401 README.md: minor improvements Shantanu Jain 2022-12-16 03:26:13 -06:00
  • 1f098ca4d7 Build wheels; update codebase Shantanu Jain 2022-12-14 17:02:16 -08:00
  • a1a9f16826 [tiktoken] hello world Shantanu Jain 2022-12-12 11:27:27 -08:00