16 Commits

Author	SHA1	Message	Date
Tat Dat Duong	67d21c3c99	Move WASM bindings to wasm folder	2023-05-12 22:01:08 +02:00
Tat Dat Duong	754a378c1e	Remove Java bindings	2023-05-08 16:58:01 +02:00
Tat Dat Duong	4866cf9fa1	Optimize performace, enable/disable features of core	2023-03-06 00:38:44 +01:00
Tat Dat Duong	ef77b1a2ea	Make sure JS builds	2023-03-01 23:36:33 +01:00
Tat Dat Duong	4d8f9afc83	Merge branch 'eisber/main'	2023-03-01 23:16:33 +01:00
Markus Cozowicz	22584d4eb3	add github action	2023-02-27 12:54:59 +01:00
Tat Dat Duong	f7fe7174b0	Update README.md, add tests, fix disallowed special bug	2023-02-24 14:07:07 +01:00
Tat Dat Duong	02d132e941	feat: add option to extend special tokens and to provide custom bfe	2023-02-23 12:28:47 +01:00
Tat Dat Duong	95aef95cb3	Add wasm-bindgen, inline ranks	2023-02-19 14:32:49 +01:00
Markus Cozowicz	f248694fa9	java bindings...	2023-02-10 22:50:13 +01:00
Markus Cozowicz	25306a5a5b	split into multiple libs	2023-02-10 20:21:27 +01:00
Markus Cozowicz	c685d9982c	moving registry into rust	2023-02-09 18:19:25 +01:00
Markus Cozowicz	09dada409a	ported loading to Rust	2023-02-09 15:44:02 +01:00
Markus Cozowicz	6c49a4129c	refactor	2023-02-08 11:32:33 +01:00
Shantanu Jain	7830ed537b	Bump version, sync codebase	2023-02-03 12:35:59 -08:00
Shantanu Jain	a1a9f16826	[tiktoken] hello world	2022-12-14 15:33:00 -08:00