Mintplex-Labs/tiktoken
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/tiktoken.git synced 2026-07-19 22:03:47 -04:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • b75f172d51 Merge pull request #53 from kdwkr/patch-1 David Duong 2023-06-23 00:51:16 +02:00
  • 1bc9ada93e Update lib.rs Daewoon Kim 2023-06-16 11:01:13 +09:00
  • 5a7be3cb5b feat: Add missing models Daewoon Kim 2023-06-16 10:52:22 +09:00
  • 0d7e051256 Merge pull request #52 from dqbd/changeset-release/main David Duong 2023-06-15 22:53:33 +02:00
  • a25079d192 Version Packages github-actions[bot] 2023-06-15 13:11:40 +00:00
  • baca5b1e38 Add changeset Tat Dat Duong 2023-06-15 15:10:27 +02:00
  • 844343f92f Merge remote-tracking branch 'origin/main' Tat Dat Duong 2023-06-15 15:05:24 +02:00
  • d1f8f9f44e Merge pull request #51 from rodumani/main David Duong 2023-06-15 15:03:42 +02:00
  • a5fb1f4ccc Merge remote-tracking branch 'upstream/main' Tat Dat Duong 2023-06-15 15:01:42 +02:00
  • ebfc814084 Add new models Changje Jeong 2023-06-14 23:23:41 +09:00
  • 5d970c1100 Use future annotations in tiktoken._educational (#143) youkaichao 2023-06-12 16:48:34 +08:00
  • 95b03bfd60 Fix up cibuildwheel Shantanu Jain 2023-06-12 00:35:54 -07:00
  • c373d9b9c3 Sync codebase Shantanu Jain 2023-06-07 15:34:08 -07:00
  • c7faff0e33 Update README.md Tat Dat Duong 2023-06-02 18:09:42 +02:00
  • c4a61809d5 Clarify the difference between js-tiktoken and tiktoken Tat Dat Duong 2023-06-02 18:07:50 +02:00
  • 3f2c59c6bd Add disclaimer about Svelte + Cloudflare Workers Tat Dat Duong 2023-06-02 18:02:52 +02:00
  • affbd6e50d Sync codebase Shantanu Jain 2023-05-31 20:03:03 -07:00
  • 1fcb052df8 Bump to 1.0.6 Tat Dat Duong 2023-05-17 14:29:45 +02:00
  • 4d3f97f524 Bump js-tiktoken to 1.0.5 Tat Dat Duong 2023-05-15 22:12:59 +02:00
  • 7fbd89944c Bump to 1.0.4 Tat Dat Duong 2023-05-15 20:09:46 +02:00
  • 265e01df64 Expose ranks Tat Dat Duong 2023-05-15 20:08:03 +02:00
  • 97fe465343 Merge pull request #42 from dqbd/changeset-release/main David Duong 2023-05-15 19:54:03 +02:00
  • dba22a9f28 Version Packages github-actions[bot] 2023-05-15 17:53:33 +00:00
  • de929d49f8 Add disallowed tokens Tat Dat Duong 2023-05-15 19:52:44 +02:00
  • 621ae351c2 Add missing main and types for compat Tat Dat Duong 2023-05-15 02:56:41 +02:00
  • 1da0b9d45f Bump js-tiktoken to 1.0.1 Tat Dat Duong 2023-05-15 02:18:29 +02:00
  • 2d18fcc4fd Expose model to encoding mapping, fix types Tat Dat Duong 2023-05-15 02:12:23 +02:00
  • dd6a6ef1f5 Merge pull request #40 from rpidanny/fix-typo David Duong 2023-05-15 01:53:31 +02:00
  • 3240a8e486 Split into smaller files Tat Dat Duong 2023-05-15 01:52:48 +02:00
  • 14f108c3d5 fix: typo Abhishek Maharjan 2023-05-14 08:55:09 +02:00
  • 975ed0f3ac Rename package once again Tat Dat Duong 2023-05-13 13:50:54 +02:00
  • 7c6a304792 Normalise workflows Tat Dat Duong 2023-05-13 11:53:33 +02:00
  • 6b00942fb7 Add publish CI step Tat Dat Duong 2023-05-13 11:48:39 +02:00
  • f245e69302 Rename description Tat Dat Duong 2023-05-13 11:29:09 +02:00
  • 81bd68934b Rename js to tiktoken-js, rename wasm to tiktoken Tat Dat Duong 2023-05-13 11:25:41 +02:00
  • 990d892517 Fix sdist Tat Dat Duong 2023-05-13 10:44:13 +02:00
  • 3debe21bea Add CDN hosting of ranks Tat Dat Duong 2023-05-13 03:26:48 +02:00
  • 184554893c More descriptive name Tat Dat Duong 2023-05-13 03:00:36 +02:00
  • 1bf31d3125 Fix actions Tat Dat Duong 2023-05-13 02:53:34 +02:00
  • 7e5ab3529d Monorepo changes Tat Dat Duong 2023-05-13 02:46:29 +02:00
  • 6b13f93ba8 Generate ranks Tat Dat Duong 2023-05-13 02:00:33 +02:00
  • fdf6121428 Rename WASM workflows Tat Dat Duong 2023-05-12 22:23:12 +02:00
  • 150cf05ac3 Add JS port Tat Dat Duong 2023-05-12 22:15:58 +02:00
  • 67d21c3c99 Move WASM bindings to wasm folder Tat Dat Duong 2023-05-12 22:01:08 +02:00
  • 243c061fa9 Add changesets folder Tat Dat Duong 2023-05-10 11:56:24 +02:00
  • 83de511cc7 Add weak refs Tat Dat Duong 2023-05-10 11:25:58 +02:00
  • b8e7817b52 Remove duplicate README.md Tat Dat Duong 2023-05-08 17:51:49 +02:00
  • 9390c03894 Replace README.md for clarity Tat Dat Duong 2023-05-08 17:01:56 +02:00
  • fe721cd038 Merge remote-tracking branch 'upstream/main' Tat Dat Duong 2023-05-08 16:59:13 +02:00
  • 754a378c1e Remove Java bindings Tat Dat Duong 2023-05-08 16:58:01 +02:00
  • 095924e02c Bump version, sync codebase Shantanu Jain 2023-05-07 13:23:25 -07:00
  • ea11d75dba Bump to 1.0.7 Tat Dat Duong 2023-04-28 13:38:51 +02:00
  • a54dd5746a Publish 1.0.7-alpha.0 build Tat Dat Duong 2023-04-28 13:18:18 +02:00
  • 33405e996c Use override for edge-light / NextJS appHandlers Tat Dat Duong 2023-04-28 13:16:36 +02:00
  • f19feecd07 Sync codebase Shantanu Jain 2023-04-23 22:26:06 -06:00
  • 26bf591198 Bump to 1.0.6 Tat Dat Duong 2023-04-12 01:03:54 +02:00
  • 0c0cf001fa Fix broken resolution when used in tests Tat Dat Duong 2023-04-12 00:55:26 +02:00
  • 496168ce29 Bump to 1.0.5 Tat Dat Duong 2023-04-12 00:40:17 +02:00
  • d40517f826 Reimplement node_modules detection for Next 13 appDir Tat Dat Duong 2023-04-12 00:40:04 +02:00
  • 83c2a857e1 Bump to 1.0.4 Tat Dat Duong 2023-04-08 20:08:01 +02:00
  • ff230c81a5 Fix Next.js + Webpack import issue due to missing exports entry Tat Dat Duong 2023-04-08 20:07:20 +02:00
  • 7261f28184 Bump to 1.0.3 Tat Dat Duong 2023-04-05 10:01:40 +02:00
  • ee98f78950 Merge remote-tracking branch 'upstream/main' Tat Dat Duong 2023-04-05 09:55:56 +02:00
  • a3c88a55f5 Merge pull request #25 from christophwitzko/patch-1 David Duong 2023-04-04 16:40:37 +02:00
  • 10e17c32bc Update lib.rs Christoph Witzko 2023-04-04 16:11:45 +02:00
  • 6bba615d0d Add gpt-4 to model_to_encoding.json Christoph Witzko 2023-04-04 11:24:32 +02:00
  • 46287bfa49 Fix typo in error message (#93) jonathanagustin 2023-04-02 16:25:05 -04:00
  • e1c661edf3 Bump version, sync codebase Shantanu Jain 2023-03-28 13:44:35 -07:00
  • b9a03a7293 Add custom exception when not initialized Tat Dat Duong 2023-03-20 00:43:43 +01:00
  • 6823b8e592 Update README.md with information about Create React App Tat Dat Duong 2023-03-19 18:09:08 +01:00
  • 82facf911f Update the perf image path in readme (#68) Mariatta Wijaya 2023-03-17 13:46:32 -07:00
  • 529de22652 Bump version in pyproject.toml Shantanu Jain 2023-03-16 18:13:26 -07:00
  • 446cb49aff Bump version, sync codebase Shantanu Jain 2023-03-16 18:11:50 -07:00
  • 481fb453c9 Bump to 1.0.2 Tat Dat Duong 2023-03-16 02:49:59 +01:00
  • 68efa86723 Clarifies usage Tat Dat Duong 2023-03-16 02:49:21 +01:00
  • c4dfaadb3d Bump to 1.0.1 Tat Dat Duong 2023-03-16 02:44:50 +01:00
  • e1c4313a35 Fix issues with duplicate initialization Tat Dat Duong 2023-03-16 02:44:43 +01:00
  • 8476ecaed9 Bump to 1.0.0 Tat Dat Duong 2023-03-15 19:22:21 +01:00
  • efe372862d Update README.md Tat Dat Duong 2023-03-15 19:13:32 +01:00
  • 4d4b921e51 Bump to 1.0.0-alpha.10 Tat Dat Duong 2023-03-15 19:00:12 +01:00
  • 7efba72caa Use compressed version to make main WASM smaller Tat Dat Duong 2023-03-15 18:55:52 +01:00
  • 6ac1a1ae16 Compress ranks Tat Dat Duong 2023-03-15 18:53:18 +01:00
  • a3baa6ca18 Bump to 1.0.0-alpha.8 Tat Dat Duong 2023-03-15 18:50:07 +01:00
  • 23bb57d0c3 Bump to 1.0.0-alpha.7 Tat Dat Duong 2023-03-15 16:03:52 +01:00
  • c01af191ed Add JSON importable modules Tat Dat Duong 2023-03-15 16:03:44 +01:00
  • 9847f4d580 Bump to 1.0.0-alpha.6 Tat Dat Duong 2023-03-15 13:44:10 +01:00
  • bfe38177e1 Update README.md Tat Dat Duong 2023-03-15 13:23:19 +01:00
  • f7487547ea Fix lite crash, add README.md Tat Dat Duong 2023-03-15 13:12:03 +01:00
  • 79ac0360de Bump to 1.0.0-alpha.5 Tat Dat Duong 2023-03-15 11:56:59 +01:00
  • 103d010700 Expose model_to_encoding.json and registry.json Tat Dat Duong 2023-03-15 11:56:53 +01:00
  • 9c8caec715 Add support for GPT-4 Tat Dat Duong 2023-03-14 19:21:25 +01:00
  • 3e8620030c Bump version, sync codebase Shantanu Jain 2023-03-12 22:01:24 -07:00
  • b2e85f1423 Build aarch64 wheels under emulation (#54) Shantanu 2023-03-12 20:34:02 -07:00
  • 8984ea764a Add disclaimer for CFW Tat Dat Duong 2023-03-11 21:42:11 +01:00
  • 1ee26c3532 Remove bundler, as it is unnecessary Tat Dat Duong 2023-03-11 21:37:51 +01:00
  • e6f072654b Fix exports for CJS (Node ESM) Tat Dat Duong 2023-03-11 21:03:46 +01:00
  • 4f6745f48c Merge pull request #15 from dqbd/lite-build David Duong 2023-03-11 17:47:40 +01:00
  • 17dd0ba5ec Add polyfill for Buffer.from Tat Dat Duong 2023-03-11 17:46:00 +01:00
  • 64178e5678 Expose loading script Tat Dat Duong 2023-03-11 17:11:06 +01:00
  • 6d8c1dc792 Cleanup Tat Dat Duong 2023-03-11 12:39:01 +01:00