tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.
published 0.0.8 10 months agoNAPI Bindings for rust pragmatic-segmenter
published 0.0.1 10 months ago🎡 Automated diagrams of CDK provisioned infrastructure
published 0.10.0 7 months ago