Search results

3 packages found

M
Q
P

cpp tokenizer module for fibjs.

published 1.2.0 3 months ago
M
Q
P

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1

published 0.6.2 17 days ago
M
Q
P