Built from scratch in Rust with Python bindings via PyO3. Produces identical output to tiktoken — same token IDs, same order, every time. Medium text (1050 chars, 511 tokens) 2.5M tok/s 68.8M tok/s ...
Python wrapper for SentencePiece. This API supports the encoding, decoding, and training of SentencePiece models. For a detailed feature and API comparison with Hugging Face Tokenizers and OpenAI's ...
Microsoft on Monday confirmed that it temporarily removed some GitHub repositories in response to a recent security incident that led to 73 of its open-source projects being compromised to inject an ...
Some results have been hidden because they may be inaccessible to you
Show inaccessible results