XPost: linux.debian.bugs.dist
From:
[email protected]
Package: wnpp
Severity: wishlist
Owner: Kohei Sendai <
[email protected]>
X-Debbugs-Cc:
[email protected],
[email protected]
* Package name : librust-spm-precompiled-dev
Version : 0.1.4
Upstream Contact: Nicolas Patry <
[email protected]>
* URL :
https://github.com/huggingface/spm_precompiled
* License : Apache-2.0
Programming Lang: Rust
Description : emulate
https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer
This crate aims to emulate
https://github.com/google/sentencepiece Dart::DoubleArray struct and it's Normalizer. It's main intent is to be used with tokenizers that is a Rust library that aims to provide facilities to tokenize string for use with
HuggingFace's transformers library
This crate is highly specialized and not intended for general use.
The core of the algorithm is to read spm's binary precompiled_charsmap
- This is highly related to tokenizers which is really important for
packaging vllm. Now, I'm planning to packaging vllm.
--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)