Gentoo Packages

dev-python/tokenizers (ha-bleeding-edge)

Search

Package Information

Description:
Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)
Homepage:
https://github.com/huggingface/tokenizers https://pypi.org/project/tokenizers/
License:
Apache-2.0

Versions

Version EAPI Keywords Slot
0.22.1 8 ~amd64 ~arm64 ~x86 0

Metadata

No metadata available.

Lint Warnings

USE Flags

Flag Description 0.22.1
fancy-regex ⚠️ ✓
hf-hub ⚠️ ✓
indicatif ⚠️ ✓
onig ⚠️ ✓

Manifest

Type File Size Versions
Unmatched Entries
Type File Size
DIST addr2line-0.24.2.crate 39015 bytes
DIST adler2-2.0.1.crate 13366 bytes
DIST ahash-0.8.12.crate 43413 bytes
DIST aho-corasick-1.1.3.crate 183311 bytes
DIST anstream-0.6.20.crate 28797 bytes
DIST anstyle-1.0.11.crate 15880 bytes
DIST anstyle-parse-0.2.7.crate 21707 bytes
DIST anstyle-query-1.1.4.crate 10192 bytes
DIST anstyle-wincon-3.0.10.crate 12558 bytes
DIST autocfg-1.5.0.crate 18729 bytes
DIST backtrace-0.3.75.crate 92665 bytes
DIST base64-0.13.1.crate 61002 bytes
DIST bitflags-2.9.4.crate 47950 bytes
DIST bumpalo-3.19.0.crate 96414 bytes
DIST castaway-0.2.4.crate 12546 bytes
DIST cc-1.2.38.crate 90139 bytes
DIST cfg-if-1.0.3.crate 8719 bytes
DIST colorchoice-1.0.4.crate 8196 bytes
DIST compact_str-0.9.0.crate 72135 bytes
DIST console-0.15.11.crate 37822 bytes
DIST crossbeam-deque-0.8.6.crate 22471 bytes
DIST crossbeam-epoch-0.9.18.crate 46875 bytes
DIST crossbeam-utils-0.8.21.crate 42691 bytes
DIST darling-0.20.11.crate 37614 bytes
DIST darling_core-0.20.11.crate 68006 bytes
DIST darling_macro-0.20.11.crate 2532 bytes
DIST dary_heap-0.3.8.crate 26544 bytes
DIST derive_builder-0.20.2.crate 36661 bytes
DIST derive_builder_core-0.20.2.crate 31397 bytes
DIST derive_builder_macro-0.20.2.crate 6360 bytes
DIST either-1.15.0.crate 20114 bytes
DIST encode_unicode-1.0.0.crate 56986 bytes
DIST env_filter-0.1.3.crate 15191 bytes
DIST env_logger-0.11.8.crate 32538 bytes
DIST errno-0.3.14.crate 12002 bytes
DIST esaxx-rs-0.1.10.crate 175210 bytes
DIST fastrand-2.3.0.crate 15076 bytes
DIST find-msvc-tools-0.1.2.crate 30461 bytes
DIST fnv-1.0.7.crate 11266 bytes
DIST futures-0.3.31.crate 54953 bytes
DIST futures-channel-0.3.31.crate 31971 bytes
DIST futures-core-0.3.31.crate 14318 bytes
DIST futures-executor-0.3.31.crate 17965 bytes
DIST futures-io-0.3.31.crate 9047 bytes
DIST futures-macro-0.3.31.crate 11341 bytes
DIST futures-sink-0.3.31.crate 7958 bytes
DIST futures-task-0.3.31.crate 11217 bytes
DIST futures-util-0.3.31.crate 162124 bytes
DIST getrandom-0.3.3.crate 49493 bytes
DIST gimli-0.31.1.crate 279515 bytes
DIST heck-0.5.0.crate 11517 bytes
DIST ident_case-1.0.1.crate 3492 bytes
DIST indicatif-0.17.11.crate 66577 bytes
DIST indoc-2.0.6.crate 17164 bytes
DIST io-uring-0.7.10.crate 103070 bytes
DIST is_terminal_polyfill-1.70.1.crate 7492 bytes
DIST itertools-0.14.0.crate 152715 bytes
DIST itoa-1.0.15.crate 11231 bytes
DIST jiff-0.2.15.crate 712996 bytes
DIST jiff-static-0.2.15.crate 76146 bytes
DIST js-sys-0.3.80.crate 56202 bytes
DIST libc-0.2.175.crate 788728 bytes
DIST linux-raw-sys-0.11.0.crate 2659624 bytes
DIST log-0.4.28.crate 51131 bytes
DIST macro_rules_attribute-0.2.2.crate 15330 bytes
DIST macro_rules_attribute-proc_macro-0.2.2.crate 8271 bytes
DIST matrixmultiply-0.3.10.crate 58170 bytes
DIST memchr-2.7.5.crate 97603 bytes
DIST memoffset-0.9.1.crate 9032 bytes
DIST minimal-lexical-0.2.1.crate 94841 bytes
DIST miniz_oxide-0.8.9.crate 67132 bytes
DIST mio-1.0.4.crate 104212 bytes
DIST monostate-0.1.16.crate 15288 bytes
DIST monostate-impl-0.1.16.crate 7783 bytes
DIST ndarray-0.16.1.crate 291026 bytes
DIST nom-7.1.3.crate 117570 bytes
DIST num-complex-0.4.6.crate 30352 bytes
DIST num-integer-0.1.46.crate 22331 bytes
DIST num-traits-0.2.19.crate 51631 bytes
DIST number_prefix-0.4.0.crate 6922 bytes
DIST numpy-0.25.0.crate 81623 bytes
DIST object-0.36.7.crate 329938 bytes
DIST once_cell-1.21.3.crate 34534 bytes
DIST once_cell_polyfill-1.70.1.crate 7510 bytes
DIST onig-6.5.1.crate 32394 bytes
DIST onig_sys-69.9.1.crate 656378 bytes
DIST paste-1.0.15.crate 18374 bytes
DIST pin-project-lite-0.2.16.crate 30504 bytes
DIST pin-utils-0.1.0.crate 7580 bytes
DIST pkg-config-0.3.32.crate 21370 bytes
DIST portable-atomic-1.11.1.crate 185506 bytes
DIST portable-atomic-util-0.2.4.crate 47043 bytes
DIST ppv-lite86-0.2.21.crate 22522 bytes
DIST proc-macro2-1.0.101.crate 53886 bytes
DIST pyo3-0.25.1.crate 1121993 bytes
DIST pyo3-async-runtimes-0.25.0.crate 51342 bytes
DIST pyo3-build-config-0.25.1.crate 34126 bytes
DIST pyo3-ffi-0.25.1.crate 78058 bytes
DIST pyo3-macros-0.25.1.crate 8886 bytes
DIST pyo3-macros-backend-0.25.1.crate 76006 bytes
DIST quote-1.0.40.crate 31063 bytes
DIST r-efi-5.3.0.crate 64532 bytes
DIST rand-0.9.2.crate 99930 bytes
DIST rand_chacha-0.9.0.crate 18258 bytes
DIST rand_core-0.9.3.crate 24543 bytes
DIST rawpointer-0.2.1.crate 7490 bytes
DIST rayon-1.11.0.crate 182470 bytes
DIST rayon-cond-0.4.0.crate 10659 bytes
DIST rayon-core-1.13.0.crate 73151 bytes
DIST regex-1.11.2.crate 166265 bytes
DIST regex-automata-0.4.10.crate 622754 bytes
DIST regex-syntax-0.8.6.crate 358808 bytes
DIST rustc-demangle-0.1.26.crate 30340 bytes
DIST rustc-hash-2.1.1.crate 14154 bytes
DIST rustix-1.1.2.crate 422717 bytes
DIST rustversion-1.0.22.crate 21096 bytes
DIST ryu-1.0.20.crate 48738 bytes
DIST serde-1.0.225.crate 28467 bytes
DIST serde_core-1.0.225.crate 63004 bytes
DIST serde_derive-1.0.225.crate 58560 bytes
DIST serde_json-1.0.145.crate 155748 bytes
DIST shlex-1.3.0.crate 18713 bytes
DIST signal-hook-registry-1.4.6.crate 19277 bytes
DIST slab-0.4.11.crate 18549 bytes
DIST smallvec-1.15.1.crate 38116 bytes
DIST spm_precompiled-0.1.4.crate 557527 bytes
DIST static_assertions-1.1.0.crate 18480 bytes
DIST strsim-0.11.1.crate 14266 bytes
DIST syn-2.0.106.crate 301514 bytes
DIST target-lexicon-0.13.3.crate 28498 bytes
DIST tempfile-3.22.0.crate 43044 bytes
DIST thiserror-2.0.16.crate 29095 bytes
DIST thiserror-impl-2.0.16.crate 21214 bytes
DIST tokenizers-0.22.1.crate 186019 bytes
DIST tokenizers-0.22.1.tar.gz 363123 bytes
DIST tokio-1.47.1.crate 829790 bytes
DIST tokio-macros-2.5.0.crate 12617 bytes
DIST unicode-ident-1.0.19.crate 47480 bytes
DIST unicode-normalization-alignments-0.1.12.crate 91546 bytes
DIST unicode-segmentation-1.12.0.crate 106323 bytes
DIST unicode-width-0.2.1.crate 279344 bytes
DIST unicode_categories-0.1.1.crate 87298 bytes
DIST unindent-0.2.4.crate 7422 bytes
DIST utf8parse-0.2.2.crate 13499 bytes
DIST version_check-0.9.5.crate 15554 bytes
DIST wasi-0.11.1+wasi-snapshot-preview1.crate 28477 bytes
DIST wasi-0.14.7+wasi-0.2.4.crate 18219 bytes
DIST wasip2-1.0.1+wasi-0.2.4.crate 132087 bytes
DIST wasm-bindgen-0.2.103.crate 47309 bytes
DIST wasm-bindgen-backend-0.2.103.crate 32238 bytes
DIST wasm-bindgen-macro-0.2.103.crate 9382 bytes
DIST wasm-bindgen-macro-support-0.2.103.crate 26418 bytes
DIST wasm-bindgen-shared-0.2.103.crate 9060 bytes
DIST web-time-1.1.0.crate 18026 bytes
DIST windows-link-0.1.3.crate 6154 bytes
DIST windows-link-0.2.0.crate 6170 bytes
DIST windows-sys-0.59.0.crate 2387323 bytes
DIST windows-sys-0.60.2.crate 2518479 bytes
DIST windows-sys-0.61.0.crate 2517134 bytes
DIST windows-targets-0.52.6.crate 6403 bytes
DIST windows-targets-0.53.3.crate 7099 bytes
DIST windows_aarch64_gnullvm-0.52.6.crate 435718 bytes
DIST windows_aarch64_gnullvm-0.53.0.crate 782443 bytes
DIST windows_aarch64_msvc-0.52.6.crate 832615 bytes
DIST windows_aarch64_msvc-0.53.0.crate 834446 bytes
DIST windows_i686_gnu-0.52.6.crate 880402 bytes
DIST windows_i686_gnu-0.53.0.crate 936973 bytes
DIST windows_i686_gnullvm-0.52.6.crate 475940 bytes
DIST windows_i686_gnullvm-0.53.0.crate 854056 bytes
DIST windows_i686_msvc-0.52.6.crate 901163 bytes
DIST windows_i686_msvc-0.53.0.crate 903450 bytes
DIST windows_x86_64_gnu-0.52.6.crate 836363 bytes
DIST windows_x86_64_gnu-0.53.0.crate 902585 bytes
DIST windows_x86_64_gnullvm-0.52.6.crate 435707 bytes
DIST windows_x86_64_gnullvm-0.53.0.crate 782434 bytes
DIST windows_x86_64_msvc-0.52.6.crate 832564 bytes
DIST windows_x86_64_msvc-0.53.0.crate 834400 bytes
DIST wit-bindgen-0.46.0.crate 60508 bytes
DIST zerocopy-0.8.27.crate 252663 bytes
DIST zerocopy-derive-0.8.27.crate 89827 bytes