Install this package:
emerge -a dev-python/tokenizersIf the package is masked, you can unmask it using the autounmask tool or standard emerge options:
autounmask dev-python/tokenizersOr alternatively:
emerge --autounmask-write -a dev-python/tokenizers| Version | EAPI | Keywords | Slot |
|---|---|---|---|
| 0.22.1 | 8 | ~amd64 ~arm64 ~x86 | 0 |
No metadata available.
Manage flags for this package:
euse -i <flag> -p dev-python/tokenizers |
euse -E <flag> -p dev-python/tokenizers |
euse -D <flag> -p dev-python/tokenizers
| Flag | Description | 0.22.1 |
|---|---|---|
| fancy-regex | ⚠️ | ✓ |
| hf-hub | ⚠️ | ✓ |
| indicatif | ⚠️ | ✓ |
| onig | ⚠️ | ✓ |
| Type | File | Size | Versions |
|---|
| Type | File | Size |
|---|---|---|
| DIST | addr2line-0.24.2.crate | 39015 bytes |
| DIST | adler2-2.0.1.crate | 13366 bytes |
| DIST | ahash-0.8.12.crate | 43413 bytes |
| DIST | aho-corasick-1.1.3.crate | 183311 bytes |
| DIST | anstream-0.6.20.crate | 28797 bytes |
| DIST | anstyle-1.0.11.crate | 15880 bytes |
| DIST | anstyle-parse-0.2.7.crate | 21707 bytes |
| DIST | anstyle-query-1.1.4.crate | 10192 bytes |
| DIST | anstyle-wincon-3.0.10.crate | 12558 bytes |
| DIST | autocfg-1.5.0.crate | 18729 bytes |
| DIST | backtrace-0.3.75.crate | 92665 bytes |
| DIST | base64-0.13.1.crate | 61002 bytes |
| DIST | bitflags-2.9.4.crate | 47950 bytes |
| DIST | bumpalo-3.19.0.crate | 96414 bytes |
| DIST | castaway-0.2.4.crate | 12546 bytes |
| DIST | cc-1.2.38.crate | 90139 bytes |
| DIST | cfg-if-1.0.3.crate | 8719 bytes |
| DIST | colorchoice-1.0.4.crate | 8196 bytes |
| DIST | compact_str-0.9.0.crate | 72135 bytes |
| DIST | console-0.15.11.crate | 37822 bytes |
| DIST | crossbeam-deque-0.8.6.crate | 22471 bytes |
| DIST | crossbeam-epoch-0.9.18.crate | 46875 bytes |
| DIST | crossbeam-utils-0.8.21.crate | 42691 bytes |
| DIST | darling-0.20.11.crate | 37614 bytes |
| DIST | darling_core-0.20.11.crate | 68006 bytes |
| DIST | darling_macro-0.20.11.crate | 2532 bytes |
| DIST | dary_heap-0.3.8.crate | 26544 bytes |
| DIST | derive_builder-0.20.2.crate | 36661 bytes |
| DIST | derive_builder_core-0.20.2.crate | 31397 bytes |
| DIST | derive_builder_macro-0.20.2.crate | 6360 bytes |
| DIST | either-1.15.0.crate | 20114 bytes |
| DIST | encode_unicode-1.0.0.crate | 56986 bytes |
| DIST | env_filter-0.1.3.crate | 15191 bytes |
| DIST | env_logger-0.11.8.crate | 32538 bytes |
| DIST | errno-0.3.14.crate | 12002 bytes |
| DIST | esaxx-rs-0.1.10.crate | 175210 bytes |
| DIST | fastrand-2.3.0.crate | 15076 bytes |
| DIST | find-msvc-tools-0.1.2.crate | 30461 bytes |
| DIST | fnv-1.0.7.crate | 11266 bytes |
| DIST | futures-0.3.31.crate | 54953 bytes |
| DIST | futures-channel-0.3.31.crate | 31971 bytes |
| DIST | futures-core-0.3.31.crate | 14318 bytes |
| DIST | futures-executor-0.3.31.crate | 17965 bytes |
| DIST | futures-io-0.3.31.crate | 9047 bytes |
| DIST | futures-macro-0.3.31.crate | 11341 bytes |
| DIST | futures-sink-0.3.31.crate | 7958 bytes |
| DIST | futures-task-0.3.31.crate | 11217 bytes |
| DIST | futures-util-0.3.31.crate | 162124 bytes |
| DIST | getrandom-0.3.3.crate | 49493 bytes |
| DIST | gimli-0.31.1.crate | 279515 bytes |
| DIST | heck-0.5.0.crate | 11517 bytes |
| DIST | ident_case-1.0.1.crate | 3492 bytes |
| DIST | indicatif-0.17.11.crate | 66577 bytes |
| DIST | indoc-2.0.6.crate | 17164 bytes |
| DIST | io-uring-0.7.10.crate | 103070 bytes |
| DIST | is_terminal_polyfill-1.70.1.crate | 7492 bytes |
| DIST | itertools-0.14.0.crate | 152715 bytes |
| DIST | itoa-1.0.15.crate | 11231 bytes |
| DIST | jiff-0.2.15.crate | 712996 bytes |
| DIST | jiff-static-0.2.15.crate | 76146 bytes |
| DIST | js-sys-0.3.80.crate | 56202 bytes |
| DIST | libc-0.2.175.crate | 788728 bytes |
| DIST | linux-raw-sys-0.11.0.crate | 2659624 bytes |
| DIST | log-0.4.28.crate | 51131 bytes |
| DIST | macro_rules_attribute-0.2.2.crate | 15330 bytes |
| DIST | macro_rules_attribute-proc_macro-0.2.2.crate | 8271 bytes |
| DIST | matrixmultiply-0.3.10.crate | 58170 bytes |
| DIST | memchr-2.7.5.crate | 97603 bytes |
| DIST | memoffset-0.9.1.crate | 9032 bytes |
| DIST | minimal-lexical-0.2.1.crate | 94841 bytes |
| DIST | miniz_oxide-0.8.9.crate | 67132 bytes |
| DIST | mio-1.0.4.crate | 104212 bytes |
| DIST | monostate-0.1.16.crate | 15288 bytes |
| DIST | monostate-impl-0.1.16.crate | 7783 bytes |
| DIST | ndarray-0.16.1.crate | 291026 bytes |
| DIST | nom-7.1.3.crate | 117570 bytes |
| DIST | num-complex-0.4.6.crate | 30352 bytes |
| DIST | num-integer-0.1.46.crate | 22331 bytes |
| DIST | num-traits-0.2.19.crate | 51631 bytes |
| DIST | number_prefix-0.4.0.crate | 6922 bytes |
| DIST | numpy-0.25.0.crate | 81623 bytes |
| DIST | object-0.36.7.crate | 329938 bytes |
| DIST | once_cell-1.21.3.crate | 34534 bytes |
| DIST | once_cell_polyfill-1.70.1.crate | 7510 bytes |
| DIST | onig-6.5.1.crate | 32394 bytes |
| DIST | onig_sys-69.9.1.crate | 656378 bytes |
| DIST | paste-1.0.15.crate | 18374 bytes |
| DIST | pin-project-lite-0.2.16.crate | 30504 bytes |
| DIST | pin-utils-0.1.0.crate | 7580 bytes |
| DIST | pkg-config-0.3.32.crate | 21370 bytes |
| DIST | portable-atomic-1.11.1.crate | 185506 bytes |
| DIST | portable-atomic-util-0.2.4.crate | 47043 bytes |
| DIST | ppv-lite86-0.2.21.crate | 22522 bytes |
| DIST | proc-macro2-1.0.101.crate | 53886 bytes |
| DIST | pyo3-0.25.1.crate | 1121993 bytes |
| DIST | pyo3-async-runtimes-0.25.0.crate | 51342 bytes |
| DIST | pyo3-build-config-0.25.1.crate | 34126 bytes |
| DIST | pyo3-ffi-0.25.1.crate | 78058 bytes |
| DIST | pyo3-macros-0.25.1.crate | 8886 bytes |
| DIST | pyo3-macros-backend-0.25.1.crate | 76006 bytes |
| DIST | quote-1.0.40.crate | 31063 bytes |
| DIST | r-efi-5.3.0.crate | 64532 bytes |
| DIST | rand-0.9.2.crate | 99930 bytes |
| DIST | rand_chacha-0.9.0.crate | 18258 bytes |
| DIST | rand_core-0.9.3.crate | 24543 bytes |
| DIST | rawpointer-0.2.1.crate | 7490 bytes |
| DIST | rayon-1.11.0.crate | 182470 bytes |
| DIST | rayon-cond-0.4.0.crate | 10659 bytes |
| DIST | rayon-core-1.13.0.crate | 73151 bytes |
| DIST | regex-1.11.2.crate | 166265 bytes |
| DIST | regex-automata-0.4.10.crate | 622754 bytes |
| DIST | regex-syntax-0.8.6.crate | 358808 bytes |
| DIST | rustc-demangle-0.1.26.crate | 30340 bytes |
| DIST | rustc-hash-2.1.1.crate | 14154 bytes |
| DIST | rustix-1.1.2.crate | 422717 bytes |
| DIST | rustversion-1.0.22.crate | 21096 bytes |
| DIST | ryu-1.0.20.crate | 48738 bytes |
| DIST | serde-1.0.225.crate | 28467 bytes |
| DIST | serde_core-1.0.225.crate | 63004 bytes |
| DIST | serde_derive-1.0.225.crate | 58560 bytes |
| DIST | serde_json-1.0.145.crate | 155748 bytes |
| DIST | shlex-1.3.0.crate | 18713 bytes |
| DIST | signal-hook-registry-1.4.6.crate | 19277 bytes |
| DIST | slab-0.4.11.crate | 18549 bytes |
| DIST | smallvec-1.15.1.crate | 38116 bytes |
| DIST | spm_precompiled-0.1.4.crate | 557527 bytes |
| DIST | static_assertions-1.1.0.crate | 18480 bytes |
| DIST | strsim-0.11.1.crate | 14266 bytes |
| DIST | syn-2.0.106.crate | 301514 bytes |
| DIST | target-lexicon-0.13.3.crate | 28498 bytes |
| DIST | tempfile-3.22.0.crate | 43044 bytes |
| DIST | thiserror-2.0.16.crate | 29095 bytes |
| DIST | thiserror-impl-2.0.16.crate | 21214 bytes |
| DIST | tokenizers-0.22.1.crate | 186019 bytes |
| DIST | tokenizers-0.22.1.tar.gz | 363123 bytes |
| DIST | tokio-1.47.1.crate | 829790 bytes |
| DIST | tokio-macros-2.5.0.crate | 12617 bytes |
| DIST | unicode-ident-1.0.19.crate | 47480 bytes |
| DIST | unicode-normalization-alignments-0.1.12.crate | 91546 bytes |
| DIST | unicode-segmentation-1.12.0.crate | 106323 bytes |
| DIST | unicode-width-0.2.1.crate | 279344 bytes |
| DIST | unicode_categories-0.1.1.crate | 87298 bytes |
| DIST | unindent-0.2.4.crate | 7422 bytes |
| DIST | utf8parse-0.2.2.crate | 13499 bytes |
| DIST | version_check-0.9.5.crate | 15554 bytes |
| DIST | wasi-0.11.1+wasi-snapshot-preview1.crate | 28477 bytes |
| DIST | wasi-0.14.7+wasi-0.2.4.crate | 18219 bytes |
| DIST | wasip2-1.0.1+wasi-0.2.4.crate | 132087 bytes |
| DIST | wasm-bindgen-0.2.103.crate | 47309 bytes |
| DIST | wasm-bindgen-backend-0.2.103.crate | 32238 bytes |
| DIST | wasm-bindgen-macro-0.2.103.crate | 9382 bytes |
| DIST | wasm-bindgen-macro-support-0.2.103.crate | 26418 bytes |
| DIST | wasm-bindgen-shared-0.2.103.crate | 9060 bytes |
| DIST | web-time-1.1.0.crate | 18026 bytes |
| DIST | windows-link-0.1.3.crate | 6154 bytes |
| DIST | windows-link-0.2.0.crate | 6170 bytes |
| DIST | windows-sys-0.59.0.crate | 2387323 bytes |
| DIST | windows-sys-0.60.2.crate | 2518479 bytes |
| DIST | windows-sys-0.61.0.crate | 2517134 bytes |
| DIST | windows-targets-0.52.6.crate | 6403 bytes |
| DIST | windows-targets-0.53.3.crate | 7099 bytes |
| DIST | windows_aarch64_gnullvm-0.52.6.crate | 435718 bytes |
| DIST | windows_aarch64_gnullvm-0.53.0.crate | 782443 bytes |
| DIST | windows_aarch64_msvc-0.52.6.crate | 832615 bytes |
| DIST | windows_aarch64_msvc-0.53.0.crate | 834446 bytes |
| DIST | windows_i686_gnu-0.52.6.crate | 880402 bytes |
| DIST | windows_i686_gnu-0.53.0.crate | 936973 bytes |
| DIST | windows_i686_gnullvm-0.52.6.crate | 475940 bytes |
| DIST | windows_i686_gnullvm-0.53.0.crate | 854056 bytes |
| DIST | windows_i686_msvc-0.52.6.crate | 901163 bytes |
| DIST | windows_i686_msvc-0.53.0.crate | 903450 bytes |
| DIST | windows_x86_64_gnu-0.52.6.crate | 836363 bytes |
| DIST | windows_x86_64_gnu-0.53.0.crate | 902585 bytes |
| DIST | windows_x86_64_gnullvm-0.52.6.crate | 435707 bytes |
| DIST | windows_x86_64_gnullvm-0.53.0.crate | 782434 bytes |
| DIST | windows_x86_64_msvc-0.52.6.crate | 832564 bytes |
| DIST | windows_x86_64_msvc-0.53.0.crate | 834400 bytes |
| DIST | wit-bindgen-0.46.0.crate | 60508 bytes |
| DIST | zerocopy-0.8.27.crate | 252663 bytes |
| DIST | zerocopy-derive-0.8.27.crate | 89827 bytes |