Gentoo Packages

Search

Install

Install this package:

emerge -a dev-python/tiktoken

Package Information

Description:

tiktoken is a byte pair encoding (BPE) tokeniser from OpenAI. It splits text into the integer tokens that GPT-family models actually consume, and decodes them back, which is what token counting, context-window budgeting and prompt truncation are built on. The hot loop is a Rust extension module bound through pyo3, so it is roughly an order of magnitude faster than pure-Python tokenisers. Encoding tables (r50k_base, p50k_base, cl100k_base, o200k_base, o200k_harmony) are downloaded on first use and cached locally.

Homepage:

https://github.com/openai/tiktoken/ https://pypi.org/project/tiktoken/

License:

MIT Apache-2.0-with-LLVM-exceptions MIT Unicode-3.0

Versions

Metadata

Description

• tiktoken is a byte pair encoding (BPE) tokeniser from OpenAI. It splits text into the integer tokens that GPT-family models actually consume, and decodes them back, which is what token counting, context-window budgeting and prompt truncation are built on. The hot loop is a Rust extension module bound through pyo3, so it is roughly an order of magnitude faster than pure-Python tokenisers. Encoding tables (r50k_base, p50k_base, cl100k_base, o200k_base, o200k_harmony) are downloaded on first use and cached locally.

Maintainers

• Lucas C.S. (lucascs@proton.me) - Type: person

Upstream

• Bugs: https://github.com/openai/tiktoken/issues
• Changelog: https://github.com/openai/tiktoken/blob/main/CHANGELOG.md
• Remote IDs:
  • github: openai/tiktoken
  • pypi: tiktoken

<pkgmetadata>
	<maintainer type="person">
		<email>lucascs@proton.me</email>
		<name>Lucas C.S.</name>
	</maintainer>
	<longdescription lang="en">
		tiktoken is a byte pair encoding (BPE) tokeniser from OpenAI. It splits
		text into the integer tokens that GPT-family models actually consume, and
		decodes them back, which is what token counting, context-window budgeting
		and prompt truncation are built on. The hot loop is a Rust extension
		module bound through pyo3, so it is roughly an order of magnitude faster
		than pure-Python tokenisers. Encoding tables (r50k_base, p50k_base,
		cl100k_base, o200k_base, o200k_harmony) are downloaded on first use and
		cached locally.
	</longdescription>
	<upstream>
		<changelog>https://github.com/openai/tiktoken/blob/main/CHANGELOG.md</changelog>
		<bugs-to>https://github.com/openai/tiktoken/issues</bugs-to>
		<remote-id type="github">openai/tiktoken</remote-id>
		<remote-id type="pypi">tiktoken</remote-id>
	</upstream>
</pkgmetadata>

Lint Warnings

• [Error] Manifest entry for unused DIST file 'aho-corasick-1.1.5.crate'
• [Error] Manifest entry for unused DIST file 'bit-set-0.8.0.crate'
• [Error] Manifest entry for unused DIST file 'bit-vec-0.8.0.crate'
• [Error] Manifest entry for unused DIST file 'bstr-1.13.1.crate'
• [Error] Manifest entry for unused DIST file 'fancy-regex-0.17.0.crate'
• [Error] Manifest entry for unused DIST file 'heck-0.5.0.crate'
• [Error] Manifest entry for unused DIST file 'libc-0.2.189.crate'
• [Error] Manifest entry for unused DIST file 'memchr-2.8.3.crate'
• [Error] Manifest entry for unused DIST file 'once_cell-1.21.4.crate'
• [Error] Manifest entry for unused DIST file 'portable-atomic-1.15.0.crate'
• [Error] Manifest entry for unused DIST file 'proc-macro2-1.0.107.crate'
• [Error] Manifest entry for unused DIST file 'pyo3-0.28.3.crate'
• [Error] Manifest entry for unused DIST file 'pyo3-build-config-0.28.3.crate'
• [Error] Manifest entry for unused DIST file 'pyo3-ffi-0.28.3.crate'
• [Error] Manifest entry for unused DIST file 'pyo3-macros-0.28.3.crate'
• [Error] Manifest entry for unused DIST file 'pyo3-macros-backend-0.28.3.crate'
• [Error] Manifest entry for unused DIST file 'quote-1.0.47.crate'
• [Error] Manifest entry for unused DIST file 'regex-1.13.1.crate'
• [Error] Manifest entry for unused DIST file 'regex-automata-0.4.18.crate'
• [Error] Manifest entry for unused DIST file 'regex-syntax-0.8.11.crate'
• [Error] Manifest entry for unused DIST file 'rustc-hash-2.1.3.crate'
• [Error] Manifest entry for unused DIST file 'serde_core-1.0.229.crate'
• [Error] Manifest entry for unused DIST file 'serde_derive-1.0.229.crate'
• [Error] Manifest entry for unused DIST file 'syn-2.0.119.crate'
• [Error] Manifest entry for unused DIST file 'syn-3.0.3.crate'
• [Error] Manifest entry for unused DIST file 'target-lexicon-0.13.5.crate'
• [Error] Manifest entry for unused DIST file 'tiktoken-0.14.0.tar.gz'
• [Error] Manifest entry for unused DIST file 'unicode-ident-1.0.24.crate'
• [Error] Repository is missing a repo-name in metadata/layout.conf or profiles/repo_name

Manifest