Gentoo Packages

sci-ml/llama-cpp (bentoo)

Search

Install

Install this package:

emerge -a sci-ml/llama-cpp

Package Information

Description:
llama.cpp is an inference engine for large language models written in plain C/C++, with no external runtime dependencies. It loads quantized models in the GGUF format and runs them on CPU, or with optional acceleration through CUDA, ROCm/HIP, Vulkan or OpenCL backends. The package provides llama-cli (interactive and one-shot inference), llama-server (an HTTP server exposing an OpenAI-compatible API, optionally with an embedded WebUI), llama-quantize, llama-bench and the ggml libraries used by many downstream projects. Models are not shipped with this package and must be downloaded separately, typically from Hugging Face. The CPU backend is tuned through the architecture's CPU_FLAGS: on x86 through CPU_FLAGS_X86, where the AVX-512 and AMX flags supported by the host processor have a large impact on inference throughput, and on ARM through CPU_FLAGS_ARM, which ggml gates at compile time via an accumulated -march string rather than per-feature build options. Note that ebuild is ported from the gentoo-zh overlay (sci-ml/llama-cpp).
Homepage:
https://github.com/ggml-org/llama.cpp
License:
MIT

Versions

Version EAPI Keywords Slot
0_pre10229 8 ~amd64 ~arm64 0

Metadata

Description

Maintainers

Upstream

Raw Metadata XML 
<pkgmetadata>
	<maintainer type="person">
		<email>lucascs@proton.me</email>
		<name>Lucas C.S.</name>
	</maintainer>
	<longdescription lang="en">
		llama.cpp is an inference engine for large language models written in plain
		C/C++, with no external runtime dependencies. It loads quantized models in
		the GGUF format and runs them on CPU, or with optional acceleration through
		CUDA, ROCm/HIP, Vulkan or OpenCL backends.

		The package provides llama-cli (interactive and one-shot inference),
		llama-server (an HTTP server exposing an OpenAI-compatible API, optionally
		with an embedded WebUI), llama-quantize, llama-bench and the ggml libraries
		used by many downstream projects.

		Models are not shipped with this package and must be downloaded separately,
		typically from Hugging Face. The CPU backend is tuned through the
		architecture's CPU_FLAGS: on x86 through CPU_FLAGS_X86, where the AVX-512
		and AMX flags supported by the host processor have a large impact on
		inference throughput, and on ARM through CPU_FLAGS_ARM, which ggml gates
		at compile time via an accumulated -march string rather than per-feature
		build options.

		Note that ebuild is ported from the gentoo-zh overlay (sci-ml/llama-cpp).
	</longdescription>
	<use>
		<flag name="blis">Build a BLIS backend</flag>
		<flag name="flexiblas">Build a FlexiBLAS backend</flag>
		<flag name="openblas">Build an OpenBLAS backend</flag>
		<flag name="hip">Build a HIP (ROCm) backend</flag>
		<flag name="wmma">Use rocWMMA to enhance flash attention performance</flag>
		<flag name="opencl">Build an OpenCL backend, so far only works on Adreno and Intel GPUs</flag>
		<flag name="rpc">Build with rpc-server</flag>
		<flag name="server">Build with example server</flag>
		<flag name="webui">Build server with embedded WebUI</flag>
	</use>
	<upstream>
		<changelog>https://github.com/ggml-org/llama.cpp/releases</changelog>
		<bugs-to>https://github.com/ggml-org/llama.cpp/issues</bugs-to>
		<remote-id type="github">ggml-org/llama.cpp</remote-id>
	</upstream>
</pkgmetadata>

Lint Warnings

USE Flags

Manage flags for this package: euse -i <flag> -p sci-ml/llama-cpp | euse -E <flag> -p sci-ml/llama-cpp | euse -D <flag> -p sci-ml/llama-cpp

Flag Description 0_pre10229
"( ⚠️ ✓
( ⚠️ ✓
) ⚠️ ✓
)" ⚠️ ✓
amx_bf16 ⚠️ ✓
amx_int8 ⚠️ ✓
amx_tile ⚠️ ✓
asimddp ⚠️ ✓
asimdhp ⚠️ ✓
avx ⚠️ ✓
avx2 ⚠️ ✓
avx512_bf16 ⚠️ ✓
avx512_vnni ⚠️ ✓
avx512bw ⚠️ ✓
avx512cd ⚠️ ✓
avx512dq ⚠️ ✓
avx512f ⚠️ ✓
avx512vbmi ⚠️ ✓
avx512vl ⚠️ ✓
avx_vnni ⚠️ ✓
blis Build a BLIS backend ✓
bmi2 ⚠️ ✓
cuda Build cycles renderer with nVidia CUDA support. ⚠️ ✓
examples Example ufw config files ⚠️ ✓
f16c ⚠️ ✓
flexiblas Build a FlexiBLAS backend ✓
fma3 ⚠️ ✓
hip Build a HIP (ROCm) backend ✓
i8mm ⚠️ ✓
openblas Build an OpenBLAS backend ✓
opencl Build an OpenCL backend, so far only works on Adreno and Intel GPUs ✓
openmp ⚠️ ⊕
rpc Build with rpc-server ✓
server Build with example server ⊕
sse4_2 ⚠️ ✓
sve ⚠️ ✓
sve2 ⚠️ ✓
vulkan Add support for the Vulkan viewport backend ⚠️ ✓
webui Build server with embedded WebUI ✓
wmma Use rocWMMA to enhance flash attention performance ✓

Manifest

Type File Size Versions
DIST llama-cpp-0_pre10229.tar.gz 36480424 bytes 0_pre10229
DIST llama-cpp-b10155-ui.tar.gz 3055595 bytes 0_pre10229
Unmatched Entries
Type File Size
DIST ggml-org_models_tinyllamas_stories15M-q4_0-99dd1a73db5a37100bd4ae633f4cfce6560e1567.gguf 19077344 bytes