sci-ml/amd-gaia::stuff
- Ebuilds: 2, Testing: 0.22.0 Description:
GAIA is AMD's open-source agent framework for local AI agents on
Ryzen AI hardware (NPU + iGPU). It orchestrates LLM-driven workflows
over any OpenAI-compatible inference endpoint, with built-in
integrations for Docker, Jira, code-search, RAG, MCP servers, and
Whisper / Kokoro voice pipelines. The reference local backend is
Lemonade Server (sci-ml/lemonade); GAIA itself is hardware-agnostic
so long as the upstream LLM API is OpenAI-compatible.
Homepage:https://github.com/amd/gaia License: MIT
sci-ml/bitnet::bennypowers
- Ebuilds: 1, Snapshot: 9999 Description:
BitNet is the official inference framework for 1-bit LLMs (BitNet b1.58).
It provides optimized kernels for fast and lossless inference of 1.58-bit
quantized models on CPU, with architecture-specific optimizations for
x86 (TL2) and ARM (TL1) platforms.
Homepage:https://github.com/microsoft/BitNet License: MIT
sci-ml/colibri::bennypowers
- Ebuilds: 4, Testing: 1.1.1, Snapshot: 9999 Description:
Colibri runs GLM-5.2 (744B-parameter MoE) on consumer hardware with
as little as 25GB RAM. Pure C engine with zero runtime dependencies,
streaming experts from disk with an LRU cache. Supports MLA attention,
MTP speculative decoding, and optional CUDA GPU offloading.
Homepage:https://github.com/JustVugg/colibri License: Apache-2.0
sci-ml/fastflowlm::stuff
- Ebuilds: 3, Testing: 0.9.46, Snapshot: 9999 Description:
FastFlowLM (FLM) is a lightweight LLM inference runtime purpose-built
for AMD Ryzen AI NPUs (XDNA2 architecture). It provides an Ollama-style
CLI and OpenAI-compatible server API for running language models entirely
on the NPU with no GPU or CPU compute required.
Supported hardware: Ryzen AI 300-series (Strix Point, Strix Halo),
400-series (Gorgon Point), and Z2 Extreme. XDNA1 (Ryzen AI 7000/8000)
is NOT supported.
The orchestration code and CLI are MIT-licensed. NPU compute kernels
(xclbins) are proprietary binaries, free for commercial use under
$10M annual company revenue.
Homepage:
https://fastflowlm.com/
https://github.com/ROCm/FastFlowLM
License: MIT FastFlowLM-Binary
sci-ml/gaia::bennypowers
- Ebuilds: 6, Testing: 0.22.0 Description:
AMD Gaia is an AI agent framework that provides tools for building
and deploying AI agents. It includes support for various AI models
and frameworks, with integration for AMD hardware acceleration.
Homepage:https://github.com/amd/gaia License: MIT
ik_llama-cpp (available in: sci-ml/ik_llama-cpp::bentoo, sci-ml/ik_llama-cpp::zGentoo)
- Ebuilds: 3, Testing: 0.0.4582 Description:
ik_llama.cpp is a fork of llama.cpp by Iwan Kawrakow, the author of most
of the k-quants, carrying additional state-of-the-art quantization types
(the IQK families) and CPU matrix-multiplication kernels that are
substantially faster than upstream for quantized inference.
It builds the same llama-cli, llama-server, llama-quantize and
llama-bench programs as sci-ml/llama-cpp. To let both be installed at
once, every binary is installed with an ik_ prefix (ik_llama-cli,
ik_llama-server, and so on) and the shared libraries live in a private
subdirectory of libdir, so nothing collides with sci-ml/llama-cpp. No
headers or pkg-config files are installed: this package is an
application, not a development target. Link against sci-ml/llama-cpp
instead.
Upstream publishes no release tags, so the package is pinned to a dated
commit snapshot fetched as a commit archive.
The CPU backend is tuned through CPU_FLAGS_X86. This matters more here
than for sci-ml/llama-cpp, because the fork's quantization kernels are
written for AVX2 and AVX-512; without those flags the generic fallback
gives up most of the fork's advantage.
Keywording is deliberately restricted to ~amd64. The SOTA quants are
hand-tuned for AVX2/AVX-512, the ARM paths are far less exercised
upstream, and no arm64 build of this snapshot has been verified.
Note that this ebuild is adapted from the guru overlay
(sci-misc/ik_llama-cpp).
Homepage:https://github.com/ikawrakow/ik_llama.cpp License: MIT
sci-ml/kokoro-onnx::bennypowers
- Ebuilds: 1, Testing: 0.5.0 Description:
kokoro-onnx provides local text-to-speech using the Kokoro-82M
neural TTS model via ONNX Runtime. Supports multiple languages
and voices with CPU inference by default. GPU acceleration is
available via the ONNX_PROVIDER environment variable when
onnxruntime is built with CUDA or MIGraphX support.
Homepage:https://github.com/thewh1teagle/kokoro-onnx License: MIT
sci-ml/kokoros::stuff
- Ebuilds: 1, Snapshot: 9999 Description:
Kokoros is a Rust implementation of the Kokoro-82M text-to-speech
model. Provides the `koko` CLI and an OpenAI-compatible HTTP server
used as the kokoro:cpu backend by sci-ml/lemonade.
Tracks upstream lucasjinreal/Kokoros directly. The lemonade-sdk
fork only diverges in CI infrastructure plus a bundled espeak-ng-data
copy that ::gentoo already provides via app-accessibility/espeak-ng,
so source-build users get the same binary either way.
Runtime model files (kokoro-v1.0.onnx + voices-v1.0.bin) are not
bundled — see pkg_postinst for a quick fetch recipe.
Homepage:https://github.com/lucasjinreal/Kokoros License: Apache-2.0
sci-ml/lemonade-bin::bentoo
- Ebuilds: 1, Testing: 11.5.1 Description:
Lemonade is a local LLM server that exposes an OpenAI-compatible HTTP API
plus a browser UI, and configures state-of-the-art inference engines for
the hardware it finds: AMD GPUs (ROCm/Vulkan), NVIDIA GPUs, Intel, Ryzen
AI NPUs and plain CPU. It ships a daemon (lemond) and a CLI client
(lemonade) able to download, load and route models from providers such as
Hugging Face.
This package installs upstream's prebuilt "embeddable" binaries instead of
building from source. The binaries are taken from the embeddable tarball,
which links its Debian-specific dependencies statically; the official .deb
is used only as a source of architecture-independent data (web UI, JSON
resources, schemas, examples, man pages and systemd units), because its
executables require Debian-13 library sonames that do not exist on Gentoo.
Inference backends are not packaged: lemond downloads them at runtime into
its cache directory.
Homepage:https://lemonade-server.ai/
https://github.com/lemonade-sdk/lemonade License: Apache-2.0 BSD MIT
llama-cpp (available in: sci-ml/llama-cpp::bentoo, sci-ml/llama-cpp::gentoo-zh, sci-ml/llama-cpp::gentooplusplus, sci-ml/llama-cpp::zGentoo)
- Ebuilds: 7, Testing: 9608, 0_pre10229, 0_pre10054, Snapshot: 9999 Description:
llama.cpp is an inference engine for large language models written in plain
C/C++, with no external runtime dependencies. It loads quantized models in
the GGUF format and runs them on CPU, or with optional acceleration through
CUDA, ROCm/HIP, Vulkan or OpenCL backends.
The package provides llama-cli (interactive and one-shot inference),
llama-server (an HTTP server exposing an OpenAI-compatible API, optionally
with an embedded WebUI), llama-quantize, llama-bench and the ggml libraries
used by many downstream projects.
Models are not shipped with this package and must be downloaded separately,
typically from Hugging Face. The CPU backend is tuned through the
architecture's CPU_FLAGS: on x86 through CPU_FLAGS_X86, where the AVX-512
and AMX flags supported by the host processor have a large impact on
inference throughput, and on ARM through CPU_FLAGS_ARM, which ggml gates
at compile time via an accumulated -march string rather than per-feature
build options.
Note that ebuild is ported from the gentoo-zh overlay (sci-ml/llama-cpp).
Homepage:https://github.com/ggml-org/llama.cpp License: MIT
sci-ml/lmstudio-bin::deftera
- Ebuilds: 1, Testing: 0.4.18 Description:
LM Studio is a proprietary desktop application by Element Labs for
discovering, downloading and running large language models locally.
It provides a chat UI and an OpenAI-compatible local server, with
GPU acceleration via bundled Vulkan and ROCm/llama.cpp backends.
This package installs the prebuilt amd64 binary published by
upstream as a .deb release artifact under /opt/LM-Studio. The
Electron 38 / Chromium 140 runtime and its supporting libraries are
bundled inside that prefix; only a small set of system X11/GTK/glibc
libraries is pulled in via RDEPEND.
Models and settings are stored under ~/.lmstudio. A modern x86-64
CPU with AVX2 is required.
Homepage:https://lmstudio.ai/ License: LM-Studio
ollama-bin (available in: sci-ml/ollama-bin::bentoo, sci-ml/ollama-bin::pingwho-overlay)
- Ebuilds: 3, Testing: 0.32.5, Snapshot: 9999 Description:
Ollama is a tool for running large language models (LLMs) locally on your
machine. It provides a simple interface to download, run, and manage models
like Llama 3.2, Mistral, Gemma, and many others.
This is a binary distribution package that installs pre-built binaries from
the official Ollama releases. The binaries are provided under the MIT license
and include GPU acceleration support for both NVIDIA (CUDA) and AMD (ROCm)
graphics cards.
Key features:
- Easy model management with pull, push, and create commands
- Built-in API server for programmatic access
- GPU acceleration support (CUDA and ROCm)
- Efficient memory management with automatic model loading/unloading
- Support for multiple models and concurrent requests
- Compatible with OpenAI API format
Models are stored in /var/lib/ollama and can range from 2GB (3B parameters)
to 40GB+ (70B parameters) in size. GPU acceleration significantly improves
inference speed but requires compatible hardware.
Security Note: This package installs pre-compiled binaries. Security
hardening features (ASLR, PIE, stack protections) depend on upstream's
build configuration. The service runs as a dedicated 'ollama' user with
restricted permissions for defense in depth.
Homepage:https://ollama.com/ License: MIT
sci-ml/sherpa-onnx::stuff
- Ebuilds: 2, Testing: 1.13.4 Description:
sherpa-onnx is a speech-stack toolkit from the k2-fsa project:
speech-to-text, text-to-speech, speaker diarization, voice activity
detection, source separation, and keyword spotting, all running on
ONNX Runtime (no PyTorch dependency).
Source build against system sci-libs/onnxruntime. For the prebuilt
-bin alternative (faster install, ships upstream's manylinux wheels)
see sci-ml/sherpa-onnx-bin.
The CMake build vendors a dozen small deps (eigen, asio, cargs, json,
kaldi-{decoder,native-fbank,fst}, openfst, kissfft, simple-sentencepiece,
hclust-cpp, optionally espeak-ng + piper-phonemize + portaudio +
websocketpp + pybind11) via FetchContent. The ebuild pre-fetches them
all via SRC_URI and stages into ${S} for the cmake fallback paths;
no network access during build.
Runtime model files for each task (ASR, diarization, TTS, etc.) live
upstream — see https://k2-fsa.github.io/sherpa/onnx/pretrained_models/
Homepage:
https://k2-fsa.github.io/sherpa/onnx/
https://github.com/k2-fsa/sherpa-onnx
License: Apache-2.0
sci-ml/sherpa-onnx-bin::stuff
- Ebuilds: 3, Testing: 1.13.4 Description:
sherpa-onnx is a speech-stack toolkit from the k2-fsa project:
speech-to-text, text-to-speech, speaker diarization, voice activity
detection, source separation, and keyword spotting, all running on
ONNX Runtime (no PyTorch dependency). Suited to CPU-only deployment
and embedded targets.
This -bin ebuild ships upstream's manylinux wheels (sherpa-onnx-core
for the C++ shared libraries plus a per-CPython-ABI wheel for the
Python bindings). Runtime model files are not bundled — see the
post-install message for download pointers.
Homepage:
https://k2-fsa.github.io/sherpa/onnx/
https://github.com/k2-fsa/sherpa-onnx
https://pypi.org/project/sherpa-onnx/
License: Apache-2.0
sci-ml/tensorzero::pingwho-overlay
- Ebuilds: 4, Testing: 2026.6.0, Snapshot: 9999 Description:
TensorZero is an open-source stack for industrial-grade LLM applications.
Gateway: access every LLM provider through a unified API low latency
Observability: monitor your LLM systems, programmatically or with a UI
Optimization: optimize your prompts, models, and inference strategies
Evaluations: benchmark individual inferences or end-to-end workflows
Experimentation: deploy with built-in A/B testing, fallbacks, etc.
Homepage:https://www.tensorzero.com/ License: Apache-2.0
sci-ml/whisper-cpp::bentoo
- Ebuilds: 1, Testing: 1.9.1 Description:
whisper.cpp is a plain C/C++ implementation of OpenAI's Whisper automatic
speech recognition model, built on the ggml tensor library and free of
external runtime dependencies. It transcribes and translates audio using
quantized models in the GGML format, on CPU or with optional acceleration
through CUDA, ROCm/HIP, Vulkan or OpenCL backends.
The package provides whisper-cli (one-shot transcription), whisper-server
(an HTTP server with an OpenAI-compatible endpoint), whisper-bench and
whisper-quantize, plus the parakeet-cli and parakeet-quantize tools for
NVIDIA's Parakeet TDT models, which upstream added alongside Whisper in the
1.9 series. With USE=sdl2 the real-time microphone examples whisper-stream,
whisper-command, whisper-talk-llama and whisper-lsp are built as well.
Models are not shipped with this package and must be downloaded separately,
typically from Hugging Face. The CPU backend is tuned through the
architecture's CPU_FLAGS: on x86 through CPU_FLAGS_X86, and on ARM through
CPU_FLAGS_ARM, which ggml gates at compile time via an accumulated -march
string rather than per-feature build options.
The ggml libraries this package builds are installed into a private
subdirectory rather than the system libdir, because sci-ml/llama-cpp
vendors the same ggml tree and owns those paths.
Note that the ebuild is adapted from the istitov/stuff overlay
(app-accessibility/whisper-cpp).
Homepage:https://github.com/ggml-org/whisper.cpp License: MIT