sci-ml/llama-cpp::bentoo
- Ebuilds: 1, Testing: 0_pre9967 Description:
llama.cpp is an inference engine for large language models written in plain
C/C++, with no external runtime dependencies. It loads quantized models in
the GGUF format and runs them on CPU, or with optional acceleration through
CUDA, ROCm/HIP, Vulkan or OpenCL backends.
The package provides llama-cli (interactive and one-shot inference),
llama-server (an HTTP server exposing an OpenAI-compatible API, optionally
with an embedded WebUI), llama-quantize, llama-bench and the ggml libraries
used by many downstream projects.
Models are not shipped with this package and must be downloaded separately,
typically from Hugging Face. The CPU backend is selected through
CPU_FLAGS_X86, so enabling the AVX-512 and AMX flags supported by the host
processor has a large impact on inference throughput.
Note that ebuild is ported from the gentoo-zh overlay (sci-ml/llama-cpp).
Homepage:https://github.com/ggml-org/llama.cpp License: MIT
sci-ml/ollama-bin::bentoo
- Ebuilds: 1, Testing: 0.31.2 Description:
Ollama is a tool for running large language models (LLMs) locally on your
machine. It provides a simple interface to download, run, and manage models
like Llama 3.2, Mistral, Gemma, and many others.
This is a binary distribution package that installs pre-built binaries from
the official Ollama releases. The binaries are provided under the MIT license
and include GPU acceleration support for both NVIDIA (CUDA) and AMD (ROCm)
graphics cards.
Key features:
- Easy model management with pull, push, and create commands
- Built-in API server for programmatic access
- GPU acceleration support (CUDA and ROCm)
- Efficient memory management with automatic model loading/unloading
- Support for multiple models and concurrent requests
- Compatible with OpenAI API format
Models are stored in /var/lib/ollama and can range from 2GB (3B parameters)
to 40GB+ (70B parameters) in size. GPU acceleration significantly improves
inference speed but requires compatible hardware.
Security Note: This package installs pre-compiled binaries. Security
hardening features (ASLR, PIE, stack protections) depend on upstream's
build configuration. The service runs as a dedicated 'ollama' user with
restricted permissions for defense in depth.
Homepage:https://ollama.com/ License: MIT