Skip to content
New this week26 prompts for Sora, Veo and Runway video workflows.
Explore video prompts→
AIPrompt123Prompt library

Search AIPrompt123

Search prompts, categories, platforms and repositories

Models & inferenceC++MIT

ggml-org/llama.cpp

llama.cpp is the engine behind much of the local inference ecosystem. It implements efficient quantisation and runs models across Apple Silicon, x86, CUDA, Vulkan and more. It is lower level than Ollama but gives finer control over memory, context and sampling, and it ships an OpenAI-compatible server.

89.0k stars13.1k forks1.9k contributors700 open issuesUpdated

What it does

  • Runs quantised language models efficiently on modest hardware.
  • Provides a CLI, server and bindings for many languages.
  • Supports GGUF model format and hardware acceleration.

Key features

  • CPU-first design with optional GPU offload.
  • Many quantisation formats.
  • OpenAI-compatible HTTP server.
  • Active performance work across backends.

How to install it

Commands are copied from the project documentation at the time of review. Check the repository for the current version.

Build
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release

Alternatives to consider

llama.cpp is a strong default for models & inference, but the right choice depends on your stack, licence constraints and how much control you need. These are the projects we compare most often.

OllamavLLMtransformers

Prompts that pair with it

All prompts →
Verified Knowledge · FAQ

llama.cpp FAQ

Adoption questions we hear most often about this project.

Related repositories

Models & inference · Go

Run open models locally with a single command and a simple local API.

152.0k13.2kMIT

Models & inference · Python

The reference library for state-of-the-art pretrained models across text, vision and audio.

149.0k30.2kApache-2.0

Models & inference · Python

A high-throughput inference and serving engine for LLMs, built around PagedAttention.

58.0k9.7kApache-2.0