Skip to content
New this week26 prompts for Sora, Veo and Runway video workflows.
Explore video prompts→
AIPrompt123Prompt library

Search AIPrompt123

Search prompts, categories, platforms and repositories

Models & inferencePythonApache-2.0

vllm-project/vllm

vLLM is the default choice for serving open models at scale. PagedAttention improves memory efficiency, continuous batching raises throughput, and the OpenAI-compatible server makes migration straightforward. It is a server-side tool; GPU capacity planning and model compatibility should be validated before production.

58.0k stars9.7k forks1.5k contributors1300 open issuesUpdated

What it does

  • Serves LLMs with high throughput and efficient memory use.
  • Provides an OpenAI-compatible API server.
  • Supports tensor parallelism and quantisation.

Key features

  • PagedAttention and continuous batching.
  • Wide model support through Hugging Face.
  • Prefix caching and speculative decoding.
  • Kubernetes and Ray deployment examples.

How to install it

Commands are copied from the project documentation at the time of review. Check the repository for the current version.

Python
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct

Alternatives to consider

vllm is a strong default for models & inference, but the right choice depends on your stack, licence constraints and how much control you need. These are the projects we compare most often.

llama.cppTensorRT-LLMOllama

Prompts that pair with it

All prompts →
Verified Knowledge · FAQ

vllm FAQ

Adoption questions we hear most often about this project.

Related repositories

Models & inference · Go

Run open models locally with a single command and a simple local API.

152.0k13.2kMIT

Models & inference · Python

The reference library for state-of-the-art pretrained models across text, vision and audio.

149.0k30.2kApache-2.0

Models & inference · C++

Efficient C/C++ inference for Llama-family models on CPUs, GPUs and edge devices.

89.0k13.1kMIT