LLM frameworks · Python
A framework for composing LLM applications from models, tools, retrievers and memory.
Search prompts, categories, platforms and repositories
Ranked list · 8 repositories
A ranked guide to the best open-source LLM repositories for local inference, serving, fine-tuning and model training.
Open-source language models are only useful if you can run them. The projects below cover the practical stack: inference engines for local hardware, high-throughput serving for GPUs, training and fine-tuning toolkits, and the frameworks that wire models into applications. Choose based on your hardware, latency target and operational appetite rather than stars alone.
For a laptop prototype, Ollama is the fastest path. For a GPU server with many concurrent users, vLLM is usually the right default. For fine-tuning, start with the model's official recipes before adding a framework. Check every licence: model weights and code often carry different terms.
LLM frameworks · Python
A framework for composing LLM applications from models, tools, retrievers and memory.
Models & inference · Python
The reference library for state-of-the-art pretrained models across text, vision and audio.
Models & inference · Go
Run open models locally with a single command and a simple local API.
Models & inference · C++
Efficient C/C++ inference for Llama-family models on CPUs, GPUs and edge devices.
Models & inference · Python
A high-throughput inference and serving engine for LLMs, built around PagedAttention.
LLM frameworks · C#
Microsoft's SDK for integrating LLMs into C#, Python and Java applications.
LLM frameworks · Python
A framework for programming LLM pipelines and optimising prompts against a metric.
LLM frameworks · TypeScript
A TypeScript toolkit for building AI applications with streaming, tools and UI hooks.
Rankings combine stars, recent commit activity, licence clarity and documentation quality. Stars alone reward old projects with large historical audiences, so we weight maintenance and documentation too. We never accept payment for placement; repository pages are written by our editors and reviewed quarterly.
Back to the directory