distributed-inference

FlashInfer: Kernel Library for LLM Serving

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

State of the art inference performance with modern accelerators on Kubernetes

High Performance Serving Runtime For Interactive Super Intelligence System

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

Best practices, reference architectures, and examples for distributed AI training and inference on AWS.

Run large AI models across your computers and servers. Powered by Colibri. Pure C.

[Official] Prima.cpp: Scale Your Local AI Beyond One Device.

Split FLUX.2 and LTX 2.3 across two GPUs (LAN or same-machine) — NVENC compresses activations live on the wire. Icarus (ComfyUI node) + Daedalus (back-half server).

Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric.

Distributed Inference for mlx LLm

一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。

Educational distributed training and inference library for heterogeneous local hardware: Mac minis, Raspberry Pis and GPUs over plain Python sockets. FSDP, GRPO, parameter-server, expert and data parallelism.

Free, open-source app that runs AI chat models on your own computer — and lets computers team up over the internet to run models too big for one machine. No account, no crypto. OpenAI- and Anthropic-compatible API.

Qwen3.8-27B and Qwen3.6-35B-A3B local inference engine for Vulkan and Metal

Pool RAM across old PCs, Macs, and Android devices into a unified local AI cluster

Distributed image and video generation in ComfyUI for two NVIDIA DGX Sparks.

Pools consumer laptops into one local runtime for Meta Llama models. Measures the free VRAM each machine reports, splits the model across CUDA and Metal devices through llama.cpp RPC, and rebuilds the pipeline when a worker drops out.

A chatbot, with a large language model split across its users, entirely in the browser.

Run any model on Intel silicon