AirLLM 2.8T LLM inference with single 4GB GPU
High-performance In-browser LLM Inference Engine
A high-throughput and memory-efficient inference and serving engine for LLMs
On-device LLM Inference Powered by X-Bit Quantization
Universal LLM Deployment Engine with ML Compilation
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar