Skip to content
TechIndexSubmit
Back to directory

llm-inference-capacity-handbook

大模型推理案头手册(开源版):给定 GPU 算力,一个模型能扛多少 QPS?三层模型 × 三面墙 × 排队论 × 开环压测

Similar tools

  • LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for custom training and inferencing.

    Models
  • 《开源大模型食用指南》针对中国宝宝量身打造的基于Linux环境快速微调(全参数/Lora)、部署国内外开源大模型(LLM)/多模态大模型(MLLM)教程

    Models
  • Open-source LLM inference benchmarks — TTFT, TPOT, Throughput, Latency & Cost-per-token for models like Llama, Qwen, Gemma, DeepSeek, Gpt-Oss etc. deployed on different dedicated GPUs.

    Models
  • Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)

    Dev Tools
  • Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

    Models
  • Empirical LLM evaluation on CPU-only testlab hardware — 5 models benchmarked for agent-capability: speed, tool-call validity, and reasoning quality

    Models