Skip to content
TechIndexSubmit
Back to directory

flash_attention_inference

Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.

Similar tools

  • High-performance In-browser LLM Inference Engine

    Models
  • 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.

    Dev Tools
  • Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

    Models
  • Optimizing inference proxy for LLMs

    Agents
  • Manage and run OpenAI fine-tuning jobs efficiently within a modular C++ toolkit for large language model infrastructure.

    Models
  • Autoscale LLM (vLLM, SGLang, LMDeploy) inferences on Kubernetes (and others)

    Dev Tools