Skip to content
TechIndexSubmit
Back to directory

JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

Similar tools

  • A high-throughput and memory-efficient inference and serving engine for LLMs

    Models
  • AirLLM 2.8T LLM inference with single 4GB GPU

    Models
  • RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

    Dev Tools
  • Self-hosted multi-model LLM and image platform that runs on free Kaggle and Colab GPUs. Chat UI, streaming, live metrics.

    Models
  • High-performance In-browser LLM Inference Engine

    Models
  • EmbeddedLLM: API server for Embedded Device Deployment. Currently support CUDA/OpenVINO/IpexLLM/DirectML/CPU

    Models