High-performance serving framework for LLMs and multimodal models in Python. RadixAttention shares KV cache across common prefixes, boosting throughput on batch workloads.
Category: Model Inference & LLM Tooling • Stars: 32K
SGLang
Related posts
Repos
Transformers
Hugging Face's standard framework for state-of-the-art machine learning models in Python — practically every well-known open LLM runs through this library.
Read more
