Proxima · out-of-tree vLLM plugin · Apache-2.0

Your GPUs can serve 4x more requests.
vLLM just doesn't know it yet.

Proxima implements STAR-KV: low-rank KV cache compression via learned soft-thresholding, wired into vLLM's V1 continuous-batching, paged-attention stack with dedicated Triton kernels. Same GPU, same model, same weights. It just stops your KV cache from being the thing that runs out first.

View on GitHub See the numbers

Measured, not simulated

Same L4 24GB. Same longchat-7b checkpoint. Just the KV cache changed.

Baseline is vLLM 0.10.1.1's own V1 FlashAttentionBackend, on real ShareGPT serving traffic, on Modal L4 hardware. Every number below is Proxima against that baseline, nothing else moved.

Concurrency & capacity

enable_prefix_caching=False · fixed 512-token output

Metric	plain vLLM	STAR-KV
Max concurrent requests @ 8192 context, zero errors/timeouts	1	4 (4x)
Max concurrent requests @ 16384 context	0 (refuses to boot)	2 (boots where plain can't start)
GPU KV-cache blocks, same memory budget (max_model_len=4096)	756	2353 (3.11x)
Max max_model_len this L4 can boot at	8192	32768 (boots where plain refuses)

At 16384 context plain vLLM refuses to boot at any gpu_memory_utilization tried (0.9/0.85/0.8/0.75): needs 8.26 GiB, only 6.99 GiB available. STAR-KV boots at 5.95 GiB and serves real requests.

When to use it

This is a capacity fix, not a latency fix.

You're memory-bound, not latency-bound

You need more concurrent sequences or longer contexts than stock vLLM fits on the GPU you have, and can tolerate slower per-token decode in exchange for not OOMing or refusing to boot.

You're running long-context workloads

16k-32k+ tokens on GPUs where plain vLLM's boot-time KV-cache pre-flight check refuses to start.

You're serving real production batch sizes

The decode kernel is built for continuous batching, not single-request demos. At batch 16-64, where real serving traffic lives, it's up to 1.46x faster than FlashAttention-2.

Decode kernel

Faster than FlashAttention-2 at the batch sizes production actually runs.

32-head MHA, head_size 128, rank 64, fp16. Up to 1.46x faster than FA2, and the win holds from batch 16 through batch 64 and beyond, exactly where continuous batching lives in real serving.

Seq len	Batch 1	Batch 16	Batch 64
1024	0.36x	1.23x	1.40x
2048	0.57x	1.33x	1.43x
4096	0.84x	1.40x	1.43x
8192	0.87x	1.44x	1.46x

Quick start

Two commands. No migration, no model switch, no rewrite.

# 1. calibrate a STAR-KV checkpoint from a base HF model
python -m proxima_vllm.calibration.distill \
    --base-model lmsys/longchat-7b-v1.5-32k \
    --output ./checkpoints/longchat-7b-star-kv \
    --profiles aggressive,balanced,conservative

# 2. serve it with vLLM
vllm serve ./checkpoints/longchat-7b-star-kv \
    --additional-config '{"proxima_vllm": {"kv_compression": "star_kv", "rank_profile": "balanced", "use_triton_kernels": true}}'

# install from source, not on PyPI yet
git clone https://github.com/Tenosra/Proxima
cd Proxima
pip install -e . # pins vllm==0.10.1.1

Verified end to end on lmsys/longchat-7b-v1.5-32k, real ShareGPT traffic, on Modal L4 hardware. Multi-model support (Qwen, Mistral, Gemma, Llama-3) is written but not yet validated on hardware beyond this one model.

Stop paying for GPU memory your KV cache is wasting.

Apache-2.0, out-of-tree, source available today. Read the full breakdown, methodology, and every kernel benchmark on GitHub.

View on GitHub Talk to us about your stack

Your GPUs can serve 4x more requests.vLLM just doesn't know it yet.