Surpassing vLLM with a
Generated Inference Stack
Infinity's infy optimization system wrote an inference engine from scratch and autonomously optimized it on Qwen3-8B. The resulting engine delivers up to 34.3% more tokens per second than vLLM when configured with identical parameters.
+34.3%
vs vLLM · 6,712 tok/s
+15.9%
vs vLLM · 22,470 tok/s
Infy Optimization Trajectory · Qwen3-8B, 111 Iterations on H100
Prefill-Heavy Workload (ISL=8k, OSL=1k)On decode-heavy workloads (ISL=1k, OSL=8k) (where inference serving is most throughput-constrained), infy reaches 6,712 tok/s, +34.3% above vLLM FP8 on identical hardware and parameters.
Prefill-heavy trajectory: 111 iterations, ISL=8192, OSL=1024, BS=88, FP8, H100 80GB SXM5, no speculative decoding.
Infy vs vLLM FP8 · Qwen3-8B
Total throughput (tok/s) · H100 80GB SXM5 · identical parameters
Decode-Heavy
ISL=1k, OSL=8kPrefill-Heavy
ISL=8k, OSL=1kFrom No Code to Frontier Performance
Infinity's infy optimization system generates inference engines from the ground up: no forks, no inherited constraints. It starts with the ecosystem's best proven techniques, then finds model-specific optimizations on top.
01
Model-specific from the start
Full-stack ownership means every scheduling and memory decision is made for this one model on this one GPU. Cross-layer kernel fusion, workload-specific prefill sizing, and overhead that only exists in frameworks simply disappears.
02
Absorbs the best of every framework
Because the stack is generated, it can cherry-pick the highest-performing technique from vLLM, SGLang, or anywhere else; not constrained to any one framework's architecture. Each addition is validated with real throughput metrics before keeping it.
03
Kernels without the framework
We use best-in-class kernel libraries (FlashInfer, DeepGEMM) while skipping the framework layer entirely. No model registry, no compatibility matrix, no conservative defaults.
Where the Gains Come From
A sampling of where the 15.9% advantage over vLLM comes from. Gains compound across many optimization categories, each individually small.
Cross-Layer Kernel Fusion
Fusing operations spanning attention, normalization, and quantization into single kernels, eliminating intermediate memory round-trips.
Scheduling Micro-Optimizations
Tuning prefill chunk sizes, unrolling decode CUDA graphs to bypass CPU-GPU sync, and dynamic batch progression.
Algorithmic Reorganization
Mathematically equivalent rearrangements (like deferring residual additions) that skip entire memory cycles per layer.
Compute Graph Refinement
Finer-grained CUDA graph capture with reduced padding waste and metadata caching that skips recomputation in decode phases.
Benchmark Integrity
Performance claims are only as trustworthy as the methodology. Every comparison uses identical parameters, the same benchmark harness, and statistical rigor that characterizes run-to-run variance against a strict noise floor.
Parameter parity
Model checkpoint, precision, concurrency, sequence lengths, sampling strategy, and random seeds matched exactly.
Statistical rigor
Reported gains are well above the characterized noise floor (~0.1% for canonical benchmarks); these are real gains, not favorable measurement noise.
Fairness auditing
Self-auditing flags asymmetries. When speculative decoding inflated random-input results, both numbers were reported with context.
Same harness
Both engines benchmarked through identical HTTP serving infrastructure: same client, same request patterns, same methodology.
Correctness validation
Every optimization is gated on output quality. Standard benchmarks (MMLU, HellaSwag, and others) are run before and after to confirm no accuracy degradation. Performance gains that compromise model quality are rejected.
Independent audit (pending)
These results have been submitted to independent third-party auditors for evaluation. This page will be updated with their findings when complete.
What This Means for Hardware Partners
A repeatable methodology, not a one-time result. Your hardware specifications go in; optimized inference engines come out.
Day 1
Model Enablement
New models supported on your hardware from release day, with performance that demonstrates your silicon’s true capabilities.
16%+
Beyond Baseline
Model-specific optimizations consistently unlock performance that general-purpose frameworks leave on the table.
Provable
Reproducible Results
Every benchmark backed by variance analysis and fairness auditing. Mathematically defensible numbers your customers can trust.
Your Hardware. Frontier AI Performance.
Infinity delivers Day 1 model enablement for any AI accelerator. Your hardware specs are the input; a production-ready inference library is the output.