Performant Full-Model Inference on d‑Matrix Corsair: 20x Speedup

Infinity and d-Matrix worked hand in hand to optimize the performance of Qwen 3 by 20x tokens / second on a single Corsair card compared with Infinity’s original implementation. Context length is also extended by 16x.

Corsair is a memory-centric inference accelerator where compute is tightly integrated with on-chip SRAM (the same general category as chips like Groq or Cerebras) and is designed to sidestep the GPU memory bottleneck where model weights must constantly move across the memory wall. The Corsair is broken into two packages per card, each organized into a hierarchy of chiplets, gangs, slices, cores, and SRAM banks. Mapping a model onto this hardware requires efficient memory packing across the spatial and temporal dimensions. Each weight, activation, and cache must be laid out in memory to prevent simultaneous same-region writes. The Infinity inference optimization system extended the d-Matrix Model Builder SDK to lay the model out across both dimensions: parallelism was matched to each operation spatially, memory was planned around residency and lifetimes temporally, and tooling was created to verify the resulting program end to end.

Tensor Parallelism + Head-Parallel Attention + Batch Parallelism + Pipelining

For Qwen 3, dense work is distributed across 16 hardware gangs. The primary axis is weight-sharding: each gang owns SRAM-resident output-column shards of the Q, output, and MLP projection weights. Qwen’s 16 query heads map one per gang; eight KV heads are duplicated across gang pairs; AllGather reconstructs complete activations where needed.

Inside each gang, weights are further sharded across four slices and their cores for parallel dot product accumulation. Because attention computation depends on state specific to each sequence, the KV cache and attention masks must reside on the slices performing that sequence’s computation. We therefore combine four forms of parallelism.

Decode deliberately switches execution modes at attention. RMSNorm, QKV, output projection, and the MLP run jointly across the batch as dense TP-sharded kernels, preserving useful kernel size and avoiding redundant work per sequence. KV updates, QKᵀ, softmax, and value-cache reads then run in private per-slice lanes because their state and addressing are sequence-specific. Those slice outputs are gathered before rejoining the shared dense path.

Load, Prefill, Decode

Parameter loading, prefill, and decode have different lifetimes, shapes, and scheduling priorities. While we currently run prefill and decode both on the Corsair, separating them also lets us presage disaggregated inference, using the Corsair to perform Decode while performing Prefill elsewhere.

Prefill preserves slice ownership through one gated graph dispatched once per batch; only the active batch’s local attention and KV writes are enabled.

During decode, prompt length, sequence slot, and iteration number determine cache addresses and attention masks. This allows a single compiled program to support varying sequence lengths while preserving an independent KV cache trajectory for each sequence. d-Matrix architected its dynamically parameterized instructions with this variability in mind. Reverse Polish Notation, or RPN, expressions compute the required addressing and masking parameters at runtime.

We package the model into three explicit phases:

The SRAM Advantage

The big advantage of an SRAM chip like the Corsair is that the large projection and feed-forward weights remain in SRAM across prefill and decode. Only small phase-specific state is rematerialized. Every layer needs its own allocation in the correct package, gang, slice, core, and bank.

Weights and KV-cache state persist, while activation and scratch regions are recycled as layers and iterations complete. This separation of lifetimes makes the full layout fit.

Agentic Tools for Full-Model Enablement

Three tools drove the optimization loop together: hardware probing and profiling to measure the chip, MemoryScope with its sanitizer to plan and prove the placement, and Agentic Debugging to inspect what actually happens on the device.

Agentic Hardware Probing and an Agentic Profiler

A datasheet gives useful ceilings. It does not tell us the latency of a real instruction at our tile shape, the effective bandwidth at our access stride, or where a particular memory path becomes a bottleneck.

We built dependency-controlled microbenchmarks that isolate one behavior at a time, then used end-to-end profiling to reveal which behaviors mattered in the complete graph. The measurements compose into an empirical performance model, giving each kernel a speed-of-light bound. This lets us ask: is meaningful headroom left, or should we move to the next bottleneck?

MemoryScope AI Compiler and Graph & Memory Sanitizer: Plan, Then Prove

At model scale, a program may compile and run while one operation overwrites a live tensor, reads bytes never produced, or interprets a valid buffer with the wrong layout. Our tooling addresses the problem at three levels: measure the execution, plan and lint the memory, then inspect numerical state.

Model shape determines weights and intermediates, context length sets KV depth, and batch size sets the number of caches. Infinity’s AI compiler, MemoryScope, searches placements and parallelism choices across all three.

Explicit lifecycles let transient regions overlap safely. Locality keeps weights resident, co-locates producers and consumers, and penalizes cross-slice traffic. The MemoryScope plan is then lowered into the d-Matrix SDK, generating the Qwen 3 workload.

The companion Graph & Memory Sanitizer then reconstructs reads, writes, and operation dependencies from compiled instruction streams. It flags conflicting writes, missing producers, incomplete producers, capacity overflows, uninitialized accesses, and suspicious reuse. It runs on every generated program, turning many hardware mysteries into compiler-time failures with concrete memory receipts. After our initial implementation, we rewrote the sanitizer in Rust, speeding it up by ~90x, fast enough to integrate into development pipelines as a static check.

Agentic Debugging: Turn Inspection Into a Workflow

d-Matrix’s original debugger could set breakpoints and inspect the chip’s address spaces. Together, we generalized it for custom load-prefill-decode programs, then documented the conventions an engineering agent needs to drive it. The agent can bisect execution, compare intermediate memory with references, and localize numerical errors end to end, compressing cases that once required days of manual bisection.

Building a Performant Model, Not Just a Fast Kernel

No single technique enables full-model performance. It comes from several choices reinforcing one another and from making each choice reusable for the next model and the next optimization.

Four Core Ideas

The work happened as a continuous joint loop with the d-Matrix engineering team. Model behavior informed SRAM placement; hardware measurements reshaped the parallel graph; integration needs extended the kernel library and Model Builder SDK; and debugging discoveries improved the tools.

There is more performance to unlock, alongside larger models, longer contexts, and broader reuse of the memory and debugging infrastructure.