InferRS is a minimal LLM inference engine built entirely in Rust. It runs Llama models on the CPU. It was designed to tackle the memory overhead and dependency weight common in standard AI stacks by keeping the architecture lean and operating close to the metal.
The project achieves this through a strict focus on structural efficiency. It uses zero-copy memory mapping for GGUF tensor loading, so the weights are never copied into a second buffer and the cost moves to page faults in the first forward pass rather than a load step. For faster and more efficient token generation, the engine implements key-value caching and Grouped-Query Attention. To maximize computational throughput, InferRS keeps INT8 weights quantized in memory and computes against them directly, through multi-accumulator kernels that split each dot product eight ways. This allows complex tensor operations to run quickly on standard hardware without relying on heavy external abstractions.
Ultimately, InferRS serves as a hands-on demonstration of how low-level systems programming can optimize AI execution. By leveraging Rust's memory safety and explicit control over hardware resources, the project provides a fast and transparent path to local model inference. It reads F32, F16 and Q8_0 weights.