Notes on the software I’m building.
Mostly distributed data systems, AI infra, and other low-level backend systems.
This paper explores why LLMs hallucinate, tracing the robust inverse relationship between hallucination and a fact's prevalence in the model's training data.
The pieces of Llama inference in Rust: model loading, tensor operations, attention, and quantized matrix kernels. It runs small Llama models on a consumer CPU.
Inside InferRS
Read model weights from a memory-mapped GGUF file.
A key-value store with a shared storage engine and two replication paths: Raft and leaderless quorum.
The elected leader orders writes in one log. A majority replicates the entry before it commits.
Ski lift events move through RabbitMQ into DynamoDB. Fleet-wide admission control keeps replicas within a shared rate limit.