LLM Inference Engine

An inference engine written from scratch for Apple Silicon. It loads a GGUF model file, tokenizes with a Llama 3 byte-level BPE tokenizer, runs the transformer and generates text. The CPU reference path came first, then Metal GPU kernels compiled at runtime, so the whole thing builds with Command Line Tools alone. The target model is Llama 3.2 3B Instruct, and llama.cpp is the reference for both correctness and speed.

What I built

• A GGUF parser that memory-maps tensor data, a Llama 3 tokenizer, and a CPU forward pass that serves as ground truth.
• Metal matvec kernels written for the memory-bound decode step: quantized blocks are streamed with one SIMD group per four rows, sharing the input loads.
• A sampler with temperature, top-k, top-p and repetition penalty, and a multi-turn chat REPL that keeps the KV cache across turns.
• An HTTP server that speaks the OpenAI chat-completions API with streaming over SSE, so any existing client works unchanged.
• Tests that check token ids and greedy output against llama.cpp, plus a benchmark scoreboard against it on Metal.

Made with

• C++20
• Metal
• Objective-C++
• GGUF