No Python, no Docker, no CUDA

I wanted a robot that could hear me, think, and talk back, with no uplink.

Not a demo on a laptop. The Pi inside the chassis.

The tools for that mostly want a Python environment, a container, or CUDA. A Pi inside a robot has none of those.

So on 24 May 2026 I started writing my own engine. In Rust.

This is what the next five months looked like. The parts that worked, and the parts I would rather not tell you.

Day one

The first commit and the first release were the same day.

Install script. Checksums. Cross-compiling for ARM. A release pipeline that built for macOS, Linux and Windows before the thing could do anything useful.

It could not run a model yet.

Real CPU inference landed the next day, in v0.1.2. The first chat output was not coherent. The fix for that is also from day two.

I am telling you this because every polished project you look up to started as a binary that printed garbage.

Day six

29 May. Five releases in one day.

Quantised GGUF loading, so memory matched the file size instead of expanding to 32-bit floats. Then SIMD and threads, 4.7 times faster. Then Metal for Apple Silicon. Then memory-mapped loading, for models bigger than RAM.

Within two weeks it had ears and a voice. Whisper on 2 June. Kokoro text-to-speech on 3 June.

Alongside that, a GPU path that runs on Intel, AMD and Nvidia through Vulkan. Because the one thing a robot never has is the exact GPU you developed on.

The release that ran out of memory

v0.4.4. 5 June.

On a 16 GB MacBook, a 1.5B model through that GPU path held 6.8 GiB of weights on the GPU and peaked at 14.7 GB of process memory.

It ran out of memory and returned empty replies. I shipped it.

v0.5.0 kept the weights quantised on the GPU and dequantised them inside the shaders. 1.06 GiB resident. 3.6 GB peak. It worked.

The same binary ran on a Jetson AGX Thor through Vulkan. Zero CUDA, zero JetPack. 2.2 tokens a second on the CPU became 10 on the GPU.

That was the first time "no CUDA" was true on Nvidia hardware and not just a line on a website.

The bug I shipped for five weeks

Until July, Llama-3.2-1B ran at 1.3 tokens a second on a Raspberry Pi 5.

That is not a chat. That is a typewriter.

Then an audit found the engine dequantising the entire embedding table on every generated token. Llama-1B ties its embedding to a 128k-token vocabulary. About 0.8 GB of dequantisation per token, to read one row.

Embedding lookup became row-wise.

Llama-1B went from 1.3 to 8.3 tokens a second on the same board. The 3B went from 0.8 to 3.4. Even the MacBook's CPU path went from 6.6 to 38.7.

That code path had been there since quantised loading landed on 29 May. Nothing flagged it, because a slow Pi looks normal.

If your thing is slow, do not assume it is the hardware. Profile it. I lost five weeks of Pi performance to it.

Make the engine watch the thermometer

Sustained decode pins all four cores of a Pi.

On passive cooling the chip reaches the 85 °C firmware trip and every core hard-throttles. Throughput collapses, then oscillates.

So the engine samples the temperature twice a second. From 80 °C it drops one core at a time, never below half. Below 70 °C it gives them back.

To test it I forced the threshold inside the board's normal plateau, at 72 °C.

Decode went from 8.70 to 8.23 tokens a second within seconds, and held there for the rest of a six-minute soak.

Five percent slower for a quarter of the cores. Decode on a Pi is bound by memory latency, not compute, so you pay in package power, not tokens.

The same governor ships in the phone SDKs now. The engine sheds threads as the phone heats.

The voice loop that interrupted itself

Here is the part that was for the robot.

sapient converse --speak. Speech is transcribed while you are still talking. The reply starts speaking after its first clause. You can cut it off mid-sentence.

Perceived latency went from about 4.4 seconds to about 2.4 on a Mac.

The first time it ran on a real microphone, the replies interrupted themselves.

The speaker played the reply. The microphone heard it. The barge-in detector decided a human had started talking, and stopped the reply. Over and over.

The fix: the speaker tracks its own playback envelope. Anything that matches it is the machine. Anything that does not is you.

Now the honest number. On a Pi 5, one turn with a 0.5B model is 2.96 seconds of transcription, 3.5 seconds of language model and 5.4 seconds of speech synthesis, run in sequence.

Streaming hides most of that, because speech starts at the first clause. It does not hide all of it.

Sub-second voice on a Pi is the thing I am still working on.

Bugs I want you to know about

phi-4-mini on Metal answered everything with "mememe". One bit of a 5-bit quant was read from the wrong byte.

A sliding-window model was coherent for about 490 tokens, then garbage. A NaN in flash attention that would have bitten every sliding-window model.

The say command ran the vision command.

The download bar crawled to 8% and jumped to done, because it counted every quant in the repository.

A user reported a 4.8 GB model holding twice its size in memory on the default build. They were right. 3.4 times while loading.

The README said "zero quality loss". I had never measured it. When I built a perplexity gate that scores exactly what llama.cpp scores, Sapient came out 0.1 to 0.4% worse on the same files.

My first set of quality numbers was wrong too. Two protocol bugs.

Every one of these is in the changelog. Negative results stay in.

So, why not llama.cpp?

You are going to ask.

It is faster. On an Apple M4 with Metal, Llama-3.2-1B decodes at 90.6 tokens a second in Sapient and 111.3 in llama.cpp. Same file, same session. I publish the table.

What Sapient is: chat, speech and vision in one binary, an 8 MB download, that installs with one command and runs on the Pi, the laptop and the Jetson from the same model name.

No daemon. No runtime. No container.

I cannot give you a grand reason for Rust. I can tell you what one binary bought me. A thing I can copy onto a robot and trust to start.

Sapient is 583 commits and 52 releases in, as of today.

It is not the fastest. It is the one that runs where you are.