infinite songs in your pocket

I wanted one thing: a music model that runs on a phone with the radio off. No server round-trip, no streaming a prompt to someone else's GPU and waiting. Just weights on the device, audio out. This is the log of getting there — what broke, what I cut, and the afternoon it finally sang back.

the memory budget

The first wall wasn't quality, it was RAM. The model fit on disk fine; loading it without the OS killing the app was the problem. A phone gives you a budget, not a limit — cross it and you don't get an error, you get terminated.

The fix wasn't a smaller model. It was being honest about what had to stay resident. The encoder runs once per generation; the decoder runs every frame. So I kept the decoder pinned and let the encoder page in and out. Counterintuitive, but the working set is what matters, not the total.

the model didn't need to be smaller. it needed to forget more gracefully.

the latency wall

On-device, the enemy is the first token. Cloud models hide cold starts behind a load balancer; here, the user is staring at a spinner while the KV cache warms up.

warm starts

The trick is to prime the cache before the user asks for anything — load the weights, pin them, and push a seed sequence through so the first real token is a warm continuation rather than a cold start.

// warm the kv cache before first token
const sess = await Atlas.load("atlas-8.gguf", {
  ctx: 2048,
  mlock: true,        // pin weights in ram
  threads: 6,
});
await sess.prime(seedTokens);

what actually moved the needle

• pinning weights with mlock cut cold-start jitter to near zero
• quantizing the decoder, not the encoder — the encoder is where quality lives
• streaming partial audio frames instead of buffering whole bars

None of these were clever. They were just the difference between a demo and something you'd actually keep on your home screen.

it sang back

The first time it worked end to end I had it generate eight bars over a chord I hummed badly into the mic. It came back with something that wasn't good, exactly, but was mine — generated on the device in my hand, plane mode on, nothing leaving the phone.

That's the whole pitch, really. Not that the model is big or smart, but that it's yours, sitting in your pocket, working with the radio off.