Speculative Decoding for Gemma 4

Speculative Decoding for Gemma 4 is a speed trick that runs two models at once to generate text faster. It works by using a small, fast "drafter" model (a 76-million-parameter helper) to guess multiple words at a time. The main target model then checks the draft and accepts it if it is right almost every time, giving a massive speed up. Without this, every word branches a hundred ways and the draft can guess wrong, causing the speed up to collapse. The same model and prompt can produce different speeds depending on what you ask.

This matters for AI development because Google's Gemma 4 ships its own built-in drafter, sharing the target model's KFO cache (a memory store for past calculations). What started as an inference hack from 2022 is now part of the architecture. If you run Gemma in production, this upgrade is not optional. Drafters are released alongside the main Gemma 4 lineup, meaning developers get immediate, concrete performance improvements without rethinking their system. For beginners, this means building faster apps or agents that can run locally on a phone or Raspberry Pi without cloud dependency or API costs. Speculative decoding turns a theoretical boost into a practical, built-in feature you can use today.

To use Speculative Decoding for Gemma 4, think of it as a speed hack where two models work together. It is not a trick anymore; it is becoming architecture. Gemma 4 ships its own drafter (a small, built-in helper model) with 76 million parameters. This drafter shares the target model's KV cache (a memory shortcut that stores previous calculations). Here is how it works step by step.

When you prompt Gemma 4, the small drafter guesses multiple upcoming tokens (predicted word pieces) at once. For structured or predictable text, it is right almost every time. The main Gemma model then verifies the drafter's guesses in one batch, giving you a massive speed up versus generating one token at a time. However, performance depends on your task. For creative writing, where every word branches a hundred ways, the draft guesses wrong often, and the speed up collapses. The same model with the same prompt runs at different speeds depending on what you ask.

Drafters are released alongside the main Gemma 4 lineup. To utilize this, ensure you are running a version that supports the built-in drafter. Avoid early downloads that had a bug breaking tool calls; always get the corrected file format. Speculative decoding is evolving from an inference hack into a standard feature, so if you ship anything running Gemma in production, this is the upgrade you do not want to skip. It allows you to use two models at once for faster inference.

Speculative decoding (a speed trick where a small model drafts tokens and a large model checks them) is built into Gemma 4, but it has pitfalls. If your tool calls return broken JSON or agents route to the wrong handler, first check your file format — early downloads of Gemma 4 had a bug that broke tool calling and produced garbled text. That bug will make speculative decoding fail because the drafter (the small model drafting guesses) can't produce valid tokens for the checker.

For agentic loops (repeated model calls for tasks), context rot builds up faster with Gemma 4 than with alternatives like Qwen 3.6, which has always-on chain of thought (reasoning traces that persist across turns). This drift causes mid-session failures. The fix: keep sessions short or reset context often.

Best practices: match the model to your hardware. On a single RTX 4090, a comparable open model runs at ~175 tokens per second, while Gemma 4 31B dense is slower. The sweet spot is the middle 26B model, which uses a clever trick to stay small in memory and run fast. For speed hacks, speculative decoding works best for deterministic tasks (code, math) where the drafter guesses right nearly every time. For creative writing, where every word branches a hundred ways, the speed-up collapses — avoid speculative decoding for open-ended prose.