Voice AI inference: under 80 ms time to first token on Gemma 4

Hitesh Jain & Divy Vasal · 2026-09-21

An AI team building a voice application on CoralBricks asked us to benchmark Gemma 4 against a clear target: under 500 ms p50 time to first token with conversations growing to 10–15K tokens.

We got the benchmark configuration below 100 ms at concurrency 1: 69.2 ms p50 and 79.9 ms p95.

The p95 mattered even though the team had not asked us to optimize it. We had built voice systems before, and knew that slow turns become awkward pauses a user can hear. Every millisecond removed from inference also leaves more budget for speech recognition, turn detection, text-to-speech, animation, and the product work users actually see.

This is an anonymized benchmarking case study, not a description of the client's production deployment. We cannot disclose the client or their complete production configuration. The measurements below come from controlled replays on isolated replicas and are an example of the benchmarking work we do for teams building on CoralBricks.

The work split in two: right-size the sequence length so one H100 was enough, then remove latency with a new chat template for Gemma 4 and incremental tokenization.

The workload

We used PygmalionAI/PIPPA, a corpus of real Character.AI-style conversations. We replayed the human turns in order and had Gemma write the character side. Each new request carried the growing conversation, so the test exercised the exact property that matters here: a short new turn on top of a long, mostly unchanged history.

The model was google/gemma-4-26B-A4B-it. Responses streamed. TTFT ran from request start to the first non-empty content token. Unless stated otherwise, latency comparisons use concurrency 1: the clearest view of the latency experienced by one interactive session. We show the final concurrency sweep near the end.

We ran more than 30 experiments, mostly on H100 hardware. The hardware-sizing comparison below used one H200.

Each comparison below stays within the experiment that produced it. We do not combine independent runs into a synthetic A/B.

The hardware decision

Gemma 4 has a native sequence length of 131,072 tokens. This voice workload did not need it. Conversations topped out around 15K tokens, so reserving memory for the full window would have meant paying for capacity the application would never use.

We capped the benchmark serving window at 49,152 tokens. That still left more than 3× headroom over the longest expected prompt, while letting the 51.6 GB bf16 checkpoint and its KV cache fit on a single 80 GB H100 at tensor parallelism 1.

We evaluated all three plausible serving options: one H100 with a 49K window, two H100s at tensor parallelism 2, and one H200 serving the full 131K window. Two H100s produced no significant latency benefit. Gemma already fit on one card, so the second GPU added cost and inter-GPU communication without moving the result enough to justify either.

The H100 delivered 141 ms p50 and 548 output tokens per second. The H200 delivered 127 ms p50 and 681 output tokens per second. That was 24% more output per GPU, but only a 10% reduction in p50 TTFT. Its larger memory and higher bandwidth were useful, but the application did not need a 131K-token serving window. For a latency-sensitive workload capped around 15K tokens, that was not enough to displace the cheaper H100.

The 49K cap did not make Gemma intrinsically faster. It made the benchmark configuration cheaper: one H100 per replica, with more than 3× the context the application required. Two H100s did not earn their cost, and the H200's modest latency advantage did not earn its larger memory for this workload. We kept tensor parallelism at 1.

Latency optimizations

The optimized benchmark reached 69.2 ms p50 and 79.9 ms p95 at concurrency 1. Both the typical turn and the p95 stayed below 100 ms.

Two pieces of CPU work sat in front of the model. Gemma's chat template did unnecessary work as the conversation grew, then the serving path rendered and tokenized the mostly unchanged history again on every turn. We measured and removed them separately.

How we got there

1. The chat template was doing quadratic work

Character chats grow in messages as well as tokens: a 15K-token history can contain hundreds of short turns. At 593 messages, Gemma 4's shipped template took 719 ms just to render the conversation. The replacement took 47 ms.

The first useful clue was that TTFT still grew with context when the GPU was idle and the prefix was warm: about 27 ms for every additional 1K tokens. The handful of new tokens could not explain it.

Gemma 4's shipped Jinja template scanned forward through the remaining messages for every message in the conversation, looking for the next non-tool role. Jinja has no break, so finding the answer did not stop the scan. Rendering grew quadratically with message count.

We rendered real conversations plus tool, reasoning, and multimodal fixtures through both versions; the prompt text stayed identical.

The concurrency-1 warm-prefix benchmark showed the shape cleanly. At 2,162 prompt tokens, p50 was unchanged at 107 ms. At 12,958 tokens, it fell from 340 ms to 187 ms. The gain appeared as the history grew, exactly where a quadratic template should hurt.

A separate 16-stream stress run showed the same effect more dramatically: at 12.5–15K prompt tokens, p50 fell from 1,092 ms to 290 ms with the linear template. Decode speed was unchanged, so this was not a disguised trade of first-token latency for slower generation.

2. Incremental tokenization removed the work that remained

The linear template fixed a bad algorithm, but the server still rendered and tokenized the entire conversation on every request just to discover that almost all of it matched the cached prefix.

The incremental path retained the rendered and tokenized history for a conversation. On the next turn it processed the new suffix and sent the same prompt token IDs downstream. If the request shape, template, tokenizer, or cached prefix did not match, it fell back to the normal full path.

At concurrency 1, holding the linear template and request set constant, incremental tokenization improved both the median and the tail:

Concurrency 1	Full tokenization	Incremental tokenization	Change
p50	75.8 ms	69.2 ms	9% lower
p95	91.8 ms	79.9 ms	13% lower

The benefit grew with history and load. In a separate 16-stream run, p50 was effectively unchanged at 0–2K tokens: 193.5 → 194.3 ms. At 10–12K tokens it fell 645.8 → 383.3 ms; at 12–15K, 679.9 → 395.9 ms. There was almost nothing to reuse near the start of a conversation and much more to remove near the end.

Why we worked on the tail without being asked

A p50 target is common, but it is not a voice-grade inference target. Half of all turns are slower than the p50; only one in twenty is slower than the p95. In a text product, a slow response may look like a spinner. In voice, it becomes jitter, drag, or dead air. And inference still has to leave latency budget for speech recognition and text-to-speech.

What did not work

We tried the obvious inference knobs first. Mixed-chunk scheduling, a unified radix tree, and smaller KV pages did not bend the long-context slope. A different MoE kernel reduced decode speed. Speculative decoding accepted only 2.4 tokens from each six-token draft on this chat workload and made decode 15–25% slower.

Those negative results pointed back to the CPU request path. The GPU was waiting behind work that GPU tuning could not remove.

The end-state load curve

While we used concurrency 1 for the comparisons above, we also replayed the same 897-request set against the optimized configuration from concurrency 1 through 16. The median prompt was 3,147 tokens, and 90% were no longer than 6,081 tokens.

Concurrent streams	p50 TTFT	p95 TTFT
1	69.2 ms	79.9 ms
4	126.6 ms	140.9 ms
8	184.5 ms	202.3 ms
16	199.1 ms	320.0 ms

This table is the end state: the linear template and incremental path were both enabled. It shows the latency cost of increasing load without mixing that question into the component comparisons.

What the benchmark established

On one H100, the optimized configuration reached 69.2 ms p50 and 79.9 ms p95 at concurrency 1. That is a controlled benchmark result demonstrating inference optimizations behind CoralBricks.

The technical result is a faster first token. The product implication is less jitter and more latency budget for the rest of the voice pipeline. The organizational implication matters too: a team building on CoralBricks can spend more time on the voice experience instead of operating an inference stack or rediscovering the same long-context failure modes.

The team did not ask us to optimize p95. We did it because we had seen this movie before: in voice AI, the typical turn does not break the experience. The occasional bad one does.