TL;DR

Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM.

We compare five implementations: ours, vLLM-Omni, SGLang-Omni△, VoxServe, and M*, under Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only one to achieve sub-50 ms p95 TTFA. We maintain sub-50 ms p95 TTFA through 10 RPS and keep it below 100 ms even at 20 RPS.

Our system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1× H100 SXM instance, this translates to ~$2 per 1M characters at full utilization1. For comparison, ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M at a higher TTFA.

We open source the implementation and benchmark. Our methodology is explained below.

Defining “Real-time” TTS

Let’s start by discussing what a real-time TTS server needs to achieve. We think it’s a four-part problem:

1. Low Audible TTFA: Time from request dispatch to the first audible sample must be low.
2. Zero underruns: Once playback starts, the client must not run out of buffered audio.
3. Capacity: 1 and 2 must hold as RPS increases.
4. Non-malformed output: Speech must be intelligible.

We choose Qwen3-TTS CustomVoice 1.7B because it is one of the most popular TTS models with a permissive license.

Based on the above definition, we target low p95 audible TTFA with zero underruns while maintaining high RPS on a single NVIDIA H100 SXM.

All benchmarks run for five minutes under Poisson open-loop traffic to approximate real workloads, following Fireworks AI’s LLM benchmark. Each engine receives the complete text in a single HTTP request, while audio output remains streamed. We detect audible TTFA, reconstruct playback from received PCM, and evaluate the completed audio using Deepgram STT.

How Do Other Engines Perform?

The table below shows the upstream/default result at 1 RPS for each engine. We only apply changes for compatibility in this run.

These defaults have substantial room for improvement. We tune each serving engine for its own latency, continuity, quality, and capacity requirements.

1. Remove leading silence

The first PCM returned by a model can contain tens of milliseconds of silence before the first sustained sound. This gap pushes audible TTFA back like so:

We add a dynamic trim. It detects sustained speech from short RMS windows, removes samples before onset, and streams the remaining audio normally. This change improves TTFA by ~80ms but does not make model inference itself faster.

2. Tune frame accumulation

We also tune how many codec frames are collected before decoding and releasing an audio chunk.

Smaller initial chunks reduce TTFA, but provide less playback headroom and create more frequent decoder work. Larger chunks are easier to batch and make continuous playback safer, but delay the first audible output. A useful configuration therefore starts with a small chunk and increases the chunk size for later output.

The exact knobs differ by engine: vLLM-Omni exposes settings such as codec_chunk_frames and codec_chunk_ramp; the other engines provide equivalent chunk or stride controls. We iterate over these values to find the config that best matches: low p95 TTFA, zero underruns and stable behavior as load increases.

Performance after tuning existing serving engines

The following table shows the selected no-underrun profile for each engine after leading-silence and frame-accumulation tuning.

VoxServe reaches sub-50 ms p95 TTFA at 1 RPS, while the other three engines do not. By around 6 RPS, every engine is at roughly 100 ms p95 TTFA or higher2.

How We Optimized Qwen3-TTS

We first need to understand Qwen3-TTS architecture. It is a 3-part model performing hierarchical multi-codebook generation. The Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15 codebook tokens, and the causal Codec converts codebook tokens into waveform samples.

Each module has its own compute profile, batching behavior, and latency requirements. Rather than optimizing each module in isolation, we focus on a broader question: how should a serving system coordinate these heterogeneous tasks?

1. Bringing three modules under one scheduler

Most Qwen3-TTS serving implementations are split into two stages: the Talker and Code Predictor run together, while the Codec runs separately. This separation enables token generation and waveform decoding to overlap across requests.

We take this a step further. We expose the Talker, Code Predictor, and Codec as three independently schedulable tasks. The key is not merely splitting them into parts, but bringing all three onto a shared scheduling surface managed by one scheduler. This design draws inspiration from M* (arXiv).

With this setup, the scheduler can decide whether to run the Talker, advance the Code Predictor, or prioritize a Codec job that is approaching its playback deadline. It can also batch requests waiting for the same module. Instead of following a fixed execution order, we can rearrange work according to urgency.

Combining the Talker and Code Predictor may appear more efficient because it removes an intermediate boundary. However, the combined operation can become a non-preemptible unit of work that blocks more urgent Code Predictor or Codec jobs. Keeping the modules separate creates shorter units of work and gives the scheduler more opportunities to interleave requests.

2. Scheduling around the needs of speech streaming

Speech streaming has two distinct notions of urgency.

Before the first chunk of audio arrives, every millisecond increases TTFA, so we need to prioritize this path. But once playback begins, the goal changes: the next chunk only needs to arrive before the current audio finishes playing. Producing it earlier provides no user-visible benefit.

Thus, we give high priority to requests that have not produced their first audio, while established streams become urgent only as they approach a playback deadline.

Running every urgent request alone would destroy batching efficiency. Instead, our scheduler selects an urgent request as an anchor and fills the rest of the batch with compatible work. This helps the critical request meet its deadline while making effective use of the GPU.

This policy works especially well because all three modules share a scheduling surface, allowing the scheduler to choose both the request and the pipeline stage to advance.

3. Exploiting the regular structure of the Code Predictor

The Code Predictor is an autoregressive transformer, but its execution is unusually regular. It always performs a fixed number of steps (15) per frame to fill the remaining audio codebooks.

We exploit its fixed structure to preallocate its KV cache and capture the entire frame-generation loop as a single CUDA graph. We also use a Triton attention kernel specialized for its short, bounded context.

By replacing a host-driven sequence with a fixed GPU program, we lower latency and simplify the execution system.

4. Rebuilding the Codec around cached state

The Qwen3-TTS Codec is made up of Transformers and CNNs. Generating the next audio chunk depends on both the Transformer context and convolutional state from previous chunks.

A naive implementation reprocesses the full frame history on every update, repeatedly decoding old audio as the utterance grows.

To avoid this, we use a state-cache-based Codec. Each request retains the Transformer context and convolutional state needed by the next chunk. Incremental decoding then reuses this cached state and processes only newly arrived frames instead of replaying the full history.

Initializing the state cache from the first frame adds overhead and hurts TTFA. We therefore use full decoding for the first audio, then switch to state-cached incremental decoding for efficient sustained playback.

We similarly vary chunk sizes over the course of a request. Smaller chunks let playback begin quickly, while larger chunks improve batching and GPU efficiency during sustained playback.

5. Additional serving optimizations

We capture CUDA graphs for a predefined set of batch sizes. If a ready cohort exceeds the largest captured batch size, we split it across scheduling turns rather than falling back to eager mode.

We also avoid unnecessary CPU–GPU synchronization. For example, while EOS is suppressed, generation cannot terminate, so we defer the termination check until EOS is enabled. This lets the CPU prepare and submit subsequent work without waiting for the GPU.

Finally, we support input streaming for modular speech-to-speech systems. As an upstream LLM generates tokens, the TTS model can begin synthesizing speech before receiving the complete response, reducing end-to-end latency.

What’s Next?

Qwen3-TTS is just the beginning of our work on multimodal inference. We plan to extend our scope to image, video, and world models, as well as fine-tuning. Our ultimate vision is to simulate the world 1:1 through realtime multimodal inference.

We are a team of experts in multimodal AI research and infrastructure. Our open TTS model, Dia, has been downloaded over two million times and has ranked #1 on Hugging Face. Our team of ex-YC, ex-KRAFTON, and ex-NAVER engineers has published research at NeurIPS and ICLR and earned three IOI and ICPC World Finals gold medals. Nari Labs is backed by Y Combinator.

If you want to work with us on anything multimodal, let’s chat.

△ Upstream SGLang-Omni support was still incomplete at the time of testing in mid-August 2026. GitHub

1 This estimate excludes networking, idle capacity, and operational overhead.

2 Note that the only change we make outside the configuration file is leading silence trimming. Despite our effort, there might be a configuration that slightly edges out ours.