Voice-first computing demands a shift from simple speech recognition to context-aware processing. Unlike traditional automatic speech recognition (ASR) pipelines that transcribe audio verbatim—capturing noise and filler—Gemini 3.5 Transcribe processes audio into clean, structured text in real time.
This article covers how Gemini 3.5 works and shows you how you can leverage it in your own work.
By Wei Hsia, Developer Advocate, Google Cloud
How Gemini 3.5 Transcribe Works
Before we get into examples, we’ll share how Gemini 3.5 Transcribe actually works. Gemini 3.5 Transcribe (launched this August) is Google’s most precise speech-to-text model yet. Designed for intelligent voice interactions, it converts raw audio into accurate, formatted text.
Gemini 3.5 Transcribe provides two endpoints:
- gemini-3.5-transcribe (Interactions API): Transcribes audio files up to one hour with speaker diarization, word timestamps, and custom vocabulary biasing.
- gemini-3.5-transcribe-live (Live API): Streams continuous audio over WebSockets with sub-second latency for live interactions.
Both endpoints offer verbatim transcription and smart transcription, which removes filler words ("ums", "ahs") and resolves spoken self-corrections. This post focuses exclusively on the gemini-3.5-transcribe-live (Live API), as its sub-second WebSocket streaming is essential for real-time, low-latency interactions.
Gemini 3.5 Transcribe is a significant iteration in how these models operate, and the following sections will show how you can leverage these capabilities for your own use cases.
Use case: Building a Debate Agent
A real-time debate agent is a practical example of a system that requires both speed and accuracy. In live fact-checking and debate analysis, waiting for a speaker to finish a statement before starting search or verification introduces too much lag.
*The standard debate is whether pineapple belongs on pizza.
This is where multi-agent pipelines built with the Agent Development Kit (ADK) come in.
Instead of waiting for a complete turn, the Live API streams interim transcription events over the WebSocket as speech occurs. As each phrase is recognized—prior to Voice Activity Detection (VAD) signaling a finished turn—the system can extract partial claims and trigger Google Search grounding in the background. By running retrieval while the user is still speaking, the agent has the necessary evidence ready the moment the speaker pauses.
Step 1: Connect to the Live API
Open a bidirectional session over WebSockets using the Google Gen AI SDK to stream audio and receive real-time transcriptions.
First, install the Google Gen AI SDK as a python library.
The following code will send your audio input to Google Cloud and transcribe it using the Live API, returning the text to be processed by the agent.
Step 2: Add Intent Filtering and Search Optimization
Now that you have the live transcriptions, the next step is to have the agent take in the text and perform the rest of the steps. Using the Agent Development Kit (ADK), structure an orchestration flow that routes incoming interim phrases through Gemini 3.7 Flash for high-speed evaluation. The router determines whether a sentence actually warrants making a call to another tool to ground the claim. If the speaker is making a subjective point, sharing an opinion, or adjusting their phrasing mid-sentence, the flow ignores search triggers entirely. This prevents unnecessary tool calls, reduces latency, and eliminates noise in downstream reasoning.
Step 3: Response Generation and Multilingual Practice
Once the speaker finishes and the retrieval tools return grounded context, the debate engine compiles the rebuttal. The generated response is passed to Cloud Text-to-Speech (google-cloud-texttospeech), which converts the text back into natural audio and streams it to the user.
Because the pipeline supports over 85 languages out of the box, you can practice debates across different languages without changing the core architecture. The agentic setup also lets you adjust system instructions to configure debate difficulty, persona tone, and judging criteria.
Getting started
Combining low-latency streaming transcription with parallel agent workflows eliminates the dead air typical of traditional voice applications. While this post showed how to create a debate agent, this real-time routing pattern applies to many practical interactive voice scenarios:
- In-Meeting Fact Lookup: Live assistants that monitor ongoing discussions, detect when participants reference specific metrics or company projects mid-sentence, and automatically display relevant charts, docs, or stats on-screen.
- Call Center Agent Assist: Customer support pipelines that transcribe calls in real time, identify customer issues or policy questions as they speak, and pull up exact troubleshooting steps before the customer finishes explaining their problem.
- Interactive Voice Agents: Customer-facing voice bots that stream user speech directly into tool-calling pipelines, resolving user intents without waiting through awkward multi-second processing pauses.
- Live Broadcast and Event Captioning: Streaming captions across 85+ languages with automatic vocabulary biasing for product names, speaker names, and technical terms.
To get started and view the full API specifications, check out the Gemini 3.5 Transcribe documentation.