内容来源：https://research.google/blog/real-time-speech-to-speech-translation/

内容总结：

近日，谷歌深度思维与核心机器学习团队联合发布了一款突破性语音实时翻译技术。该技术能在外语对话过程中，仅用两秒延迟即可用说话人原声生成翻译音频，让跨语言交流如同面对面交谈般自然流畅。

传统语音翻译系统通常采用"语音转文字→文字翻译→语音合成"的串联模式，导致4-5秒的延迟，且存在误差累积、缺乏语音个性化等问题。新技术通过三大创新突破瓶颈：首先构建了可扩展的时序对齐数据采集系统，能自动筛选优质语音样本；其次采用基于AudioLM框架的流式传输架构，实现对连续音频的实时处理；最后通过SpectroStream编解码技术，将音频转化为分层表征的令牌序列，在保证音质的同时实现高效传输。

目前该技术已成功应用于谷歌会议系统及Pixel 10设备的端侧功能，支持英语与西班牙语、德语、法语、意大利语、葡萄牙语五种拉丁语系语言的实时互译。研究团队透露，印地语等更多语种的适配工作也在积极推进中。为确保技术合理使用，系统会在每次翻译前明确提示用户生成内容为合成语音。

这项突破使《星际迷航》中描绘的实时宇宙翻译器走向现实，标志着人工智能在消除人类语言隔阂的道路上迈出关键一步。随着算法持续优化，未来有望实现更符合语境的意义翻译，进一步推动全球无障碍沟通。

中文翻译：

实时语音翻译技术
2025年11月19日
Google DeepMind研究工程师 Karolis Misiunas 与 Google Core ML 软件工程师 Artsiom Ablavatski 联合撰文

我们推出了一款创新的端到端语音翻译模型，能以原说话者的声音实现仅延迟2秒的实时翻译——这项憧憬已久的技术终于成为现实，让跨语言交流变得更加自然流畅。

实时沟通是工作与生活中不可或缺的环节。当跨越语言障碍进行远程对话时，仅依靠最先进的翻译字幕难以实现真正的情感共鸣，因为它们缺乏鲜活个性与实时反馈能力。语音翻译技术的问世通过直接生成翻译音频弥补了这一缺陷，使交流更自然。现有语音翻译系统通常存在明显延迟（4-5秒）、容易产生误差累积，且普遍缺乏个性化特征。

今日我们介绍的创新型端到端语音翻译模型突破了这些限制，能以原声实现仅2秒延迟的实时翻译。该创新架构采用流式处理框架，结合时间同步数据训练，显著缩短了原始语音与翻译语音之间的延迟。为支持多语言扩展，我们开发了可扩展的时间同步数据采集流程，逐步将更多语言纳入系统。该技术已在实时性要求极高的应用场景中成功验证其有效性。

级联式语音翻译技术
早期实时语音翻译技术采用多模块级联方案：

• 首先通过自动语音识别模型将源音频转写为文本
• 接着通过自动语音翻译进行逐字翻译
• 最后通过文本转语音系统将译文转换为音频

尽管各环节组件质量优异，但实现流畅的实时语音翻译仍面临三大挑战：

• 4-5秒的显著延迟导致必须采用轮替式对话
• 翻译流程中的误差逐级累积
• 通用文本转语音技术导致个性化特征缺失

新型端到端个性化语音翻译方案
为实现技术突破，我们构建了可扩展数据采集流程并开发出端到端模型，实现仅2秒延迟的实时直译：

• 可扩展数据采集流程：通过整合现有语音识别与文本转语音技术，配合精确对齐步骤，将原始音频转化为时间同步的输入/目标数据集。采用严格过滤验证机制剔除难以对齐的样本。
• 实时语音翻译架构：基于AudioLM框架与基础Transformer模块，构建专用于音频的流式机器学习架构。该设计能处理连续音频流，让模型自主决定翻译输出时机，并利用SpectroStream编解码技术管理分层音频表征。

| 个性化语音翻译应用示例：西班牙语原声转译英语 |
可扩展数据采集流程
针对特定语言对，首先进行原始音频采集。我们采用多样化音源（包括文本转语音模型生成数据），经过清洁过滤确保单说话人源语言音频与合理噪声水平。完成初始采集后，通过语音识别转写字幕文本，利用强制对齐算法生成时间戳标注（音频-文本映射），剔除对齐失败的片段。

剩余片段经机器翻译转换为目标语言后，通过自动化过滤器验证译文准确性与输入文本对应关系。随后对原文字幕与译文进行对齐处理，生成对应时间戳标注（文本-译文映射）。

通过定制化文本转语音引擎将译文转换为音频，在保持原声音色特征的同时确保输出自然度。流程最终对译文与生成语音进行强制对齐（语音-文本映射）。

利用前三步生成的对齐结果，计算其重叠区域得出源目标音频对齐掩码，这些掩码将指导训练过程中的损失计算。

未满足延迟要求或无效对齐的翻译将从训练集中剔除。剩余对齐数据以最长60秒片段训练流式语音翻译模型，训练中同步应用多种音频增强技术，包括采样率调整、混响、饱和与降噪处理。

实时语音翻译架构
端到端语音翻译模型基于基础Transformer模块，包含两大核心组件：

• 流式编码器：根据前10秒输入内容汇总源音频数据
• 流式解码器：基于压缩编码器状态与历史预测结果自回归生成翻译音频

该模型创新性地将音频表示为二维令牌集合（即RVQ音频令牌）。如下图所示，X轴代表时间，Y轴代表描述当前音频段的令牌集合。通过机器学习编解码器，特定集合中的所有令牌可即时合成为音频流。令牌数量控制每段音频质量，更多令牌意味着更高保真度。模型按序预测令牌，优先处理起始位置令牌。通常16个令牌足以高质量表征100毫秒音频片段。

除音频令牌外，模型还输出一个文本令牌。该文本令牌既为音频生成提供额外先验信息，也支持直接计算BLEU指标而无需依赖代理语音识别系统。

训练过程中对模型施加逐令牌损失以确保翻译准确性。通过右移真实值令牌可调整模型预测延迟（即前瞻量），从而根据目标语言复杂度灵活配置。实时对话通常采用标准2秒延迟，适用于多数语言。虽然更长的前瞻量能通过提供更多上下文提升翻译质量，但会对实时交流体验产生负面影响。

除内部2秒延迟外，模型推理时间也会影响系统整体延迟。为实现实时性能，我们实施了混合低比特（int8与int4）量化、优化分类器自由引导预计算等多种优化技术。

实际应用场景
新一代端到端语音翻译技术已应用于两大关键领域，彰显实时跨语言交流的重要性：现已在Google Meet服务器端部署，并作为内置功能集成于新款Pixel 10设备。虽然两款产品采用不同的技术实现策略，但共享训练数据与模型架构。Pixel设备端语音翻译功能同时采用级联方案以最大化语言覆盖范围。为防范功能滥用，每次翻译前均会告知终端用户该翻译为合成生成。

当前端到端模型在五大拉丁语系语言对（英语与西班牙语、德语、法语、意大利语、葡萄牙语互译）中表现稳定，支撑了我们的首发产品部署。我们在其他语言（如印地语）中也观察到良好潜力，计划持续推进开发。未来改进将聚焦于增强模型前瞻动态性，使语音翻译技术能自适应处理语序与英语迥异的语言，实现更注重上下文语境而非逐字对应的翻译效果。

我们相信这项语音翻译技术的突破性进展将彻底革新实时跨语言交流，使长久以来的构想成为现实。

致谢
衷心感谢所有项目贡献者，他们的关键性工作让此技术得以实现。特别鸣谢同事Kevin Kilgour、Pen Li、Félix de Chaumont Quitry、Michael Dooley、Jeremy Thorpe、Mihajlo Velimirović、Alex Tudor、Christian Frank、Daniel Johansson、Hanna Silén、Christian Schuldt、Henrik Lundin、Esbjörn Dominique、Marcus Wirebrand、Daniel Kallander、Pablo Barrera González、Huib Kleinhout、Niklas Blum、Fredric Lindstrom、Esha Uboweja、Karthik Raveendran、Frédéric Rechtenstein、李兴、林盛毅、杨成、Jason Fan、Matsvei Zhdanovich、魏佳宁与Matthias Grundmann。

英文来源：

Real-time speech-to-speech translation
November 19, 2025
Karolis Misiunas, Research Engineer, Google DeepMind, and Artsiom Ablavatski, Software Engineer, Google Core ML
We introduce an innovative end-to-end speech-to-speech translation (S2ST) model that enables real-time translation in the original speaker's voice with only a 2-second delay — bringing long-imagined technology into reality and making cross-language communication more natural.
Real-time communication is an integral part of both our professional and personal lives. When speaking to people remotely across language barriers, it can be difficult to truly connect by just relying on state-of-the-art translated captions, as they lack personality and real-time responsiveness essential for fluid conversation. The arrival of speech-to-speech translation (S2ST) bridges this gap by directly generating translated audio, leading to more natural communication. Existing speech-to-speech translation systems often incur significant delays (4–5s), tend to accumulate errors, and typically lack personalization.
Today we describe an innovative end-to-end S2ST model that overcomes these limitations, enabling live translation in the original speaker's voice with only 2 second delay. The novel architecture leverages a streaming framework and, with training on time-synchronized data, significantly reduces the delay between the original input and the translated speech. To support a breadth of languages, we introduce a scalable time-synced data acquisition pipeline that allows us to gradually expand the system to include more languages. This technology has demonstrated its effectiveness through successful deployment in real-time sensitive use cases.
Cascaded S2ST
Prior real-time speech-to-speech technologies employed a cascaded pipeline of individual processing blocks:

• Firstly, the source audio is transcribed to text using automatic speech recognition (ASR) AI models.
• Next, the transcribed text is translated word-for-word to the target language using automatic speech translation (AST).
• Finally, the translated text is converted back to audio using text-to-speech pipelines (TTS).
  Despite the high quality of the individual cascade components, achieving a seamless, real-time S2ST experience has been challenging due to three primary factors:
• Significant delays of 4–5 seconds, forcing turn-based conversations.
• Accumulated errors at each stage of the translation process.
• A notable lack of personalization due to the general-purpose TTS technology.
  A novel end-to-end, personalized S2ST
  To significantly advance S2ST, we created a scalable data acquisition pipeline and developed an end-to-end model that provides direct, real-time language translation with just a two-second delay:
• Scalable data acquisition pipeline: We created a data processing pipeline to convert raw audio into a time-synchronized input/target dataset. This was achieved by integrating existing ASR and TTS technologies with precise alignment steps, ensuring translated audio best matches the input. Rigorous filtering and validation were employed to remove difficult-to-align examples.
• Real-time speech-to-speech translation architecture: We introduced an audio-specific streaming machine learning architecture to support training on this time-synchronized data. Building on the AudioLM framework and fundamental transformer blocks, this architecture is designed to handle continuous audio streams, allowing the model to decide when to output translations. It is also structured to manage hierarchical audio representations using the SpectroStream codec technology.
  | An example of our personalized S2ST applied to a Spanish original translated to English. |
  Scalable data acquisition pipeline
  For a given language pair, initial work starts with raw audio acquisition. We utilize a diverse set of audio sources, including data generated by TTS models. This audio undergoes a cleaning and filtering process to ensure it contains a single speaker of the source language and has an appropriate noise level. After initial data collection, an ASR step transcribes the source text. With both source audio and text available, the forced alignment algorithm generates alignment timesteps (audio-to-text mapping). Any audio segments where alignment fails are discarded.
  The remaining clips are machine translated from the source into the target language. Subsequently, a series of automated filters validate the translated output, ensuring accuracy and correspondence to the input text. Next, the original transcribed and translated texts are also aligned to generate corresponding timestamp annotations (text-to-translated text mapping).
  Using a custom text-to-speech generation engine, the translated text is converted into translated audio, preserving the voice characteristics from the original audio while producing natural-sounding output. The pipeline concludes with one more forced alignment step of the translated text and the generated speech (speech-to-text mapping).
  Utilizing the three generated alignments from the previous steps, the overlap between them is calculated, yielding alignment masks between the source and target audio. These alignment masks are then used to guide the loss computation during training.
  Invalid overlaps or translations that fail to meet delay requirements are filtered from the training dataset. The remaining aligned data is used to train the streaming S2ST model in chunks of up to 60 seconds. Various audio augmentation techniques are also applied during training, including sample rate reduction, reverberation, saturation, and denoising.
  Real-time speech-to-speech translation architecture
  The end-to-end S2ST model leverages fundamental transformer blocks and consists of two main components:
• Streaming encoder: Summarizes source audio data based on the preceding 10 seconds of input.

• Streaming decoder: Predicts translated audio autoregressively, using the compressed encoder state and predictions from previous iterations. A feature of these models is their representation of audio as a 2D set of tokens, known as RVQ audio tokens. As shown below, the X-axis represents time, while the Y-axis represents a set of tokens that describe the current audio segment. When summed, all tokens in a specific set can be readily converted into an audio stream using an ML codec. The number of tokens controls the audio quality for every segment, with more tokens yielding higher fidelity. The model predicts tokens sequentially, prioritizing those at the beginning. Typically, 16 tokens are sufficient for high-quality audio representation of a 100 ms chunk. The model outputs a single text token in addition to the audio tokens. This text token acts as an extra prior for audio generation and enables direct metric calculation (BLEU) without relying on proxy ASR systems. During training, a per-token loss is applied to the model to ensure accurate translation. The model's prediction delay, or lookahead, can be adjusted by shifting ground truth tokens to the right, allowing for flexibility based on the target language's complexity. For real-time conversations, a standard 2-second delay is typically used, which is suitable for most languages. While a longer lookahead improves translation quality by providing more context, it negatively impacts the real-time communication experience. In addition to the internal 2 second delay, the model's inference time contributes to the overall system latency. To minimize this and achieve real-time performance, we implemented several optimization techniques, including hybrid low-bit (int8 and int4) quantization and optimized CFG precomputation. The examples of translation for different language pairs with developed models with corresponding ground truth (taken from publicly available CVSS dataset) follow:	Direction	Input audio	Translated audio	Ground truth
  Spanish to English
  In coastal areas there is a larger accumulation of water molecules in the air.
  English to Spanish
  Su portaaviones insignia, el Portaviones Formidable fue tocado por un kamikaze sin gravísimas consecuencias.
  German to English
  The electrician used a piece of aluminum foil to splice the fuse.
  English to German
  Margaret versucht mit allen Mitteln, die bevorstehende Katastrophe zu verhindern.
  Italian to English
  By clicking on the right, on the bell icon, you can activate Pash notifications so that they are updated in real time.
  English to Italian
  La voce popolare parla dei proprietari terrieri, dei mafiosi e dei rappresentanti del Partito conservatore e dei loro nomi, che sono noti a tutti.
  Portuguese to English
  This text is made available under the respective license.
  English to Portuguese
  Um homem de camisa azul observa algo projetado na frente dele.
  French to English
  The volunteer firefighters are a full part of our civil security arsenal.
  English to French
  Ce gouvernement et cette majorité portent donc seuls la responsabilité de cette situation.
  Exemplar bidirectional translation generated by the trained models for English, Spanish, German, Italian, Portuguese, and French languages with corresponding ground truth.

  Real-world applications
  The new end-to-end S2ST technology has been launched in two key areas, highlighting the importance of real-time cross-language communication. It is now available in Google Meet on servers, and as a built-in on-device feature for the new Pixel 10 devices. Although the products utilize different strategies for running the S2ST pipeline, they share training data and model architecture. The Pixel Voice Translate on-device feature also employs a cascade approach to maximize language coverage. To mitigate potential feature misuse, prior to each translation session, we inform the end-user that the translation is synthetically generated.
  The current end-to-end model delivers robust performance for five Latin-based language pairs (English to and from Spanish, German, French, Italian, Portuguese), enabling our initial product launches. We are also observing promising capabilities in other languages, such as Hindi, that we plan to develop further. Future enhancements will focus on improving the dynamism of the model's lookahead. This will enable the S2ST technology to seamlessly adjust to languages with word orders significantly different from English, facilitating more contextual rather than literal word-for-word translation.
  We believe that this breakthrough in S2ST technology will revolutionize real-time, cross-language communication, turning a long-envisioned concept into reality.
  Acknowledgements
  We are sincerely grateful to everyone who contributed to this project; their critical contributions were instrumental in making it a reality. We are particularly thankful to our colleagues Kevin Kilgour, Pen Li, Félix de Chaumont Quitry, Michael Dooley, Jeremy Thorpe, Mihajlo Velimirović, Alex Tudor, Christian Frank, Daniel Johansson, Hanna Silén, Christian Schuldt, Henrik Lundin, Esbjörn Dominique, Marcus Wirebrand, Daniel Kallander, Pablo Barrera González, Huib Kleinhout, Niklas Blum, Fredric Lindstrom, Esha Uboweja, Karthik Raveendran, Frédéric Rechtenstein, Xing Li, Shengyi Lin, Cheng Yang, Jason Fan, Matsvei Zhdanovich, Jianing Wei, and Matthias Grundmann.