今天推荐的是我们的社区成员 BoJack 创建的 GitHub 仓库，如果你在关注 Voice Agent 开发，想了解最前沿的语音模型都有哪些，这个仓库的列表就非常值得关注。
BoJack 正在上海交大读博，研究方向为语音多模态，语音交互系统，自监督预训练。他也是近期发布的语音全双工模型 LSLM、TTS 语音合成模型 F5-TTS 的作者之一。
仓库地址：
https://github.com/ddlBoJack/Awesome-Speech-Language-Model
Awesome-Speech-Language-Model
论文、代码与资源：语音语言模型和端到端语音对话系统。
通用语音、音频和音乐理解模型
Universal Speech, Audio and Music Understanding
模型
Model
- LTU: Listen, Think, and Understand - ICLR 2024
https://arxiv.org/abs/2305.10790
- SALMONN: Towards Generic Hearing Abilities for Large Language Models- ICLR 2024
https://arxiv.org/abs/2310.13289
- LTU-AS: Joint Audio and Speech Understanding - ASRU 2024
https://arxiv.org/abs/2309.14405
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models - arXiv 2023
https://arxiv.org/abs/2311.07919
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities - ICML 2024
https://arxiv.org/abs/2402.01831
- Qwen2-Audio Technical Report - arXiv 2024
https://arxiv.org/abs/2407.10759
- WavLLM: Towards Robust and Adaptive Speech Large Language Model - EMNLP 2024
https://arxiv.org/abs/2404.00656
- DiVA: Distilling an End-to-End Voice Assistant Without Instruction Training Data - arXiv 2024
https://arxiv.org/abs/2410.02678
基准
Benchmark
- Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech - ICASSP 2024
https://arxiv.org/abs/2309.09510
- AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension - ACL 2024
https://arxiv.org/abs/2402.07729
- SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond
Words - arXiv 2024
https://arxiv.org/abs/2406.13340
- AudioBench: A Universal Benchmark for Audio Large Language Models -
arXiv 2024
https://arxiv.org/abs/2406.16020
- SALMon: A Suite for Acoustic Language Model Evaluation - arXiv 2024
https://arxiv.org/abs/2409.07437
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark - arXiv 2024
https://www.arxiv.org/abs/2410.19168
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks -ICLR 2024 open review
https://openreview.net/forum?id=s7lzZpAW7T
端到端语音对话系统
End2End Speech Dialogue System
模型
Model
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities - EMNLP 2023
https://arxiv.org/abs/2305.11000
- GPT-4o Voice Mode -API 2024
https://openai.com/index/hello-gpt-4o/
https://www.arxiv.org/abs/2408.05211
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming - arXiv 2024
https://arxiv.org/abs/2408.16725
- LLaMA-Omni: Seamless Speech Interaction with Large Language Models -arXiv 2024
https://arxiv.org/abs/2409.06666
- Moshi: a speech-text foundation model for real-time dialogue - arXiv 2024
https://arxiv.org/abs/2410.00037
- Westlake-Omni - GitHub 2024
https://github.com/xinchen-ai/Westlake-Omni
- EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions - arXiv 2024
https://arxiv.org/abs/2409.18042
- IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities - arXiv 2024
https://arxiv.org/abs/2410.08035
https://github.com/MooreThreads/MooER
- GLM-4-Voice - GitHub 2024
https://github.com/THUDM/GLM-4-Voice
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM - arXiv 2024
https://arxiv.org/abs/2411.00774
https://github.com/Standard-Intelligence/hertz-dev
https://github.com/fishaudio/fish-speech
- Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities - arXiv 2024
https://arxiv.org/abs/2410.11190
基准
Benchmark
- VoiceBench: Benchmarking LLM-Based Voice Assistants - arXiv 2024
https://arxiv.org/abs/2410.17196
全双工建模
Full Duplex Modeling
- A Full-duplex Speech Dialogue Scheme Based On Large Language Models -NeurIPS 2024
https://arxiv.org/abs/2405.19487
- MiniCPM-duplex: Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models - EMNLP 2024
https://arxiv.org/abs/2406.15718
- LSLM: Language Model Can Listen While Speaking - arXiv 2024
https://arxiv.org/abs/2408.02622
- SyncLLM: Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents - arXiv 2024
https://arxiv.org/abs/2409.15594
- Enabling Real-Time Conversations with Minimal Training Costs - arXiv 2024
https://arxiv.org/abs/2409.11727
综述
Survey
- Towards audio language modeling -- an overview - arXiv 2024
https://arxiv.org/abs/2402.13236
- Recent Advances in Speech Language Models: A Survey - arXiv 2024
https://arxiv.org/abs/2410.03751
https://github.com/ga642381/speech-trident
- A Survey on Speech Large Language Models - arXiv 2024
https://arxiv.org/abs/2410.18908
编辑：林瑞丽，傅丰元
更多 Voice Agent 学习笔记：
从开发者工具转型 AI 呼叫中心，这家 Voice Agent 公司已服务 100+ 客户
WebRTC 创建者刚加入了 OpenAI，他是如何思考语音 AI 的未来？
人类级别语音 AI 路线图丨 Voice Agent 学习笔记
语音 AI 革命：未来，消费者更可能倾向于与 AI 沟通，而非人工客服
语音 AI 迎来爆发期，也仍然隐藏着被低估的机会丨 RTE2024 音频技术和 Voice AI 专场
下一代 AI 陪伴 | 平等关系、长久记忆与情境共享 | 播客《编码人声》
Voice-first，闭关做一款语音产品的思考｜社区来稿
↙↙↙阅读原文可查看相关链接，并与作者交流