홈 › Papers

ESPnet-SpeechLM: An Open Speech Language Model Toolkit

2025-02-21 · Jinchuan Tian, Jiatong Shi, William Chen, Siddhant Arora, Yoshiki Masuyama, Takashi Maekaku, Yihan Wu, Junyi Peng, Shikhar Bharadwaj, Yiwen Zhao, Samuele Cornell, Yifan Peng, Xiang Yue, Chao-Han Huck Yang, Graham Neubig, Shinji Watanabe

We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks. The toolkit and its recipes are fully transparent and reproducible at: https://github.com/espnet/espnet/tree/speechlm.

📄 PDF Abstract BibTeX arXiv:2502.15218

Code (1)

espnet/espnet 공식 구현 pytorch

Tasks

Language ModelingLanguage Modelling

ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit

2023-04-10 · Brian Yan, Jiatong Shi, Yun Tang, Hirofumi Inaguma 외

ESPnet-ST-v2 is a revamp of the open-source ESPnet-ST toolkit necessitated by the broadening interests of the spoken language translation community. ESPnet-ST-v2 supports 1) offline speech-to-text translation (ST), 2) si…

Benchmarking Simultaneous Speech-to-Text Translation Speech-to-Speech Translation Speech-to-Text+2

ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet

2021-11-29 · Siddhant Arora, Siddharth Dalmia, Pavel Denisov, Xuankai Chang 외

As Automatic Speech Processing (ASR) systems are getting better, there is an increasing interest of using the ASR output to do downstream Natural Language Processing (NLP) tasks. However, there are few open source toolki…

Spoken Language Understanding text-to-speech Text to Speech

ESPnet: End-to-End Speech Processing Toolkit

2018-03-30 · Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi 외

This paper introduces a new open source platform for end-to-end speech processing named ESPnet. ESPnet mainly focuses on end-to-end automatic speech recognition (ASR), and adopts widely-used dynamic neural network toolki…

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition

ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

2019-10-24 · Tomoki Hayashi, Ryuichi Yamamoto, Katsuki Inoue, Takenori Yoshimura 외

This paper introduces a new end-to-end text-to-speech (E2E-TTS) toolkit named ESPnet-TTS, which is an extension of the open-source speech processing toolkit ESPnet. The toolkit supports state-of-the-art E2E-TTS models, i…

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition+2

EURO: ESPnet Unsupervised ASR Open-source Toolkit

2022-11-30 · Dongji Gao, Jiatong Shi, Shun-Po Chuang, Leibny Paola Garcia 외

This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR). EURO adopts the state-of-the-art UASR learning method i…