moshi

# Moshi: a speech-text foundation model for real time dialogue

![precommit badge](https://github.com/kyutai-labs/moshi/workflows/precommit/badge.svg)
![rust ci badge](https://github.com/kyutai-labs/moshi/workflows/Rust%20CI/badge.svg)

[[Read the paper]][moshi] [[Demo]](https://moshi.chat) [[Hugging Face]](https://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd)

[Moshi][moshi] is a speech-text foundation model and **full-duplex** spoken dialogue framework.
It uses [Mimi][moshi], a state-of-the-art streaming neural audio codec.
[Talk to Moshi](https://moshi.chat) now in our live demo.

## Organisation of the repository

There are three separate versions of the Moshi inference stack in this repo.


- **[PyTorch](#pytorch-implementation): for research and tinkering.** The code is in the [`moshi/`](moshi/) directory.
- **[MLX](#mlx-implementation-for-local-inference-on-macos): for on-device inference on iPhone and Mac.** The code is in the [`moshi_mlx/`](moshi_mlx/) directory.
- **[Rust](#rust-implementation): for production.** The code is in the [`rust/`](rust/) directory.
    This contains in particular a Mimi implementation in Rust, with Python bindings available
    as `rustymimi`.

Finally, the code for the web UI client used in the [Moshi demo](https://moshi.chat) is provided in the [`client/`](client/) directory.

If you want to fine tune Moshi, head out to [kyutai-labs/moshi-finetune](https://github.com/kyutai-labs/moshi-finetune).

### Other Kyutai models

The Moshi codebase is also used to run related models from Kyutai that use a multi-stream architecture similar to Moshi:
- **Hibiki: simultaneous speech translation.** Check out the [Hibiki repo](https://github.com/kyutai-labs/hibiki) for more info.
- **Kyutai Text-To-Speech and Speech-To-Text.** Check out the [Delayed Streams Modeling repo](https://github.com/kyutai-labs/delayed-streams-modeling) for more info.

## Model architecture

Moshi models **two streams of audio**: one corresponds to Moshi speaking, and the other one to the user speaking.
Along with these two audio streams, Moshi predicts text tokens corresponding to its own speech, its **inner monologue**,
which greatly improves the quality of its generation.
A small Depth Transformer models inter-codebook dependencies for a given time step,
while a large, 7B-parameter Temporal Transformer models the temporal dependencies. Moshi achieves a theoretical latency
of 160ms (80ms for the frame size of Mimi + 80ms of acoustic delay), with a practical overall latency as low as 200ms on an L4 GPU.

<p align="center">
<img src="./moshi.png" alt="Schema representing the structure of Moshi. Moshi models two streams of audio:
    one corresponds to Moshi, and the other one to the user. At inference, the audio stream of the user is taken from the audio input, and the audio stream for Moshi is sampled from the model's output. Along that, Moshi predicts text tokens corresponding to its own speech for improved accur