Ubisoft logo Concordia University logo Mila logo Stellenbosch University logo

Factorized Self-Supervised Speech Tokenization

Paper: [arXiv]

Authors:

Benjamin Van Niekerk1,2,3
benjamin.l.vanniekerk@gmail.com
Jean-Philippe Letendre1
jean-philippe.letendre@ubisoft.com
Nicol Visser4
16986431@sun.ac.za
Hugo Seute1
hugo.seute@ubisoft.com
Herman Kamper4
kamperh@sun.ac.za
Mirco Ravanelli2,3
mirco.ravanelli@concordia.ca

1Ubisoft La Forge    2Concordia University    3Mila    4Stellenbosch University

Abstract:

Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker’s voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto’s convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling.

Text-to-Speech results on Seed-TTS

Below, we present some examples from the text-to-speech evaluation described in Section 3.3 of the paper.
We assess 12.5 Hz and 25 Hz versions of the tokenizers on the Seed-TTS test-en set.
We compare Facto (with codebook sizes of 8192 and 16384) against FocalCodec and Kanade.

25 Hz

Target Text Prompt Facto (codebook 8192)Facto (codebook 16384)FocalCodecKanade
She was one of several driven onto the beach.
Real schools, secondary schools giving a general practical education.
There was no replacement for armed forces lost in the fight.
The society of the day held that reward without labour was misguided.
Do unto others as you would have them do to you.
The forehead and throat are black; the back of the head is grey.
The work of the tailor is seen on each side.
He then visited Italy for a few months to regain his health.

12.5 Hz

Target Text Prompt Facto (codebook 8192)Facto (codebook 16384)FocalCodecKanade
She was one of several driven onto the beach.
Real schools, secondary schools giving a general practical education.
There was no replacement for armed forces lost in the fight.
The society of the day held that reward without labour was misguided.
Do unto others as you would have them do to you.
The forehead and throat are black; the back of the head is grey.
The work of the tailor is seen on each side.
He then visited Italy for a few months to regain his health.

Text-to-Speech Results on Expresso

Next, we test how well the tokenizers generalize beyond read speech.
We present supplementary examples on a mix of expressive and spontaneous data from Expresso (refer to Table 3 in the paper).

25 Hz

Target Text Style Prompt Facto (codebook 8192)Facto (codebook 16384)FocalCodecKanade
For the other parties, it is back to business as usual.Read - Happy
Had it not been for that squirrel, I'm pretty sure I would have went to Harvard.Read - Sad
Today in New York, it isn't gonna be hazy.Read - Whisper
Yes, today in Charleston there's a very low chance of heavy thunderstorms and rain.Read - Sad
This is the first time I've ever seen the Northern Lights.Improvised - Awe
Um I'm not gonna lie, you're scaring me a bit.Improvised - Sad+Sympathetic
Oh my awful. That is so gross.Improvised - Disgusted
That's awesome. Do you have all the ingredients you need, or should I run to the store just down the street?Improvised - Projected

12.5 Hz

Target Text Style Prompt Facto (codebook 8192)Facto (codebook 16384)FocalCodecKanade
For the other parties, it is back to business as usual.Read - Happy
Had it not been for that squirrel, I'm pretty sure I would have went to Harvard.Read - Sad
Today in New York, it isn't gonna be hazy.Read - Whisper
Yes, today in Charleston there's a very low chance of heavy thunderstorms and rain.Read - Sad
This is the first time I've ever seen the Northern Lights.Improvised - Awe
Um I'm not gonna lie, you're scaring me a bit.Improvised - Sad+Sympathetic
Oh my awful. That is so gross.Improvised - Disgusted
That's awesome. Do you have all the ingredients you need, or should I run to the store just down the street?Improvised - Projected

Examples including interjections, hesitations, and laughter

We also include a few examples showing how the tokenizers handle prompts or text containing interjections, hesitations, and laughter.

Target Text Type Prompt Facto (8192, 12.5 Hz)Facto (8192, 25 Hz)Facto (16384, 12.5 Hz)Facto (16384, 25 Hz)
Sure. Yeah. Um wow.Interjection
Ooh, ooh, Polly?Interjection
I uh I f I found out through browsing through social media, which is uhHesitation
Oh my God. I can't believe that I lost five thousand dollars.Laughing Mood
Mrs. Clements comes storming onto stage. What does she say?Laughing Mood

Voice conversion results on LibriSpeech

This section presents some examples from the voice conversion evaluation described in Section 3.4 of the paper.
We sample source lines from the LibriSpeech test-clean set and convert to the target speaker using just a few seconds of speech for reference.
We compare Facto to Kanade and CozyVoice2.

25 Hz

SourceTargetFacto (cont.)Facto (codebook 8192)Facto (codebook 16384)KanadeCosyVoice

12.5 Hz

SourceTargetFacto (cont.)Facto (codebook 8192)Facto (codebook 16384)KanadeCosyVoice