How to introduce more emotive tags?

[jaskaransingh1-ai]

Thanks for making this model opensource.
I have been experimenting with the model a lot and wanted to introduce more inline emotional tags like [Sneeze] [Excited] [Whisper] mid sentences to change the prosody of the speech mid-sentence as in Orpheus-TTS.

I have started by creating a synthetic dataset (tagged) -20% (12 speakers, 1000 tagged sentences, 1000 non-tagged, across 4 tags) and natural data - non-tagged- 80% (8000 sentences).
Full fine-tuning for 5 epochs. First epoch checkpoint was usable, after that became overfit.
The tags are not reliable, but previous tags are preserved and working.
The new tags are ignored, no affect at all.

Can you please share a guide on how the tags were introduced in the models or how can we go about it?

[a710128]

Hi, thanks for your interest and for sharing your experiments!

I’m not fully sure about the exact setup on your side, but in principle, full fine-tuning should be able to achieve the effect you want if the tagged data is consistent and sufficient.

During synthesis/training, the tags are inserted directly into the text in the form of [tag_name], placed inline at the desired position in the sentence. For example, tags like [Whisper] or [Excited] can be written directly in the middle of the text to indicate where the prosody should change.

That said, making newly introduced tags reliable may require enough paired data for each tag and careful balancing, especially for mid-sentence prosody changes.

[jaskaransingh1-ai]

@a710128 Thanks for your response.
Could you suggest an approximate amount of paired data per tag it would require? I was referring to the Orpheus-TTS insights on fine-tuning with 8 speakers and 300 lines each.

Maybe the amount of [laughing] tags that were in the training dataset of Voxcpm2 can give an approximate.