Unwanted subtitles for east Asian languages

[andypotato]

When generating videos where characters talk in east Asian languages like Chinese, Japanese or Korean, there is a 99% chance that the resulting video has unwanted (and completely garbled) subtitles. The problem also appears when using Chinese audio to create lip synced videos. It does not matter if a start image is used or not. The subtitles will appear.

There are a number of suggested fixes out there, for example using NAG, using different samplers, but none of these methods work.

This appears to be a dataset issue with a large amount of videos with burn-in subtitles being inside the training set. If possible this should be reviewed and cleaned up. I would rate this a major issue as currently this otherwise fantastic model is unusable for anything with east Asian speech.

[Deepsagar-uniqcotech]

The issue is most likely coming from the training data, not from samplers or generation settings. A large amount of Chinese, Japanese, and Korean videos on the internet already contain hardcoded subtitles, karaoke captions, or streaming overlays. Because of this, the model learns that East Asian speech usually appears together with subtitles. During video generation, it tries to recreate that pattern, which results in random and broken subtitle text appearing in the output.

This is why changing samplers, CFG values, or using NAG does not properly solve the problem. Those settings only change the way frames are generated, but they do not remove the learned connection between East Asian speech and subtitle overlays.

The correct fix is to clean the training dataset and reduce subtitle-heavy videos during training. Videos with burned-in captions should either be removed completely or cleaned before training. OCR tools such as PaddleOCR, CRAFT, or DBNet can be used to detect subtitle regions automatically. Once detected, those videos can be filtered out or the subtitle area can be inpainted before the clips are added to the dataset.

Another important improvement is adding subtitle detection during training. The model can be trained with a simple “subtitle present / subtitle absent” label. This gives the model a direct understanding that speech does not always require visible captions.

An OCR penalty can also help. During training, generated frames can be scanned for text. If unwanted text is detected, the model receives a penalty loss. Over time this teaches the model to avoid generating fake subtitles and random characters.

For lip sync models, the issue becomes stronger because the audio and subtitle visuals are being learned together. Chinese speech audio is repeatedly paired with subtitle overlays in the dataset, so the model starts treating subtitles as part of the expected output. Using cleaner speech datasets without captions is important to break this connection.

A practical short-term solution is:

filter subtitle-heavy videos from the dataset
fine-tune using clean talking-head videos
add OCR-based text detection during training
reject generated frames that contain detected subtitles

This should significantly reduce the problem without harming overall video quality.

So overall, this is not really a sampler issue. It is mainly a dataset contamination and training alignment problem caused by too many East Asian videos containing burned-in subtitles.