I’ve read that it’s possible to extract multiple HuBERT layers and combine them with a weighted mix. It’s also commonly noted that different layers emphasize different information. For example:
- Layer 9 (used by RVC v1) tends to capture strong phonetic/phoneme-level content while still retaining some local prosody (F0/energy) and a bit of speaker dependence.
- Layer 12 (used by RVC v2) emphasizes more abstract, speaker-invariant linguistic content, with pitch/prosody and timbre cues largely suppressed.
Do you think this is worth exploring/adding?
|
def extract_features(model, suffix, feats, version, device="cpu"): |
|
with torch.no_grad(): |
|
if suffix == ".pt": |
|
logits = model.extract_features(**{"source": feats, "padding_mask": torch.BoolTensor(feats.shape).fill_(False).to(device), "output_layer": 9 if version == "v1" else 12}) |
|
feats = model.final_proj(logits[0]) if version == "v1" else logits[0] |
|
elif suffix == ".onnx": |
|
logits = model.run([model.get_outputs()[0].name, model.get_outputs()[1].name], {"feats": feats.detach().cpu().numpy()}) |
|
feats = torch.as_tensor(logits[int(version != "v1")], dtype=torch.float32, device=device) |
|
elif suffix == ".safetensors": |
|
logits = model(feats)["last_hidden_state"] |
|
feats = model.final_proj(logits[0]).unsqueeze(0) if version == "v1" else logits |
|
else: raise ValueError(translations["option_not_valid"]) |
|
|
|
return feats |
I’ve read that it’s possible to extract multiple HuBERT layers and combine them with a weighted mix. It’s also commonly noted that different layers emphasize different information. For example:
Do you think this is worth exploring/adding?
Vietnamese-RVC/main/library/utils.py
Lines 223 to 236 in 91f51ed