Partial clothes like a skirt gets transferred from driving video to reference image in final output.

[essence25]

Fantastic work! I've been playing with SCAIL2 for the past two weeks and it is the best available!!

There is one issue I'm trying to solve or understand. I have a driving video where a dancer wearing a yellow tutu/skirt (flange material hanging down like a ballet dancer clothes). The tutu part of the skirt (lower part of the round material) is getting transferred to my reference photo dancer lower body/butt area which is naked. It suddenly appears mid video or sometimes randomly and soon after few movements it disappears/fades away. I only want to keep the background from the driving video and my own character just like in the reference image and the dance motion (pose).

My reference image already has a black background(removed previously with specific BG Remove node), Replace Mode options are OFF on both SAM3 colored mask node and WanSCAILtoVideo node. Everything is great except part of that yellow skirt keeps transferring to my character in the final video (fades in and then out). I tried to specify/exclude it in prompts, positive and negative, tried to lower/increase lora strength (lightx2v I2V 480p rank256/128/64 tried T2V different lora also)

I also used the video prompt in SAM3 to include skirt or not include where I only mention detect just "person" and still not able to get rid of this effect. I tried various Clip Vision models and the effect still happens.

Other less important issue:
In addition to this effect sometimes I get teleporting, where the subject form reference image gets teleported back to reference image initial position, or just randomly closer to the viewer. Fades out and fades in closer randomly. (especially on long videos)
I was able to fix this by resizing driving video or sometimes resizing the reference image playing with video/image sizes/crop.

The main issue is the clothes transfer from driving video to final video. How can I fix this? Any help would be appreciated.
Let me know which parts of the workflow/models are responsible for this.
Thank you!!

[fengjia-guo]

Hi! Glad to hear you're enjoying SCAIL-2! For the skirt issue, are you trying to do Motion Transfer or Character Replacement? You said that you want to keep the background of the driving video, but you toggled Replace Mode options to OFF in your workflow.

Regarding the issue of teleporting, SCAIL-2 tends to refer more to the initial position of reference image than the driving video (e.g. if the reference is half-portrait, while the driving video is full-portrait, the output is more likely half-portrait). Aligning the reference frame to the driving video may helps. Also, if your workflow uses context-window instead of loop sampling, and the driving video has dramatic changes in scale, this problem may be more severe because each segment tends to start with the framing of the reference frame.

[essence25]

I'm trying to do a motion transfer and bring my own character into the driving video's background. So my character replaces driving video character and poses without taking any features or clothes from the driving video original character. I'm keeping the original background of the driving video, by providing a reference photo/character with a black background in my reference image. That automatically places my character from the reference image into the background of the driving video. If I enable replacement mode then my character from the reference image gets blended/morphed with the character from the driving video which I do not want. This might behave differently when using different nodes, but body and face can take form of the driving video character even when not in replacement mode, does happen to a small degree, an unwanted blending.

The interesting part is if I wanted an actual black background in my final video, I would have to strip the background from the driving video with SAM3 track2mask. It seems when SCAIL2 detects a black background in the reference image automatically preserves the character and brings it into the background from the driving video which is great. In scail-1 for example that would have not happened since it does not use SAM3, and I would just get motion transfer into a black background.

"Also, if your workflow uses context-window instead of loop sampling, and the driving video has dramatic changes in scale, this problem may be more severe because each segment tends to start with the framing of the reference frame."

Exactly what I was suspecting!! I noticed the teleportation happens exactly at the end of 6 seconds (chunk) and beginning of 7 second mark or end of 12 beginning of 13 etc. 6 second chunks from context ?

And yes I'm aligning reference image with video as much as possible, but the character in the video does move in and out (further away and back to front fast)

Here are some screenshots from my workflow. This is the best combo I was able to do after many hours of trial and error. Extracting 43 seconds at 30fps then interpolate 2x for 60fps final. All that in 1:12hr which is impressive in my opinion after trying various combinations of samplers. This setup that does not change color on long videos and does not over-bake(wet/plastic skin) like other workflows/combinations in long videos. I'm doing 43 second long videos in one go, that is my preferred work.
What is your recommendation for long videos and keep colors from shifting and skin changes/overbaking towards the end. That color correcting node does not really work.

Here is my workflow if you see any settings I need to play with, let me know: ( I would love to show you my video samples, if I can send you a private link, let me know where or can I use the email in your profile here?)

[fengjia-guo]

Thanks for sharing the details of your workflow!

Now I understand your purpose is replacement but you're doing it in animation mode with a reference image with a black background. We've also received feedback from the community saying that this also works and in many situations the results are better than replacement mode (we attribute this to the incompatibility of LightX2V loras with our model, because our replacement mode follow neither i2v nor t2v pattern). And they observed exactly the same phenomenon as you've described:

The interesting part is if I wanted an actual black background in my final video, I would have to strip the background from the driving video with SAM3 track2mask. It seems when SCAIL2 detects a black background in the reference image automatically preserves the character and brings it into the background from the driving video which is great. In scail-1 for example that would have not happened since it does not use SAM3, and I would just get motion transfer into a black background.

But there's one thing that can be improved in your workflow - tests from community (and my personal experiences) show that a LightX2V T2V lora for Wan 2.1, instead of an I2V lora, is more beneficial in replacement (no matter in replacement mode, or in your current settings of running replacement with replacement mode OFF).

Regarding context window and long video generation, yes, loop sampling leads to color shifting and other kinds of degradation, and context window may suffer from teleportation. We're still working with the community to alleviate the issues of both sampling techniques.

So my primary recommendation is setting the LightX2V lora to a T2V one. Feel free to send the workflow's json file and your input to my email on my profile.