What's the function of celebahq_crop when preprocess frames?

[Mark-Dou]

Hello, could you explain what's the function of celebahq_crop when preprocess frames is? I'm not very clear about that.
Thank U!

[jhaotingc]

It's function is to detect face/faces in an image and then crop the face out. Since the faces in are not aligned in raw videos (or videos in FF++), we have to preprocess the frames by aligning the faces then crop them out.

The code is modified from chail/patch-forensics.

[Mark-Dou]

I have another question: after the celebahq_crop which align the faces in raw videos, however the background of the image have been changed, how to eliminate this consequence where the background of the manipulated images are different from the pristine images?

[jhaotingc]

Sorry for the late response.

I did not quite get the question. The background of the manipulated image is different from the pristine image that the manipulated face belongs to. The background of the manipulated image belongs to the pristine image of the cropped-out face. The two backgrounds are pristine, the manipulated part is the cropped-out face and the pasted face.

In the paper, they don't consider the consequence of the backgrounds, since only the border of the manipulated faces is inconsistent. The pasted face is inconsistent with the other part of the face. Hence, we can compute the consistency volume between the background and the face.

When we align the faces, we rotate, resize the image so that it match the faces at the same position in every images. So the consistency between the face and the background in a pristine image seems to have minor consequences only.

If we don't align the face, the model would have to find the face first in an image. It involves facial recognition, face detection as well as deepfake detection in the same task. Seems to be too hard to achieve.