fix bug for videomt model device mismatch#45204

ydshieh merged 5 commits intohuggingface:mainhuggingface/transformers:mainfrom

kaixuanliu:videomt-xpukaixuanliu/transformers:videomt-xpuCopy head branch name to clipboard