[core] fix group offloading when using torchao

[core] fix group offloading when using torchao #37436