[core] fix group offloading when using torchao (#13276)

[core] fix group offloading when using torchao (#13276) #5459