Revert torch-family pin centralization #4351

Job	Run time
test-arm / test-backend-linux (arm_vgf_fp, operators) / linux-job	25m 16s
test-arm / test-backend-linux (arm_tosa_fp, operators) / linux-job	22m 25s
test-arm / test-backend-linux (arm_vgf_fp, models) / linux-job	37m 55s
test-arm / test-backend-linux (arm_tosa_fp, models) / linux-job	35m 9s
test-arm / test-backend-macos	-1s
test-arm / package-golden-artifacts	45s
	2h 1m 29s

Provide feedback