Arm backend: Remove setup flag for model test deps #5487

Job	Run time
generate-matrix / generate	6s
pytorch/executorch / build-manywheel-py3_10-cpu	24m 54s
pytorch/executorch / upload / upload-manywheel-py3_10-cpu	25s
	25m 25s

Provide feedback