BLD: cleanup build dependencies #252

Job	Run time
Build wheels for win_amd64 on windows-latest	8m 16s
Build ubuntu-22.04 (aarch64) wheels	26m 23s
Build ubuntu-22.04 (x86_64) wheels	3m 4s
Build source distribution	1m 18s
Build macos-13 (x86_64) wheels	5m 40s
Build macos-14 (arm64) wheels	2m 42s
Show artifacts	4s
Publish to PyPI	0s
	47m 27s

Provide feedback