fix an issue for real inputs of `irfft` #425

Job	Run time
build_windows (3.9)	18m 12s
build_windows (3.10)	17m 1s
build_windows (3.11)	21m 49s
build_windows (3.12)	21m 47s
build_linux (3.9)	2m 24s
build_linux (3.10)	2m 32s
build_linux (3.11)	2m 6s
build_linux (3.12)	2m 35s
test_linux (3.9, false, ubuntu-latest)	1m 29s
test_linux (3.10, false, ubuntu-latest)	1m 48s
test_linux (3.11, false, ubuntu-latest)	1m 52s
test_linux (3.12, false, ubuntu-latest)	1m 56s
test_windows (3.9, false, windows-2019)	4m 27s
test_windows (3.10, false, windows-2019)	3m 32s
test_windows (3.11, false, windows-2019)	3m 22s
test_windows (3.12, false, windows-2019)	3m 35s
	1h 50m 27s

Provide feedback