testing: Reduce output precision of geomath and trig, collapse variants (#2126)

Analyzing the geomath and trig testsuite entires, each of them had
multiple reference outputs due to small precision differences between
platforms and optimization levels. By printing only 4 digits of
precision (and in one case of an unstable value to pass to tan, just
picking a different value), we are able to collapse those tests to one
reference output each.

only flushes near-zero values, not precision. With the reduced precision every
platform/backend produces identical text, so the four variant references are
removed and a single ref/out.txt remains. The JIT and C++
(OSL_DEBUG_OUTPUT_CPP=3) backends now produce byte-identical output and both
pass against it.

Assisted-by: Claude Code / Claude Opus 4.8

Signed-off-by: Larry Gritz <lg@larrygritz.com>

Author: lgritz