Issues with compare primitives

[glennsl]

While trying to figure out some unexpected behavior with NaNs, I've discovered the following:

1. Caml_obj uses Caml.int_compare to compare numbers, both ints and floats.
2. Caml.int_compare considers NaN > NaN and any other value. (which is of course fine if it's never used on NaNs, but it is)
3. Caml.float_compare considers NaN == NaN.
4. Caml.float_compare considers NaN less than any other value.
5. Caml.obj.lessthan and Caml_obj.greatherthan uses Caml_obj.compare, and therefore also considers NaN the greatest of all values! This is inconsistent with both OCaml and JavaScript behaviour. Comparing NaN to anything, in any way, should return false.

Simple repro:

// These will all print true

Js.log(compare(nan, nan) == 0) // Caml.float_compare
Js.log(compare(nan, 33.3) == -1) // Caml.float_compare
Js.log(compare(nan->Obj.magic, nan->Obj.magic) == 1) // Caml_obj.compare (which in turn uses Caml.int_compare on numbers)
Js.log(nan > nan == false) // inlined as >, correct behaviour
Js.log(nan->Obj.magic > nan->Obj.magic == true) // Canl_obj.greaterthan

[glennsl]

This is also inconsistent with the behavior exhibited if the comparisons are inlined, since then > and < are used instead of Caml_obj.less_than and Caml_obj.greaterthan.

See rescript-lang/rescript-core#90 (comment) for repro.

[cristianoc]

@glennsl would you write a few tests for the expected behaviour? Inline in this issue is fine.

[cristianoc]

Something perhaps not mentioned above: Caml_obj.compare and Caml.float_compare disagree on NaN.

Also, what should one do when sorting an array of floats. The sorting algorithm relies on consistency between equal and compare.

[cristianoc]

Looks like ocaml gives up the consistency of infix operators and compare. So one can do that here too.

So all the infix operators (generic or specialised to float) would return false. But compare, whether generic, or specialised to float, should return 0.

[glennsl]

Yep, from the OCaml manual:

[The] comparison predicates treat nan as different from any other float value, including itself; while compare treats nan as equal to itself and less than any other float value. This treatment of nan ensures that compare defines a total ordering relation.

[glennsl]

Something perhaps not mentioned above: Caml_obj.compare and Caml.float_compare disagree on NaN.

Yes. It's sort of implied by int_compare and float_compare being different, and compare using int_compare for numbers rather than float_compare.

[glennsl]

For a point of comparison in the JavaScript world, Array.sort also effectively will consider NaN equal to any other value, since any operation on NaN will result in NaN: https://tc39.es/ecma262/multipage/indexed-collections.html#sec-array.prototype.sort

[glennsl]

@glennsl would you write a few tests for the expected behaviour? Inline in this issue is fine.

Added to the top post

[cristianoc]

@glennsl take a look at #6042
Let me know if it solves all the corner cases you have observed.