Add type check for rb_big_modulo first argument

Add type check for rb_big_modulo first argument #6816