large-count reductions

[jeffhammond]

Motivation

Because MPI reductions require user-defined reductions with user-defined datatypes (unlike RMA), it is difficult albeit not impossible to write large-count reductions using user-defined datatypes. However, because user-defined reductions provide no support for MPI_IN_PLACE, this is unsolvable except in a pipelined implementation.

MPI_Reduce_scatter_x is unsolvable in MPI-3, short of using MPI_Reduce and MPI_Alltoallw. The latter solution is inefficient, as discussed in #2.

Function Definitions

int MPI_Reduce_x(const void *sendbuf, void *recvbuf, MPI_Count count, 
                 MPI_Datatype datatype,MPI_Op op, int root, MPI_Comm comm);

int MPI_Allreduce_x(const void *sendbuf, void *recvbuf, MPI_Count count,
                    MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

int MPI_Reduce_scatter_block_x(const void *sendbuf, void *recvbuf, MPI_Count recvcount, 
                               MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

int MPI_Reduce_scatter_x(const void *sendbuf, void *recvbuf, const MPI_Count recvcounts[],
                         MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

int MPI_Ireduce_x(const void *sendbuf, void *recvbuf, MPI_Count count, 
                  MPI_Datatype datatype,MPI_Op op, int root, MPI_Comm comm);

int MPI_Iallreduce_x(const void *sendbuf, void *recvbuf, MPI_Count count,
                     MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

int MPI_Ireduce_scatter_block_x(const void *sendbuf, void *recvbuf, MPI_Count recvcount, 
                                MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

int MPI_Ireduce_scatter_x(const void *sendbuf, void *recvbuf, const MPI_Count recvcounts[],
                          MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);

In order to support user-defined reductions, a new function signature for the reduction operator. While we are changing it anyways, we add a second datatype argument, to address the issues brought up in Trac ticket #339.

int MPI_Op_create_x(MPI_User_function_x* user_fn, int commute, MPI_Op* op);

void MPI_User_function_x(void* inbuf, MPI_Datatype *intype,
                         void* inoutbuf, MPI_Datatype *inouttype, MPI_Count *len);

Implementation

BigMPI already implements reduce, allreduce and reduce_scatter_block (see reductions_x.c). The reduce_scatter functions will be implemented soon.