Do We Still Need OFI CXI (FI_DELIVERY_COMPLETE and FI_ATOMIC_READ) Work Arounds?

[markbrown314]

In transport_ofi.h we have the following workaround:

static inline
void shmem_transport_atomic_fetch(shmem_transport_ctx_t* ctx, void *target,
                                  const void *source, size_t len, int pe,
                                  int datatype)
{
#ifdef ENABLE_MR_ENDPOINT
    /* CXI provider currently does not support fetch atomics with FI_DELIVERY_COMPLETE
     * That is why non-blocking API is used which uses FI_INJECT. FI_ATOMIC_READ is
     * also not supported currently */
    long long dummy = 0;
    shmem_transport_fetch_atomic_nbi(ctx, (void *) source, (const void *) &dummy,
                                     target, len, pe, FI_SUM, datatype);
#else
    shmem_transport_fetch_atomic_nbi(ctx, (void *) source, (const void *) NULL,
                                     target, len, pe, FI_ATOMIC_READ, datatype);
#endif
}

Is this still necessary? Needs testing.

[markbrown314]

If we deem FI_DELIVERY_COMPLETE is now supported and we don't call shmem_transport_fetch_atomic_nbi anymore, we need to check that none of the callers of shmem_transport_atomic_fetch in OFI transport actually want blocking behavior. see #1192