GR only have a nonblocking (nonpersistent) form. A template for a GR
is made with MPI_GR_INIT(start_fn, push_fn, complete_fn, cancel_fn,
MPI_Gr *gr_req). This is like the current MPI_META_REQUEST_CREATE but
the extra_state is removed and it returns a request that is a new type
of handle instead of an MPI_Request. Since the gr_req is never used
in MPI_START, MPI_WAIT/TEST, etc. this should make error detection a
little easier. The gr_req is never used directly in an active GR so
it is always "inactive" and you can start it as many times as you want.
A GR is started with
MPI_GR_BEGIN(MPI_Gr gr_req, extra_state, MPI_Request *req).
This starts the GR and returns a standard MPI request. req is active
when MPI_GR_BEGIN returns and causes start_fn to be called. push_fn can go
on later in the "background". Note that extra_state is given here so
you can have unique values for each instantiation of the GR.
MPI_MARK_REQUEST_COMPLETE(MPI_Request req) internally marks the GR as
done. The only type of req allowed in this call is one from an
MPI_GR_BEGIN (i.e., can't do a regular MPI_Request). The req is still
active when this is done.
MPI_WAIT/TEST(req, status) will return true/complete on the GR if
MPI_MARK_REQUEST_COMPLETE has been called. This causes the
complete_fn to be called with the status provided. complete_fn will
finish before wait/test returns. On return, req is deallocated and
the handle is null by setting it to MPI_REQUEST_NULL. By definition
the req is now "inactive".
MPI_GR_FREE(MPI_Gr *gr_req) marks gr_req for deallocation and makes the
handle null by setting it to MPI_GR_NULL. In the MPI way, you can do
this even if there are GR what are outstanding that were created from
this template.
As with pt-to-pt you can do a MPI_CANCEL(req) on a req that is a GR.
As with pt-to-pt you still need to do a test/wait on req. Advice to
users is that since complete_fn gets called at the test/wait, you need
to see if req has been cancelled if that matters.
You can mark a GR for deallocation and cause the handle to become null
(MPI_NULL_REQUEST) by calling MPI_REQUEST_FREE(req). The req may
still be active. It will become inactive after
MPI_MARK_REQUEST_COMPLETE finishes. Advice to impl: The complete_fn
can and must be called after MPI_MARK_REQUEST_COMPLETE even if the req
has been freed. The status supplied to complete_fn is MPI_STATUS_NULL
in this case.
Interaction with handlers: If a handler and the complete_fn are both
elegible to execute (for example, the GR finishes while a wait is
active on the GR and a handler is attached to the GR) then MPI makes
no statement about the order in which they will execute. If someone
thinks we need to specify order and has a proposal I am open to this.
Geneal advice: If callbacks in GR (or handlers) change the state of MPI
handles (such as freeing a request), care must be taken that the "main
thread" of code will not generate errors due to this change.
Clarification to MPI-1: To help avoid errors from asynchronous
changes, MPI_CANCEL will accept the null handle MPI_REQUEST_NULL.
When this happens, MPI_CANCEL will simply return without generating an
error.