1. The particular example of nonblocking MPI_All_gather used in the
chapter doesn't use MPI_Test or MPI_Wait on a generalized request, but
relies on MPI_Post_handler. To implement a nonblocking I/O function,
however, I would create a generalized request, start the operation,
and somewhere later on in the program, the user will call MPI_Test or
MPI_Wait.
2. A single poll_fn is not sufficient. We need two functions: test_fn
and wait_fn corresponding to test and wait.
For example, if I were to implement nonblocking I/O on Intel PFS, I
would use iodone() for test_fn and iowait() for wait_fn.
If the gen. request is for some operation that uses
communication, test_fn and wait_fn may well call MPI_Test or MPI_Wait
on some regular (nongeneralized) request for that communication.
If these two functions are not provided, MPI_Test and MPI_Wait will
not know how to test or wait.
3. Why force the user to always start the nonblocking operation by
defining a start_fn and then calling MPI_Start, which will internally
call start_fn? Instead, the user can start the nonblocking operation
himself. In cases where the user is actually doing the nonblocking
operation using persistent send/recv (such as the MPI_Allgather
example in the chapter), he would of course call MPI_Start on a
regular (non-generalized) request. But in other cases, e.g. I/O, he
can directly call say aiowrite.
MPI_Start need not be used for the MPI implementation to know that
the operation has been "started", because it doesn't need to know that
the operation has been started. When MPI_Test and MPI_Wait see
that the request is a generalized request, they treat it differently
and call test_fn or wait_fn.
4. Instead of two functions MPI_Meta_request_create and
MPI_Request_init, only the foll. one function is needed:
MPI_Gen_request_create(test_fn, wait_fn, cancel_fn, complete_fn,
extra_state, request)
where test_fn = function to be called by MPI_Test
wait_fn = function to be called by MPI_Wait
cancel_fn = function to be called by MPI_Cancel
complete_fn = a combination of the free_fn and complete_fn in
current proposal. It is called internally by
MPI_Wait after wait_fn returns, or by MPI_Test if
test_fn returns true, or by MPI_Request_free.
This function deallocates memory, and does
whatever other cleanup is necessary (user defined).
This function creates a new generalized request and associates those
callback functions with it. Note that there is no init_fn. I don't see
the need for a separate init_fn, because the user can do all the
initialization before calling MPI_Gen_request_create and pass it in
extra_state.
5. A corresponding MPI_Gen_request_free function is not needed, because
the request is automatically freed by MPI_Test, MPI_Wait or
MPI_Request_free. They, of course, call complete_fn for that purpose.
6. For reasons above, there is no need for MPI_Request_mark_complete
or MPI_Meta_request_free of the current proposal. If the user wants to
explicitly free a generalized request, he should call
MPI_Request_free.
In other words, only one function MPI_Gen_request_create with the four
callback functions test_fn, wait_fn, cancel_fn, complete_fn
is needed.
Using this, I would implement a nonblocking write, MPI_Iwrite, as
follows:
MPI_Iwrite(.....)
{
maybe some initialization and setting of "extra_state"
MPI_Gen_request_create(test_fn, wait_fn, cancel_fn, complete_fn,
extra_state, request)
start_the_operation
}
/* The defn. of the callback function would depend on what file system
I am using */
MPI_Wait would have to be implemented as follows:
MPI_Wait(request)
{
if (request is a generalized_request) {
call wait_fn
call complete_fn
do own cleanup for generalized requests
}
else {
do the usual
}
}
MPI_Test would have to be implemented as follows:
MPI_Test(request)
{
if (request is a generalized_request) {
call test_fn
if (true) {
call complete_fn for its cleanup
do own cleanup for generalized requests
}
else return false
}
else {
do the usual
}
}
The nonblocking MPI_All_gather example in the chapter could be
implemented as follows:
{
What is done in init_fn is done here.
/* Note that it sets the various fields of "extra_state" */
MPI_Gen_request_create(test_fn, wait_fn, cancel_fn, complete_fn,
extra_state, request)
What is done in start_fn is done here.
/* Note that it uses MPI_Start, because it needs to start a
persistent send on a regular request */
/* In handler_fn, use MPI_Request_free instead of
MPI_Request_mark_completed. */
/* test_fn and wait_fn could be set to null, because
this code doesn't use test and wait. */
}