15.4.5. Split Collective Data Access Routines

MPI provides a restricted form of ``nonblocking collective'' I/O operations for all data accesses using split collective data access routines. These routines are referred to as ``split'' collective routines, because a single collective operation is split in two: a begin routine and an end routine. The begin routine begins the operation, much like a nonblocking data access (e.g., MPI_FILE_IREAD). The end routine completes the operation, much like the matching test or wait (e.g., MPI_WAIT). As with nonblocking data access operations, the user must not use the buffer passed to a begin routine while the routine is outstanding; the operation must be completed with an end routine before it is safe to free buffers, etc.

Split collective data access operations on a file handle fh are subject to the semantic rules given below.

• On any MPI process, each file handle may have at most one active split collective operation at any time.

• Begin calls are collective over the group of processes that participated in the collective open and follow the ordering rules for collective calls.

• End calls are collective over the group of processes that participated in the collective open and follow the ordering rules for collective calls. Each end call matches the preceding begin call for the same collective operation. When an ``end'' call is made, exactly one unmatched ``begin'' call for the same operation must precede it.

• An implementation is free to implement any split collective data access routine using the corresponding blocking collective routine when either the begin call (e.g., MPI_FILE_READ_ALL_BEGIN) or the end call (e.g., MPI_FILE_READ_ALL_END) is issued. The begin and end calls are provided to allow the user and MPI implementation to optimize the collective operation.

  According to the definitions in Section MPI Procedures, the begin procedures are incomplete. They are also nonlocal procedures because they may or may not return before they are called in all MPI processes of the process group.
  Advice to users.

  This is one of the exceptions in which incomplete procedures are nonlocal and therefore blocking. ( End of advice to users.)
  

• Split collective operations do not match the corresponding regular collective operation. For example, in a single collective read operation, an MPI_FILE_READ_ALL on one process does not match an MPI_FILE_READ_ALL_BEGIN/ MPI_FILE_READ_ALL_END pair on another process.

• Split collective routines must specify a buffer in both the begin and end routines. By specifying the buffer that receives data in the end routine, we can avoid the problems described in Problems with Code Movement and Register Optimization, Section Problems with Code Movement and Register Optimization, but not all of the problems, such as those described in Sections Problems Due to Data Copying and Sequence Association with Subscript Triplets, Problems Due to Data Copying and Sequence Association with Vector Subscripts, and Optimization Problems, an Overview.

• No collective I/O operations are permitted on a file handle concurrently with a split collective access on that file handle (i.e., between the begin and end of the access). That is, the following example is erroneous.
  Example Erroneous example fragment of concurrent split collective access on a file handle:

  

• In a multithreaded implementation, any split collective begin and end operation called by a process must be called from the same thread. This restriction is made to simplify the implementation in the multithreaded case. (Note that we have already disallowed having two threads begin a split collective operation on the same file handle since only one split collective operation can be active on a file handle at any time.)

For the purpose of consistency semantics (Section File Consistency), a matched pair of split collective data access operations (e.g., MPI_FILE_READ_ALL_BEGIN and MPI_FILE_READ_ALL_END) compose a single data access.