4.4. Communication Modes

PreviousUpNext
Up: Point-to-Point Communication Next: Semantics of Point-to-Point Communication Previous: Data Conversion

The send call described in Section Blocking Send is blocking: it does not return until the message data and envelope have been safely stored away so that the sender is free to modify the send buffer. The message might be copied directly into the matching receive buffer, or it might be copied into a temporary system buffer.

Message buffering decouples the send and receive operations. A blocking send can complete as soon as the message was buffered, even if no matching receive has been executed by the receiver. On the other hand, message buffering can be expensive, as it entails additional memory-to-memory copying, and it requires the allocation of memory for buffering. MPI offers the choice of several communication modes that allow one to control the choice of the communication protocol.

The send call described in Section Blocking Send uses the standard communication mode. In this mode, it is up to MPI to decide whether outgoing messages will be buffered. MPI may buffer outgoing messages. In such a case, the send call may complete before a matching receive is invoked. On the other hand, buffer space may be unavailable, or MPI may choose not to buffer outgoing messages, for performance reasons. In this case, the send call will not complete until a matching receive has been started, and the data has been moved to the receiver.

Thus, a standard mode send can be started whether or not a matching receive has been started. It may complete before a matching receive is started. The standard mode send is nonlocal: successful completion of the send operation may depend on the occurrence of a matching receive.


Rationale.

The reluctance of MPI to mandate whether standard sends are buffering or not stems from the desire to achieve portable programs. Since any system will run out of buffer resources as message sizes are increased, and some implementations may want to provide little buffering, MPI takes the position that correct (and therefore, portable) programs do not rely on system buffering in standard mode. Buffering may improve the performance of a correct program, but it doesn't affect the result of the program. If the user wishes to guarantee a certain amount of buffering, the user-provided buffer system of Section Buffer Allocation and Usage should be used, along with the buffered-mode send. ( End of rationale.)
There are three additional communication modes.

A buffered mode send operation can be started whether or not a matching receive has been started. It may complete before a matching receive is started. However, unlike the standard send, this operation is local, and its completion does not depend on the occurrence of a matching receive. Thus, if a send is executed and no matching receive is started, then MPI must buffer the outgoing message, so as to allow the send call to complete. An error will occur if there is insufficient buffer space. The amount of available buffer space is controlled by the user---see Section Buffer Allocation and Usage. Buffer allocation by the user may be required for the buffered mode to be effective.

A send that uses the synchronous mode can be started whether or not a matching receive was started. However, the send will complete successfully only if a matching receive is started, and the receive operation has started to receive the message sent by the synchronous send. Thus, the completion of a synchronous send not only indicates that the send buffer can be reused, but it also indicates that the receiver has reached a certain point in its execution, namely that it has started executing the matching receive. If both sends and receives are blocking operations then the use of the synchronous mode provides synchronous communication semantics: a communication does not complete at either end before both MPI processes rendezvous at the communication. A send executed in this mode is nonlocal.

A send that uses the ready communication mode may be started only if the matching receive is already started. Otherwise, the operation is erroneous and its outcome is undefined. On some systems, this allows the removal of a hand-shake protocol that is otherwise required and results in improved performance. The completion of the send operation does not depend on the status of a matching receive, and merely indicates that the send buffer can be reused. A send operation that uses the ready mode has the same semantics as a standard send operation, or a synchronous send operation; it is merely that the sender provides additional information to the system (namely that a matching receive is already started), that can save some overhead. In a correct program, therefore, a ready send could be replaced by a standard send with no effect on the behavior of the program other than performance.

Three additional send functions are provided for the three additional communication modes. The communication mode is indicated by a one letter prefix: B for buffered, S for synchronous, and R for ready.

MPI_BSEND(buf, count, datatype, dest, tag, comm)
IN bufinitial address of send buffer (choice)
IN countnumber of elements in send buffer (non-negative integer)
IN datatypedatatype of each send buffer element (handle)
IN destrank of destination (integer)
IN tagmessage tag (integer)
IN commcommunicator (handle)
C binding
int MPI_Bsend(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
int MPI_Bsend_c(const void *buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
Fortran 2008 binding
MPI_Bsend(buf, count, datatype, dest, tag, comm, ierror)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER, INTENT(IN) :: count, dest, tag
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_Bsend(buf, count, datatype, dest, tag, comm, ierror) !(_c)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER(KIND=MPI_COUNT_KIND), INTENT(IN) :: count
TYPE(MPI_Datatype), INTENT(IN) :: datatype
INTEGER, INTENT(IN) :: dest, tag
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
Fortran binding
MPI_BSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)

<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in buffered mode.

According to the definitions in Section MPI Procedures, MPI_BSEND is a completing procedure and the user can re-use all resources given as arguments, including the message data buffer. It is also a local procedure because it returns immediately without depending on the execution of any MPI procedure in any other MPI process.
Advice to users.

This is one of the exceptions in which a completing and therefore blocking operation-related procedure is local. ( End of advice to users.)

MPI_SSEND(buf, count, datatype, dest, tag, comm)
IN bufinitial address of send buffer (choice)
IN countnumber of elements in send buffer (non-negative integer)
IN datatypedatatype of each send buffer element (handle)
IN destrank of destination (integer)
IN tagmessage tag (integer)
IN commcommunicator (handle)
C binding
int MPI_Ssend(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
int MPI_Ssend_c(const void *buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
Fortran 2008 binding
MPI_Ssend(buf, count, datatype, dest, tag, comm, ierror)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER, INTENT(IN) :: count, dest, tag
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_Ssend(buf, count, datatype, dest, tag, comm, ierror) !(_c)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER(KIND=MPI_COUNT_KIND), INTENT(IN) :: count
TYPE(MPI_Datatype), INTENT(IN) :: datatype
INTEGER, INTENT(IN) :: dest, tag
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
Fortran binding
MPI_SSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)

<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in synchronous mode.

MPI_RSEND(buf, count, datatype, dest, tag, comm)
IN bufinitial address of send buffer (choice)
IN countnumber of elements in send buffer (non-negative integer)
IN datatypedatatype of each send buffer element (handle)
IN destrank of destination (integer)
IN tagmessage tag (integer)
IN commcommunicator (handle)
C binding
int MPI_Rsend(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
int MPI_Rsend_c(const void *buf, MPI_Count count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
Fortran 2008 binding
MPI_Rsend(buf, count, datatype, dest, tag, comm, ierror)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER, INTENT(IN) :: count, dest, tag
TYPE(MPI_Datatype), INTENT(IN) :: datatype
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
MPI_Rsend(buf, count, datatype, dest, tag, comm, ierror) !(_c)

TYPE(*), DIMENSION(..), INTENT(IN) :: buf
INTEGER(KIND=MPI_COUNT_KIND), INTENT(IN) :: count
TYPE(MPI_Datatype), INTENT(IN) :: datatype
INTEGER, INTENT(IN) :: dest, tag
TYPE(MPI_Comm), INTENT(IN) :: comm
INTEGER, OPTIONAL, INTENT(OUT) :: ierror
Fortran binding
MPI_RSEND(BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)

<type> BUF(*)
INTEGER COUNT, DATATYPE, DEST, TAG, COMM, IERROR

Send in ready mode.

There is only one receive operation, but it matches any of the send modes. The receive procedure described in the last section is blocking: it returns only after the receive buffer contains the newly received message. A receive can complete before the matching send has completed (of course, it can complete only after the matching send has started).

In a multithreaded implementation of MPI, the system may de-schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user's responsibility not to modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined.


Advice to implementors.

Since a synchronous send cannot complete before a matching receive is started, one will not normally buffer messages sent by such an operation.

It is recommended to choose buffering over blocking the sender, whenever possible, for standard sends. The programmer can signal a preference for blocking the sender until a matching receive occurs by using the synchronous send mode.

A possible communication protocol for the various communication modes is outlined below.

ready send:
The message is sent as soon as possible.

synchronous send:
The sender sends a request-to-send message. The receiver stores this request. When a matching receive is started, the receiver sends back a permission-to-send message, and the sender now sends the message.

standard send:
First protocol may be used for short messages, and second protocol for long messages.

buffered send:
The sender copies the message into a buffer and then sends it with a nonblocking send (using the same protocol as for standard send).

Additional control messages might be needed for flow control and error recovery. Of course, there are many other possible protocols.

Ready send can be implemented as a standard send. In this case there will be no performance advantage (or disadvantage) for the use of ready send.

A standard send can be implemented as a synchronous send. In such a case, no data buffering is needed. However, users may expect some buffering.

In a multithreaded environment, the execution of a blocking communication should block only the executing thread, allowing the thread scheduler to de-schedule this thread and schedule another thread for execution. ( End of advice to implementors.)


PreviousUpNext
Up: Point-to-Point Communication Next: Semantics of Point-to-Point Communication Previous: Data Conversion


Return to MPI-4.1 Standard Index
Return to MPI Forum Home Page

(Unofficial) MPI-4.1 of November 2, 2023
HTML Generated on November 19, 2023