[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Generalized Request Progress

In my example:

The first send of process 0 must complete. This, because it is matched by a receive on process 1 and this receive is not matched to another send.
Similarly, the second send of process o must complete.

The receive of process 1 does not complete -- this because the receive complete operation does not occur.

The receive of process 2 must complete.

I think that the standard is pretty unambiguous and not contradictory: "complete" is applied separately to the send operation and to the receive operation. In some cases, the standard requires that a send completes, even though the matching receive may never complete; and vice-versa, with ISend. (I also happen to remember the "original intent" of the guy that wrote the paragraph on progress.)

I agree that (a) this definition may slow down MPI on some systems (although the damage can be minimized: the odd cases where it is not sufficient to quick the progress engine only when MPI calls occur are very rare, so that the progress engine can be throttled down). I am also aware that it takes some work to implement an efficient progress engine. These are the two reasons we did not push the case originally, and allowed implementors to ignore the issue -- we basically agreed not to clarify the implications of the standard; but the standard is pretty unambiguous. By now, any smart NIC should provide enough hooks to be able to have a correct progress engine.

As for the comment of Rolf, in my example, each process calls MPI_Finalize before it exits.

Marc Snir
Interim Director, Illinois Informatics Initiative
University of Illinois at Urbana Champaign
Library and Information Systems Building, MC-493
501 E. Daniel St, Room 123
Champaign, IL 61820-6211
Tel: (217) 244 6568 Fax: (217) 244 3302
www.cs.uiuc.edu/homes/snir





On Sep 4, 2007, at 10:56 AM, Stephen Siegel wrote:


On Sep 4, 2007, at 5:06 AM, Hubert Ritzdorf wrote:

Hi,

Marc Snir wrote:
The problem occurs with nonblocking send-receives as well. IMHO, one cannot implement MPI without a progress engine, either threads or interrupts or offload engine. Of course, most implementations are noncompliant on that account.

To be more specific, my understanding of the MPI standard is that in the following example:

MPI_Init(...);
...
MPI_Comm_dup(MPI_COMM_WORLD, &comm);
MPI_Comm_rank(MPI_Comm_World, &myrank);
switch(myrank) {
case 0: {
foo(); /* do some long computation
MPI_Send(a, large_number, MPI_REAL, 1, tag, MPI_COMM_WORLD);
MPI_Send(b, 1, MPI_REAL, 2, tag, comm);
}
case 1: {
MPI_Irecv(a, large_number, MPI_REAL, MPI_ANY_SOURCE, tag, MPI_COMM_WORLD);
while(1) ;
}
case 2 : {
MPI_Recv(b, 1, MPI_REAL, 0, tag, comm);
MPI_Abort(MPI_COMM_WORLD, 666);
}
MPI_Finalize();

The program should abort, and should not be stuck in an infinite loop: the receive on process 1 should make progress even though no MPI call occurs on that process after the receive. Therefore, the first send on process 0 should complete, the second send should start, and the receive on process 2 should complete.
I didn't attend the MPI standardization committee but I don't agree to this view.
The MPI 1.1 standard says in Chapter 3.7 "Nonblocking communication".

Similarly, a nonblocking receive start call initiates the receive operation, but doesn't complete it.
The call will return before a message is stored into the receive buffer.
A separate receive complete (!!) call is needed to complete the receive operation and verify that the data
has been received into the receive buffer.

Thus, the 1st blocking send may wait until the "separate receive complete call" is issued which is never executed since
MPI process 1 hangs for ever in the while operation and message number 2 is never sent.


But the same Standard also says (in Section 3.5, p.31, paragraph labeled
"Progress"):

"If a pair of matching send and receives have been initiated
on two processes, then at least one of those two operations will complete,
independently of other actions in the system: the send operation will
complete, unless the receive is satisfied by another message, and completes;
the receive operation will complete unless the message sent is consumed by
another matching receive that was posted at the same destination process."

That is pretty unambiguous and supports Marc Snir's interpretation.
However, it does seem to contradict the statement "...a separate receive
complete call is needed to complete the receive operation..." I have
always felt this language was ambiguous and maybe self-contradictory.
The essence of the ambiguity, to my mind, is that the word "complete"
is overloaded in the MPI standard. It is never really defined but seems
to be used in two different ways. As an adjective, it is sometimes used
to mean that the data has been completely copied out of send buffer (for
a send request) or into receive buffer (for recv request), and sometimes
to mean that AND that the actions guaranteed to have taken
place when MPI_Wait returns (such as deallocation of the request
object) have taken place. I would use "complete" for the first notion
and maybe "finalized" for the second. So a request can be complete,
but not finalized. MPI_Wait will block until the request has completed,
and then finalize the request. A separate receive finalization call
is needed to finalize the receive request operation. Etc. This would
resolve the ambiguity (I think).

-Steve




In this view, there is no difference between generalized requests and send or receive requests; it is just that it is much easier to run into problems with progress of generalized requests. The standard does need any fixing, but many implementations do.
Thus, the behaviour of the MPI implementations which wait for the "separate receive complete call"
(or other MPI communication calls such as MPI_Iprobe or MPI_Test)
in order to complete the 1st send operation correspond to the MPI standard and don't need a fix.

Hubert Ritzdorf
IT Research Labs
NEC Europe



-------------------------------------------------------------------
Stephen F. Siegel, Assistant Professor

  address: Department of Computer & Information Sciences
           103 Smith Hall
           University of Delaware
           Newark DE 19716
   office: 432 Smith Hall
      web: http://www.cis.udel.edu/~siegel
    email: siegel@xxxxxxxxxxxx
      tel: +1 302 831 0083