The scenary:
The producer wants to produce data without synchronizing with
its consumer (e.g. a simulation program), and
the consumer sometimes wants to look at the produced data
(e.g. to visualize the current state of the application).
An other example is a flight simulator -- each workstation
simulates one aircraft and must look to the positions of
the aircrafts of the other workstations.
The problem:
The consumer can rewrite the data while the consumer is
reading it with MPI_GET.
One possible solution:
Producer:
#define WRAP_LNG 10000
volatile struct flg_area {
int flg_begin, flg_end;
} flags;
volatile struct .... {...} data, result;
MPI_RMA_init( (void *)flags, 2, MPI_INT, comm, newcomm_flags);
MPI_RMA_init( (void *)data, .., ......., comm, newcomm_data);
flags.flg_begin = 0;
flags.flg_end = 0;
while (...) /*mainloop*/
{
result = ... /* producing the data */
/* begin of the critical section */
/* flags.flg_begin++ */
flags.flg_begin = (flags.flg_begin+1) % WRAP_LNG;
/* data = result */
bcopy((void *)result, (void *) data, sizeof(data));
/* flags.flg_end++ */
flags.flg_end = (flags.flg_end+1) % WRAP_LNG;
/* end of the critical section */
}
Consumer:
#define WRAP_LNG 10000
struct flg_area {
int flg_begin, flg_end;
} flags;
struct .... {...} data;
int old_flg_begin;
MPI_RMA_init( MPI_BOTTOM, 0, MPI_INT, comm, newcomm_flags);
MPI_RMA_init( MPI_BOTTOM, 0, MPI_INT, comm, newcomm_data);
flags.flg_begin = 0;
while (...) /*mainloop*/
{
old_flg_begin = flags.flg_begin;
MPI_GET((void *)flags, 1, MPI_INT, producer_rank,
0, 2, MPI_INT, 1, newcomm_flags);
while (flags.flg_begin != old_flg_begin) {
/* new data are produced */
while (flags.flg_begin != flags.flg_end) {
/* but not completely stored by producer */
/* ... perhaps sleep for a few nanoseconds or
microseconds until the chance is better */
MPI_GET((void *)(&flags), ...);
}
/* now there is a chance that the data can be got */
MPI_GET((void *)data, ..., ..., producer_rank,
0, ..., ..., 1, newcomm_data);
old_flg_begin = flags.flg_begin;
MPI_GET((void *)(&flags), ...);
/* but if the producer has begun to rewrite the data
then the flags.flg_begin has changed */
}
/* consuming the data */
}
This solution requires that the bcopy in the critical section
in the producer needs a clearly shorter time than the
producing and consuming of the data.
Comment:
I used MPI_GET because the semantic of MPI_GET is like a
synchronous remote procedure call -- first send the address
to the target process and then get back the value.
Marc proposed to forbid such programs:
In "Minutes of MPI meeting March 6-8" page 3, Thursday:
"For example,
Process 0 Process 1
flag = 1 while(get(flag) == 0)
is not allowed. Note that in a case where the remote
memory is implemented either with message passing or with
a weak memory coherency, this restriction is needed by
implementations".
Questions to all (my opinions must not be good):
1) Is this example semantically correct?
2) Which are the needs about the semantic of MPI_GET?
a) Atomicity:
I think, it is not necessary, that flags.flg_begin and
flags.flg_end are fetched together atomically.
I think, in theory, each of them should be fetched atomically,
but in practice, it is enough that the lowest byte is fetched
atomically.
b) Cache coherence and volatile:
It is necessary that the three operations in the critical section
of the producer are done in the "memory" in the given sequence!
"memory" is that object, that is seen by MPI_GET.
This needs that the compiler of the producer stores the
information immediately (e.g. forged by "volatile" or a cache
flushing).
3) Are these semantical needs guaranteed by the text in section
4.9 Semantics of PUT and GET Operations?
I think not, because there is no sentence about the correlation
of "local writes to memory" and MPI_GET?
4) Is there any need for MPI_WINDOW_LOCK?
I think, MPI_WINDOW_LOCK was proposed to solve the scenary
above, but I think, it does not help and it is not necessary.
5) Is the syntax of the example correct?
6) Is this the single case where we should mention that in C
a volatile declaration is necessary?
7) Is it possible to implement the semantic above on the
different hardware models?
a) message passing system without global memory access:
a1) With efficient thread library:
I think MPI_PUT and MPI_GET can be implemented with
an additional thread in the target process.
a2) Without efficient thread library:
I think, with out-of-band-data and a signal handler
for out-of-band-data the functionality of MPI_PUT and
MPI_GET in the target process can be implemented.
b) global memory access, but without cache coherency
in writing to memory
I think, the semantics can be implemented, but messages
must be used, but perhaps only to tricker the store
from the cache to the memory.
c) global memory access with cache coherency
I think, MPI_GET and MPI_PUT can be implemented by
this global memory access.
d) Which case I have forgotten?
8) Does normally a high-quality implementation of the
semantices proposed in the Draft, March 4, page 53
also implement this additional functionality for
asynchronous producer/consumer applications?
I think in the cases of question 7)
a1) -- Yes
a2) -- Yes
b) -- I do not know, but perhaps there isn't
any machine of this type
c) -- Yes
d) ???
9) Do we want to say something additional about MPI_PUT?
If these questions can be answered mostly positive then we must
decide:
A) Should MPI-2 allow such asynchronous producer/consumer
applications?
B) Should the interface be as described above?
C) Which text must be added to "Semantics of PUT and GET Operations"
D) Should this example be added to the
subsection 4.3.3 "Examples" of 4.3 "PUT and GET Calls"?
Rolf
Rolf Rabenseifner (Computer Center )
Rechenzentrum Universitaet Stuttgart (University of Stuttgart)
Allmandring 30 Phone: ++49 711 6855530
D-70550 Stuttgart 80 FAX: ++49 711 6787626
Germany rabenseifner@rus.uni-stuttgart.de