In a previous message, I claimed that 3rd-party communication could be handled
in the PUT/ACCEPT proposal by allowing the user to implement an agent and
performing 2-party communications to the agent. This message contains two
parts
(1) Illustrates how an agent might be built with current constructs in
the PUT/ACCEPT proposal, to examine the overheads involved, etc.
(2) Suggests a minor extension to the PUT/ACCEPT proposal, which I had
suggested before in a more general form but rejected, to allow the
construction of more efficient agents.
Building an agent using the existing PUT/ACCEPT proposal
===============================================
Perhaps the most important aspect of the agent is that it block and do
nothing most of the time -- i.e. unless a process is attempting to communicate
with it. PROBE works well for this purpose. (Earlier, I proposed using a new
RMA_PROBE. It now appears that the standard PROBE will suffice, by extending
the STATUS argument returned from it to include an "MPI_OP" field which denotes
whether a GET, PUT, GETHOLD, PUTHOLD, or SEND is in the queue. It may not
be necessary to delineate GETHOLD from GET and PUTHOLD from PUT.)
So, a "personal agent" might look like the following. The tag is used to
determine the general region ("window") to which communication will occur:
10 continue
CALL MPI_PROBE(MPI_ANY_SOURCE, MPI_ANY_TAG, COMM, STATUS, IERR)
C Determine window based on tag
ITAG = STATUS(MPI_TAG)
IF (ITAG .GT. MAXTAG) ...error... bad tag
BASE = BASES(ITAG)
SIZE = SIZES(ITAG)
C Perform ACCEPT or OFFER based on type of operation probed
if (STATUS(MPI_OP) .EQ. MPI_GET_PROBED .OR.
& STATUS(MPI_OP) .EQ. MPI_GETHOLD_PROBED) THEN
CALL ACCEPT(BASE, SIZE, DISP_UNIT, ITAG, COMM, 1)
else if (STATUS(MPI_OP) .EQ. MPI_PUT_PROBED .OR.
& STATUS(MPI_OP) .EQ. MPI_PUT_PROBED) THEN
CALL OFFER (BASE, SIZE, DISP_UNIT, ITAG, COMM, 1)
else
...error... somebody sent a message to agent
end if
goto 10
Any GET or PUT operations posted to the above agent, optionally preceded by
one or more GETHOLD or PUTHOLD operations, will be serviced. However, the
transfer on a shared-memory system will probably not take place until after
the first operation in the originator, since the agent will not execute the
matching operation until it notices the operation from the originator. For
example, in the following sequence on an originator:
PUTHOLD data to agent
PUTHOLD data to agent
PUT data to agent
ACCEPT(0)
the first PUTHOLD might be delayed until the second PUTHOLD is performed.
In the sequence
PUT
ACCEPT(0)
the PUT might be delayed until the ACCEPT is performed. The act of delaying
the first PUTHOLDs or PUTs will probably incur overhead that would not be
present if the operation could be performed immediately. One way around this
might be for the originator to always start each sequence with an empty
PUTHOLD or GETHOLD. This *might* wake up the agent in time for the ACCEPT
operation to be executing by the time the next PUT or PUTHOLD operation
executed, but it would probably be cleaner (or as clean) for the originator
to just send a message to the agent to tell it what kind of operation to
expect next. In any case, there is extra overhead involved.
This extra overhead could be avoided if the agent always had an ACCEPT and
OFFER executing, waiting for any PUTs or GETs that might occur. Since both
an OFFER and an ACCEPT cannot execute over the same window at the same time,
all circumstances cannot be covered by this approach, but there may still be
a significant number of cases where it can be used. For example, if a
program consists of iterations where, in each iteration, old values are
read and new values are created in a similar (but separate) array, double
buffering can be used, with an ACCEPT running on the new buffer and OFFER
running on the old.
Reviving a Rejected Construct
=============================
There is no intrinsic reason that agents need to be so inefficient or hard to
construct. All that is really required of the agent is that the agent block
until PUT or GET operations are performed by external clients, and that it
then service them.
In the PUT/ACCEPT proposal discussion, I mentioned the idea of a combined
operation (which I called AVAIL) which would act like either an OFFER or
ACCEPT, depending upon what was needed. I rejected it based on the idea that,
if AVAIL allowed both PUTs and GETs during the same operation (or "cycle" or
"epoch"), some semantic problems could result -- e.g. users could be led to
believe that 3rd-party communications were supported. However, if an AVAIL
operation was guaranteed to match *either* a PUT *or* GET, but not both, these
problems would disappear, and the coding of an agent would be much simpler and
more efficient. I suggest that an AVAIL always have an implicit count of
one -- i.e. that it match with exactly one PUT or GET, along with any
corresponding PUTHOLD and GETHOLD operations before that PUT or GET.
The above agent now becomes:
C Start up IAVAILs on five windows, each identified by a different tag
CALL IAVAIL(BASE(1), SIZE(1), DISP_UNIT(1), ITAG(1), COMM, IAVREQS(1))
CALL IAVAIL(BASE(2), SIZE(2), DISP_UNIT(2), ITAG(2), COMM, IAVREQS(2))
CALL IAVAIL(BASE(3), SIZE(3), DISP_UNIT(3), ITAG(3), COMM, IAVREQS(3))
CALL IAVAIL(BASE(4), SIZE(4), DISP_UNIT(4), ITAG(4), COMM, IAVREQS(4))
CALL IAVAIL(BASE(5), SIZE(5), DISP_UNIT(5), ITAG(5), COMM, IAVREQS(5))
10 continue
C Wait for something to happen
CALL WAITANY(5, IAVREQS, INDEX, STATUS)
C Re-initialize that "window"
CALL IAVAIL(BASE(INDEX), SIZE(INDEX), DISP_UNIT(INDEX), ITAG(INDEX),
& COMM, IAVREQS(INDEX))
goto 10
Communicating with such an agent should be very efficient on most systems.
With such an agent running, clients on shared-memory systems may be able to
"move bits" to and from the agent's memory immediately upon encountering any
PUT, GET, PUTHOLD, or GETHOLD. Programmers may find the above approach useful
in application programs other than just agents.
In real life, the above agent would probably be extended slightly so that it
can receive control messages -- e.g. to tell the agent that it is time to die.
This can be done just by adding an IRECV to the list of IAVAILs.
I am hoping that this will help to put aside some of the complaints that might
arise from the fact that the PUT/ACCEPT proposal does not directly handle
3rd-party communications.
I am not suggesting that matching collective operations, AVAILC and IAVAILC,
should be proposed, unless someone can describe a situation where they might
be needed.
-Dave
===============================================================================
David C. DiNucci | MRJ, Inc., Rsrch Scntst |USMail: NASA Ames Rsrch Ctr
dinucci@nas.nasa.gov| NAS (Num. Aerospace Sim.)| M/S T27A-2
(415)604-4430 | Parallel Tools Group | Moffett Field, CA 94035