Re: we should wait for 1sided implementations

Lance Shuler (shuler@cs.sandia.gov)
Wed, 22 May 1996 16:18:33 -0600 (MDT)

>
> :-) :-) :-) *** (:- (:- (:-
>
> Counters do not require a handler at the target, but will add significant
> overhead for **short** put/get transfers on shared memory systems. If the
> system does not have an atomic fetch&add operation, then the caller will need
> to execute lock_counter; read_counter; increment_counter; write_counter;
> unlock_counter. This is the reason I presented this counter feature in a
> separate section, asking for a separate vote: it makes sense in the
> context of distributed memory implementations, but makes less sense
> for shared memory systems, and adds some complexity to the user.
> Remember, however, that no overhead is incurred when the increment is
> zero, i.e., when the feature is not used (except the overhead of
> reading an extra argument and testing for zero).
>
> Marc Snir

Under the Puma operating system (for distributed memory
architectures) here at Sandia, we support the 0 increment
and the 1 increment with minimal overhead. Since the
kernel is handling the increment, it is fast and can easily
be done on a per message basis. It also requires no
additional "field" in the message header indicating how
much to increment by. (Here, "message header" refers to
the actual message header that goes across the backplane.
It is a resource limited by size and structure.)

In order to support an arbitrary counter increment, we would
have to 1) reserve 4 bytes in the message header (a limited
resource) strictly for MPI-2 semantics, or 2) run a handler
to do the increment. Neither of these is an attractive option.

In addition, I'm not sure I like the idea of having N/2
MPI_Puts completed, while the counter only gets incremented
after N puts. This provides a performance improvement for
handler based incrementing, at the cost of a counter that
correctly indicates the number of puts completed at any
given point in time. In this scenario, the window owner
could be stalled from doing work by a sender that waits too
long to increment the window counter.

The arbitrary increment supports protocols that want to do
a sequence of MPI_Put's followed by a final MPI_Put which
does an increment that indicates "I'm done". This could
easily be achieved within the protocol itself by doing a
final MPI_Put to a known location that indicates "I'm
done". This solution would actually be faster than a
handler based increment. If there are other scenario's
where this type of solution will not work, please post
them. Currently, I haven't heard a good motivating reason
for supporting arbitrary counter increments.

These are just a couple reasons why SNL would just assume
see the counter increment options be only 0 or 1.

Lance

===========================================================================
Lance Shuler W: (505) 844-2159
Parallel Computing Sciences Department, 9223 Fax: (505) 845-7442
Sandia National Labs
Albuquerque, NM 87185-1109 shuler@cs.sandia.gov
===========================================================================