Flow Control with Collective Ops.

Lewins, Lloyd J (llewins@msmail4.hac.com)
15 Dec 1995 16:07:48 -0800

The MPI Standard supports flow control for point to point sends using the
standard mode. However, the standard doesn't mention flow control for
collective operations. For example, consider a master/slave style program, in
which the master repeatedly broadcasts data to the slaves. Should an MPI
implementation automatically throttle the master so that it matches the
consumption rate of the slaves? If so, a portable implementation of MPI_BCAST
should use MPI_SEND exclusively, since this is the only operation which is
guaranteed to include the necessary flow control. In particular, an
implementation of MPI_BCAST should not use MPI_ISEND, unless it also includes
explicit flow control. Since the amount of free resources is not available, it
is hard to determine how loose one can make this flow control without reaching
resource limits.

As we consider inter-communicator collective operations, this becomes a more
important issue. This is because, one (stated) use of inter-communicators is
to support the data-flow style of programming. This style naturally involves
groups of processors in a pipeline. This pipeline should be loosedly coupled
to maximize operlap between stages, and minimize pipeline stalls. A programmer
will naturally wish the implementation of collective operations between
pipeline stages to provide flow control, while minimizing coupling.

Note: the programmer may also need/wish to be able to specify the amount of
buffering between pipeline stages. If collective operations are implemented
using the standard send there may be no buffering between stages (if the
standard mode doesn't buffer). This leads to pipelines which are too tightly
coupled, leading to excessive stalling, or lack of overlap between stages.

Lloyd Lewins
Hughes Aircraft Co.
llewins@msmail4.hac.com