The notion that Bill Gropp outlined can be further extended, so as to create
an architecture independent format for data files, without resorting to an
asci encoding of values.
The current defition of MPI_PACK/MPI_UNPACK assumes that these calls have a
communicator argument that indicate which are the processes that may possibly
receive the data after it was packed (or where does the dat come from).
Suppose that we extend the definition of these two functions so as to allow a
NULL Communicator argument, to indicate that the packed data can be exchanged
across arbitrary architectures. One possible implementation is that whenever
data is packed with a NULL communicator, then one either use a machine
independent format, such as XDR; whenever data is unpacked with a NULL
communicator, one translates from XDR. Another possible implementation is
that one attaches to the data a header that describes the source architecture
when the data is packed, and uses the header to do conversion, if needed, when
the data is unpacked. In this second scenario there is little overhead when
data is read back on the same machine it was written from.
Now, data packed with a NULL communicator can be stored on a file, next
retrieved by any other machine and unpacked, and data conversion will occur to
take care of differences between source and target architecture. By layering
the MPI-IO read and write calls on top of MPI_UNPACK and MPI_PACK,
respectively, one can support an architecture independent storage system, that
can be used in a heterogenous environment: Data written on one machine using
MPI_IO write calls can be read by any other machine, using MPI_IO read calls.
One can further elaborate on this scheme, and pack an MPI datatype, together
with the data, so as to obtain self-describing files/objects, with an
architecture independent description.
Marc Snir
IBM T.J. Watson Research Center
P.O. Box 218, Yorktown Heights, NY 10598
email: snir@watson.ibm.com
phone: 914-945-3204
fax: 914-945-4425
URL: http://www.research.ibm.com/people/s/snir