Dave Wright
wrightdv@pwfl.com
Introduction
This proposal is based on my experience writing portable binary I/O
libraries. We support numerous Unix flavors, PC's, VAX's and IBM
mainframes. In our production environment, portability and
transparency of binary data files is both expected and required.
I am not overly concerned with the specifics of the Application
Programming Interface. Several approaches would work acceptably.
However, I am concerned with providing the necessary functionality to
make MPI-IO useful in a heterogeneous, production environment.
Possibilities
There are many ways to address file portability. Several options are
presented here for your consideration. This proposal recommends a
combination of methods to provide maximum flexibility and ease of
implementation.
1. Do nothing
This is bad because there is no built in support for file
portability. Case specific conversion programs would be required
where necessary. However, this option is actually preferable for
the environment where only one architecture of parallel
machine(s) is being used or during development. In those
situations, portability is not typically an issue.
2. Specify required binary format of file(s).
This is a viable option. However, it requires that we predefine
which portable binary formats must be supported by MPI-IO on
each platform. In addition, all reads/writes must be typed so
the proper conversions can be performed automatically by MPI-IO.
3. Install special marker in each file defining the binary format and
perform any required binary conversion during read.
This option is poor because it makes a major assumption about
the format of the data in the file. Some portion of the file,
perhaps only one byte, must be reserved for the magic cookie
that defines the binary format. This inhibits portability of the
file to other applications that do not use MPI-IO. In
non-generic implementations, this approach has advantages.
However, I do not think it is appropriate for MPI-IO.
4. Install special marker in a meta file defining the binary format and
perform any required binary conversion during read.
This option is especially poor because we can't ensure that the
meta file is always kept with the binary data file(s). If the
meta file is lost, there is no way to know the binary format of
the data in the MPI-IO files.
Recommendations
I recommend that we support both option 1 and 2. Option 1 essentially
means that the implementors do nothing to explicitly support MPI-IO
files that are portable to multiple machine architectures.
Option 2 requires explicit typing of data during read and write to
enable the implementation to perform the required data conversions.
We must agree on a limited, standard, subset of binary data formats
that MPI-IO would support. I recommend that we support both
big-endian IEEE (which is actually XDR) and little-endian IEEE
formats. This would support both the current generation workstations
and PC's. Each implementation of MPI-IO would have to support
conversion to and from big and little endian IEEE binary from the
native binary format. We could enable ambitious implementations to
support even more binary formats if they choose.
This proposal results in the following scenarios:
1. use native binary
2. use big-endian IEEE (XDR)
3. use little-endian IEEE
4. use special binary format supported by some implementation(s)
Application Programming Interface
The application programming interface would have to be modified to
support this proposal. The MPI_OPEN needs to have an additional
parameter to specify whether native binary, big-endian IEEE, or
little-endian IEEE. The info parameter could be used for
implementation specific binary format specification.
Each read and write call may need an additional parameter to
specify the type of the data being written. This would enable the
implementation to perform required binary conversions on the fly.
The current MPI-IO interface includes etype in the MPI_OPEN call
which may be sufficient for an implementation to determine what
binary format conversions are required. Keep in mind that the
conversion from native binary to IEE (for example), is different
for integers, reals, doubles, etc. For maximum flexibility, we may
have to include etype as an argument in each read/write routine.
----------------------------------------------------------------------
Dave Wright High Performance Computing/Computational Fluid Dynamics
voice:407-796-8477 fax:407-796-6049 internet: wrightdv@pwfl.com
Pratt & Whitney, PO Box 109600, M/S 712-63, West Palm Beach, FL 33410