Re: I/O error handling proposal

Nick Maclaren (nmm1@cus.cam.ac.uk)
Tue, 04 Feb 1997 17:27:11 +0000

One of the things that puzzles me about the MPI error model is what on
earth you can do in an error handler! There seems to be no way to find
any useful information about the error, sort it out or anything else.
In this, it seems to follow the C and Unix fiascos (and I am afraid that
fiascos IS the right term). The two biggest problem with them are:

1) All errors are reduced to a single number, which is forbidden to
have any context-dependent information encoded in it. Perhaps the worst
disaster here is the IP stack, but other areas are nearly as bad. This
means that it is quite impossible to diagnose problems in the field, and
most people just fiddle until they reduce to a manageable level.

2) There is often the context of a single level of errors, which
includes everything from a format discrepancy to a hard disk failure.
Even worse, the C function clearerr clears the error state and cannot
fail! That is just plain stupid, but even the Unix (POSIX) level makes
no difference between recoverable and unrecoverable errors.

Now this is a problem for general use, but is a disaster for I/O. If
MPI is ever going to be used as a basis for writing industrial strength
applications (e.g. databases), it MUST allow good I/O error diagnosis
and recovery. So here is a proposal that I believe would help. It is
quite obviously a general mechanism, because it didn't seem worth
inserting artificial I/O dependencies.

MPI_ERROR_HANDLE (fh, code, severity, scope, message)
IN fh File handle (handle)
OUT code Error code associated with the handler state
OUT severity Severity of the error state
OUT scope Scope of the error state
OUT message Context-dependent messages
OUT length Length of messages returned

MPI_ERROR_COMM (comm, code, severity, scope, message)
IN comm Communicator
OUT code Error code associated with the communicator
OUT severity Severity of the error state
OUT scope Scope of the error state
OUT message Context-dependent messages
OUT length Length of messages returned

These functions can be called at any time that the relevant file handles
or communicators are defined. Note that the results need not be exactly
the same as seen by all processes in a communicator, though they must be
consistent (see later).

The error code is one of the error codes that could be passed to
MPI_ERROR_STRING, but need not be an error code that has been returned
to any previous call (let alone the last one). It is associated with
the file handle or communicator state and not necessarily with an
individual operation. If it is MPI_SUCCESS, the values of the remaining
arguments are undefined (but implementors should set them to their
lowest values).

The severity can be one of the following values:

MPI_ERR_IGNORABLE No special action is needed, and future calls
will work as if no error had occurred

MPI_ERR_RECOVERABLE Specific action is needed (e.g. an operation
needs retrying, or the position resetting)

MPI_ERR_RESTARTABLE All outstanding operations must be abandoned
and the object resynchronised

MPI_ERR_CORRUPTED This is beyond hope - no further operation
may be performed, not even closing it down

The scope can be one of the following values:

MPI_ERR_ACTION The failure affects only the operation that
returned an error code

MPI_ERR_LOCAL The failure affects only the local processor
and others may be unaware of it

MPI_ERR_GLOBAL The failure affects the all processes with
the file handle or the whole communicator

MPI_ERR_UNIVERSAL The failure is not localised (e.g. it is the
file itself that is corrupted)

Note that all processes with a file handle open or all processes in a
communicator will get the same return values for code, severity and
scope if scope is MPI_ERR_GLOBAL or MPI_ERR_UNIVERSAL, but that errors
detected on other processors are not required to be flagged until there
is a synchronisation operation. Under other circumstances, programmers
should not assume any consistency.

For example, a severity of MPI_ERR_RESTARTABLE and a scope of
MPI_ERR_LOCAL may imply that the current processor has lost track of what
the other processors are doing.

The message returned is neither a duplicate nor an expansion of that
returned by MPI_ERROR_STRING. It is additional, context-dependent,
possibly multi-line, information that may be useful to experts for
diagnosing problems. Implementors should attempt to make it useful to
local MPI support staff, but should feel free to include low-level data
in any convenient human-readable format, and need not make the message
consistent across processors. Programmers should use the result of
MPI_ERROR_STRING in primary diagnostics.

MPI_CLEAR_ERROR_HANDLE (fh, code, error)
IN fh File handle (handle)
IN code Error code associated with the handler state
OUT error Error code associated with this call

MPI_CLEAR_ERROR_COMM (comm, code, error)
IN comm Communicator
IN code Error code associated with the communicator
OUT error Error code associated with this call

code must be the value associated with the current file handle or
communicator state (i.e. returned from one of the information calls
defined above); its sole purpose is to reduce the likelihood of
confusion caused by asynchronous errors. While this could be done
securely, it is not worth the effort, because the only case where
the wrong error can be cleared is one with MPI_ERR_ACTION and the same
code as the previous state.

error will be set to MPI_SUCCESS if the error state was successfully
cleared and to MPI_ERROR_OP if not. The two most likely reasons for
failure are that the codes did not match, and that prerequisite
operations for cleaning up have not yet been completed. It will always
fail if the severity is MPI_ERR_CORRUPTED and always succeed if the
code is MPI_SUCCESS (and is supplied correctly).

The above does not solve all of the problems. For example, errors may
be recoverable on one system and not on another. But I believe that it
provides enough of a framework to write serious, reliable applications,
at least up to the level of a bank's database.

Nick Maclaren,
University of Cambridge Computer Laboratory,
New Museums Site, Pembroke Street, Cambridge CB2 3QG, England.
Email: nmm1@cam.ac.uk
Tel.: +44 1223 334761 Fax: +44 1223 334679