Main thread Other threads
MPI_Init thread spawned
| |
\|/ \|/
[ 1, a, II ] ----------|---[ 2, b, II ]
any comm | |-----|spawn | Main thread calls MPI_Thread_init
\|/ | \|/
[ 1, b, II ] | [ 1, b, I ]
MPI_Thread_init, | | | MPI_Thread_init
spawn thread \|/ \|/ \|/
[ 2, b, II ]
The states represent the combinations of the following:
1) Safe to call an MPI comm function
2) Not safe to call an MPI comm function
a) Safe to call MPI_Thread_init
b) Not safe to call MPI_Thread_init
I) Safe to spawn a thread that will call MPI
II) Not safe to spawn a thread that will call MPI
The transition events are:
Calling MPI_Thread_init
Spawning another thread that might use MPI
Calling an MPI communication function
Transitions not shown explicitly return to the same state.
I would summarize these states as follows:
[1, a, II] MPI_PRE_THREAD_INIT
[1, b, II] MPI_SINGLE_THREADED
[1, b, I] MPI_MULTI_THREADED
[2, b, II] MPI_NO_COMMUNICATION
As the diagram shows, MPI_NO_COMMUNICATION can be an error
state in some circumstances and a legal (but crippled) state
in other circumstances. In both situations, however, the
user has the same (limited) choice of things to do from a
thread in this state. Note that a thread can sometimes
leave this state, but never as a result of its own actions.
In a nonthreaded implementation, the main thread is always in
state MPI_SINGLE_THREADED, and any other threads are in
MPI_NO_COMMUNICATION. However, a nonthreaded implementation
probably doesn't want to worry about detecting this difference
between threads, so it can always tell all threads that they
are in MPI_SINGLE_THREADED state and let the user worry about
which one can communicate. A multithreaded implementation
can be in MPI_SINGLE_THREADED state if the main thread does
communication before calling MPI_Thread_init. Just as in the
nonthreaded case, other threads are then in MPI_NO_COMMUNICATION.
I don't know whether it's reasonable to expect an implementation
to detect when a thread is in this state. At any rate, I think
we can get by with just two functions:
MPI_Thread_init()
Initializes multithreading. Must be called from the
thread that called MPI_Init. Erroneous to call after
any MPI communication function. Legal but ineffective
to call from nonthreaded implementation.
MPI_Thread_get_state( state )
Returns one of the four states listed above; nonthreaded
implementations would always return MPI_SINGLE_THREADED.
And a library function could do this:
void library()
{
int state;
MPI_Thread_get_state( &state );
if( state == MPI_PRE_THREAD_INIT ) {
MPI_Thread_init();
MPI_Thread_get_state( &state );
}
if( state == MPI_MULTI_THREADED ) {
/* do multithreaded version of library */
} else if ( state == MPI_SINGLE_THREADED ) {
/* do single threaded version of library */
} else {
/* notify user that we can't do MPI from here */
}
}
Note that if the user calls the library from a secondary thread in
a single threaded implementation, this will fail. I don't see a
way to avoid that if single threaded implementations don't keep
track of which thread can call MPI.
What do you think?
John