george.
On Nov 20, 2007, at 1:34 PM, Erez Haba wrote:
The 'const' keyword is a contract saying that the called function will not alter the buffer in any way. A copy of the buffer is required only if the function implementation need to change the buffer before sending.
I fail to see how Hubert's sequence of events implies the restrictions put in the standard.
This implementation can result in different values being sent to process B and process A (assuming sending to self is implemented as reading from the cache) which is incorrect.
The description below seems more like an application issue rather than MPI library issue. Such an application should flush the cache at the right time before calling MPI_Send, if it wants to maintain coherency.
Thanks, .Erez
-----Original Message-----
From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@xxxxxxxxxxxxx] On Behalf Of Steven Ericsson-Zenith
Sent: Tuesday, November 20, 2007 11:20 AM
To: mpi-21@xxxxxxxxxxxxx
Subject: Re: [mpi-21] MPI Forum suggestions 11/14
The second step here is a violation of any parallel construction - since it is a write. The question here did not allow for a concurrent write. It was rather: is there a set of state transitions in which a concurrent read would cause a cache flush and potentially apprehend a differing value. The answer appears to be no.
The coherency problem would indeed be solved if the API had specified a const buffer. The only problem I see with a use of const is that it may require an extra copy operation at the function interface - and this is undesirable. It's an old problem too (and one of the reasons I originally invented the Ease primitives).
With respect, Steven
-- Dr. Steven Ericsson-Zenith Institute for Advanced Science & Engineering http://iase.info http://senses.info
On Nov 19, 2007, at 11:40 PM, Hubert Ritzdorf wrote:
(*) Assuming MPI process A with an int buffer "buf" and "buf[0] = 0"
was
set by MPI Process A. The cache and main memory contains buf[0] = 0.
(*) Another MPI process or thread sets buf[0] = 1 after it was
triggered
to do this. buf[0] is now 1 within the memory.
But if MPI process A would perform now perform a = buf[0],
the value of a would be 0 (since the SX system is not cache coherent
and the cache still contains buf[0] = 0)
(*) Now MPI process A calls MPI_Send (buf, ....)
(*) If the buf[0] is still in the cache and the MPI_Send would perform
b = buf[0], b would still get the value 0.
(*) Now, the buffer "buf" is sent to the destination process.
When "sending" the data to another MPI process B in the same node,
the cache is bypassed and the correct value buf[0] = 1 is written
("sent") to MPI process B. In addition, the value of buf[0] is
flushed within the cache.
(*) If now MPI process A would perform c = buf[0] within MPI_Send,
c would get value 1.
When I correctly understand the meaning and definition of keyword const the usage of keyword const would not applicable for "*buf". When I remember correctly, also other vector systems were not cache coherent. As far as I know, the IBM cell processor is also not always cache coherent. When looking to the memory wall, future (scalar) systems may also not be cache coherent.
Hubert
Steven Ericsson-Zenith wrote:I think that there are two issues here.
1. What exactly is the behavior that relates to this cache coherency that would distinguish the use of const from the absence of const?
2. Why would the standard accommodate such a propriety behavior?
If the send buffer is read from memory and causes a cache flush that has not maintained coherency isn't this a problem in anycase? It appears to suggest that any value in the cache is irrelevant.
Perhaps you mean it the other way around: that is is the value in the cache that is the effective value of the send buffer, and that this value may be flushed if a read causes a cache update? This *would* potentially cause the effective value of the send buffer to change during the send.
With respect, Steven
On Nov 18, 2007, at 4:55 PM, Erez Haba wrote:
Can you elaborate or send a pointer related to that architecture?
The const keyword is a contract between the caller and the caller and the callee. It means in our example that the MPI_Send function will not change the user supplied buffer. Is this the case with the SX systems?
-----Original Message----- From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi- forum.org] On Behalf Of Hubert Ritzdorf Sent: Sunday, November 18, 2007 11:27 AM To: mpi-21@xxxxxxxxxxxxx Subject: Re: [mpi-21] MPI Forum suggestions 11/14
This is correct. The SX systems are not cache coherent. The send buffer is read directly from memory and may cause a cache flush. Thus, the const keyword would not be correct.
Hubert Ritzdorf NEC Laboratories Europe IT Research Division NEC Europe Ltd.
Snir, Marc wrote:If I remember correctly, NEC or some other Japanese vendor had a system with non-choherent shared memory were this added restriction was important. I guess that a read could cause a flush to be canceled.
On 11/17/07 1:45 AM, "Steven Ericsson-Zenith" <steven@xxxxxxxxxxxxx> wrote:
I think there is an intersect here with Erez's interest in const.
In a parallel language it should be possible to send a constant
value
and read it at the same time (formally that would be fine).
Indeed, it
is possible in Occam and Ease/Carnap. I don't know the semantics of
const in C functions, I assume it requires either a constant
parameter, a pass by value copy operation or some other guarantee
that
the const is not written to (even by concurrent processes).
The issue in MPI has to do with the termination properties, as I
recall; i.e., the validity of the send buffer is not deterministic
since the read is not synchronized with the send (so you can't
know in
an interleaving that a subsequent alteration - including
deallocation
- to the buffer value has not occurred). Futzing with the send
buffer
during a send operation seems to me to be an unforgivable rational.
It really is equivalent to the usage rules one finds in a process
oriented language (like CSP/Occam or Ease/Carnap), an asynchronous
operation is, in effect, a distinct process and the buffer (if it
is a
variable) ceases to be in scope in all other processes during the
period of concurrency. Indeed, such variables can only come back
into
scope if there is a cooperative barrier, they cannot come back into
scope from subordinate processes. Occam overcomes this by not
allowing subordinates, Ease/Carnap overcomes this by only allowing
constants (i.e., pass by value) to be passed to subordinates.
With respect, Steven -- Dr. Steven Ericsson-Zenith Institute for Advanced Science & Engineering http://iase.info http://senses.info
On Nov 16, 2007, at 3:25 PM, Erez Haba wrote:
Thanks for the pointer!!!
Can someone elaborate on the rational to prohibit read access to a
buffer being sent?
I think that it's more than a little loss of functionally. Consider a multi threaded application that wishes to send some data to other ranks and at the same time process this data locally? Also consider the scenario when rank x sends the same data to rank y and z concurrently? does the text below prohibit that?
Quote --------------
In a multi-threaded implementation of MPI, the system may de- schedule a thread that is blocked on a send or receive operation, and schedule another thread for execution in the same address space. In such a case it is the user's responsibility not to access or modify a communication buffer until the communication completes. Otherwise, the outcome of the computation is undefined.
[] Rationale.
We prohibit read accesses to a send buffer while it is being used,
even though the send operation is not supposed to alter the
content
of this buffer. This may seem more stringent than necessary, but
the
additional restriction causes little loss of functionality and
allows better performance on some systems --- consider the case
where data transfer is done by a DMA engine that is not cache-
coherent with the main processor. ( End of rationale.)
------------
-----Original Message----- From: Lisandro Dalcin [mailto:dalcinl@xxxxxxxxx] Sent: Friday, November 16, 2007 3:03 PM To: mpi-21@xxxxxxxxxxxxx Cc: Erez Haba Subject: Re: [mpi-21] MPI Forum suggestions 11/14
On 11/16/07, Erez Haba <erezh@xxxxxxxxxxxxx> wrote:
Such implementations would be really bad for muti-threaded
applications, or those using async send. Is there an MPI contract
that the application should not access the send buffer for read
while the send is not complete?
AFAIK, there is. Please, see:
http://www.mpi-forum.org/docs/mpi-11-html/node40.html
-----Original Message----- From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi- forum.org] On Behalf Of Darius Buntinas Sent: Friday, November 16, 2007 1:11 PM To: mpi-21@xxxxxxxxxxxxx Subject: Re: [mpi-21] MPI Forum suggestions 11/14
I believe the standard allows an implementation to modify the buffer in a send call, as long as it's changed back before it completes.
I heard (don't remember from who), that there was an implementation that would do endian conversion in place on the source buffer before sending, then change it back.
Changing the standard to require the buffer pointer to be const could theoretically disallow that implementation. So adding consts wouldn't be a minor change.
-d
On 11/16/2007 02:06 PM, David Gingold wrote:
On Nov 16, 2007, at 2:42 PM, Jeff Squyres wrote:
1. const: Sure, this seems like a good idea. But it *is* an
API
change, and I am in agreement with Bill Gropp that there should
be no
API changes for MPI 2.1 (except for bug fixes). Even if the
API
changes supposedly won't matter (because someone will find a
case
where it does matter). More specifically: this is a slippery
slope
for 2.1. Making one "minor" change will lead to more minor
changes
which will lead to ...
If an implementation declares in its mpi.h:
int MPI_Send(const void *, int, MPI_Datatype, int, int, MPI_Comm);
does this violate the existing standard?
I don't see how this would break code that calls MPI_Send(). C
libraries made a similar transition with the advent of ANSI C; I
believe
the const semantics are defined in a way that lets one do just
this.
The const declaration allows an important optimization when compiling the calling functions. But if an implementation is allowed to do this, then perhaps there is no need to change to the standard.
-dg
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Marc Snir Director, Illinois Informatics Initiative U Illinois at Urbana-Champaign LIS Bldg., Room 123 501 E Daniel St Champaign, IL 61820 (217) 244 6568
-- Dr. Steven Ericsson-Zenith Institute for Advanced Science & Engineering http://iase.info http://senses.info
Attachment:
smime.p7s
Description: S/MIME cryptographic signature