RE: [mpi-21] MPI Forum suggestions 11/14

["Bronis R. de Supinski" <bronis@xxxxxxxx>]

Jim:

Re:
> > If one cannot read from the send buffer while an MPI send is in
> > progress, then it seems to me that all Java and C# MPI bindings are
> > in violation of the MPI standard. The garbage collector can come
> > through and scan memory at any time.
>
> I don't think so...
>
> The buffer which is being sent cannot reasonably contain any references
> which the GC would need to see (it makes no sense to send them,

When has that ever stopped a code from doing something? Also I
could envision a good reason although at this point all of the
ones I have thought of are contrived.

> and there is no MPI data type which would let you express that the buffer contains them).

MPI_BYTE?

> If you were running the Boehm GC in a C program, then there might be an
> issue, since it has to scan everything and find pointers without any
> type help (though it won't run until you invoke it, so you could control
> this),

I think that is the key. The garbage collector has to be
controllable; ideally, the MPI implementation could set
a flag that says "Don't garbage collect now." Personally,
I see this as a short coming in the design of the garbage
collector/GCed language and not the MPI standard...

Bronis

> but in the languages which are designed to run with GC the
> runtime knows where the references it needs can be, and won't be
> looking elsewhere (at arrays of floats, for instance). So it's never
> going to look inside the buffers you're sending.



> -- Jim
>
> James Cownie <james.h.cownie@xxxxxxxxx>
> SSG/DPD/PAT
> Tel: +44 117 9071438
>
>
>
>
> > -----Original Message-----
> > From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@xxxxxxxxxxxxx] On
> > Behalf Of Doug Gregor
> > Sent: 04 December 2007 14:37
> > To: mpi-21@xxxxxxxxxxxxx
> > Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> >
> > If one cannot read from the send buffer while an MPI send is in
> > progress, then it seems to me that all Java and C# MPI bindings are
> > in violation of the MPI standard. The garbage collector can come
> > through and scan memory at any time.
> >
> > 	- Doug
> >
> > On Nov 30, 2007, at 2:20 AM, Erez Haba wrote:
> >
> > > Do you favor implementers over MPI users?
> > >
> > > The programming model is very important for users. This specific
> > > restrictions puts users knowingly (unlikely) or unknowingly (very
> > > likely) in direct violation of the standard.  We might choose not
> > > to change it but this will continue to surprise application
> > > developers. I expect to find many existing applications violating
> > > this MPI restriction unknowingly.
> > >
> > > That issue is delicate, that's true. I would like to hear about one
> > > implementation and/or application that take advantage of this
> > > restriction and why.
> > >
> > > Thanks,
> > > .Erez
> > >
> > > -----Original Message-----
> > > From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi-
> > > forum.org] On Behalf Of Rusty Lusk
> > > Sent: Thursday, November 29, 2007 7:25 PM
> > > To: mpi-21@xxxxxxxxxxxxx
> > > Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> > >
> > > Hi Steven,
> > >
> > > An explicit goal of the MPI Forum was to constrain implementations as
> > > little as possible.   This is just an example.  I think it was a good
> > > goal, and encouraged implementations.  If we break existing
> > > implementations by invalidating them, we will expose ourselves to
> > > being ignored.   This is not to say that it must be one way or the
> > > other, only that the issue is a delicate one, and that the
> > > conservative approach (not changing the existing bindings) has merit.
> > >
> > > Rusty
> > >
> > > On Nov 29, 2007, at 5:54 PM, Steven Ericsson-Zenith wrote:
> > >
> > >>
> > >> Of course Rusty you know that this is an implementation detail
> > >> invading the programming model. Is this pragmatic of early
> > >> implementations still required?
> > >>
> > >> With respect,
> > >> Steven
> > >>
> > >> --
> > >> Dr. Steven Ericsson-Zenith
> > >> Institute for Advanced Science & Engineering
> > >> http://iase.info
> > >> http://senses.info
> > >>
> > >>
> > >>
> > >> On Nov 29, 2007, at 2:18 PM, Rusty Lusk wrote:
> > >>
> > >>> Unless I misunderstand the issue you are discussing, the original
> > >>> intent of the MPI Forum did not have to do with cache issues at
> > >>> all, but deliberately intended to make the buffer available to the
> > >>> MPI implementation for it to use as it saw fit.  One illustrative
> > >>> example is that at least one implementation, and maybe more, used
> > >>> the buffer to do in-place byte swapping before sending the buffer
> > >>> to another process running on a machine with a different byte
> > >>> order.  If you envision a set of sends going to different machines
> > >>> with different byte orders, you can see why each send buffer must
> > >>> be left alone by the application until the Isend completes.
> > >>>
> > >>> On Nov 29, 2007, at 4:10 PM, Hubert Ritzdorf wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> Erez Haba wrote:
> > >>>>> The 'const' keyword is a contract saying that the called
> > >>>>> function will not alter the buffer in any way. A copy of the
> > >>>>> buffer is required only if the function implementation need to
> > >>>>> change the buffer before sending.
> > >>>>>
> > >>>>> I fail to see how Hubert's sequence of events implies the
> > >>>>> restrictions put in the standard.
> > >>>>> This implementation can result in different values being sent to
> > >>>>> process B and process A (assuming sending to self is implemented
> > >>>>> as reading from the cache) which is incorrect.
> > >>>>>
> > >>>> This would be no problem since the cache is also bypassed when
> > >>>> sending data to self.
> > >>>>> The description below seems more like an application issue
> > >>>>> rather than MPI library issue. Such an application should flush
> > >>>>> the cache at the right time before calling MPI_Send, if it wants
> > >>>>> to maintain coherency.
> > >>>>>
> > >>>> I agree.
> > >>>>
> > >>>> Hubert
> > >>>>> Thanks,
> > >>>>> .Erez
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi-
> > >>>>> forum.org] On Behalf Of Steven Ericsson-Zenith
> > >>>>> Sent: Tuesday, November 20, 2007 11:20 AM
> > >>>>> To: mpi-21@xxxxxxxxxxxxx
> > >>>>> Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> > >>>>>
> > >>>>>
> > >>>>> The second step here is a violation of any parallel construction -
> > >>>>> since it is a write. The question here did not allow for a
> > >>>>> concurrent
> > >>>>> write. It was rather: is there a set of state transitions in
> > >>>>> which a
> > >>>>> concurrent read would cause a cache flush and potentially
> > >>>>> apprehend a
> > >>>>> differing value. The answer appears to be no.
> > >>>>>
> > >>>>> The coherency problem would indeed be solved if the API had
> > >>>>> specified
> > >>>>> a const buffer. The only problem I see with a use of const is
> > >>>>> that it
> > >>>>> may require an extra copy operation at the function interface -
> > >>>>> and
> > >>>>> this is undesirable. It's an old problem too (and one of the
> > >>>>> reasons I
> > >>>>> originally invented the Ease primitives).
> > >>>>>
> > >>>>> With respect,
> > >>>>> Steven
> > >>>>>
> > >>>>> --
> > >>>>> Dr. Steven Ericsson-Zenith
> > >>>>> Institute for Advanced Science & Engineering
> > >>>>> http://iase.info
> > >>>>> http://senses.info
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Nov 19, 2007, at 11:40 PM, Hubert Ritzdorf wrote:
> > >>>>>
> > >>>>>
> > >>>>>> (*) Assuming MPI process A with an int buffer "buf" and "buf[0]
> > >>>>>> = 0"
> > >>>>>> was
> > >>>>>>  set by MPI Process A. The cache and main memory contains buf
> > >>>>>> [0] = 0.
> > >>>>>> (*) Another MPI process or thread sets buf[0] = 1 after it was
> > >>>>>> triggered
> > >>>>>>  to do this. buf[0] is now 1 within the memory.
> > >>>>>>  But if MPI process A would perform now perform a = buf[0],
> > >>>>>>  the value of a would be 0 (since the SX system is not cache
> > >>>>>> coherent
> > >>>>>>  and the cache still contains buf[0] = 0)
> > >>>>>> (*) Now MPI process A calls MPI_Send (buf, ....)
> > >>>>>> (*) If the buf[0] is still in the cache and the MPI_Send would
> > >>>>>> perform
> > >>>>>>  b = buf[0], b would still get the value 0.
> > >>>>>> (*) Now, the buffer "buf" is sent to the destination process.
> > >>>>>>  When "sending" the data to another MPI process B in the same
> > >>>>>> node,
> > >>>>>>  the cache is bypassed and the correct value buf[0] = 1 is
> > >>>>>> written
> > >>>>>>  ("sent") to MPI process B. In addition, the value of buf[0] is
> > >>>>>>  flushed within the cache.
> > >>>>>> (*) If now MPI process A would perform c = buf[0] within
> > >>>>>> MPI_Send,
> > >>>>>>  c would get value 1.
> > >>>>>>
> > >>>>>> When I correctly understand the meaning and definition of keyword
> > >>>>>> const
> > >>>>>> the usage of keyword const would not applicable for "*buf".
> > >>>>>> When I remember correctly, also other vector systems were not
> > >>>>>> cache coherent. As far as I know, the IBM cell processor is also
> > >>>>>> not always cache coherent. When looking to the memory wall,
> > >>>>>> future (scalar) systems may also not be cache coherent.
> > >>>>>>
> > >>>>>> Hubert
> > >>>>>>
> > >>>>>> Steven Ericsson-Zenith wrote:
> > >>>>>>
> > >>>>>>> I think that there are two issues here.
> > >>>>>>>
> > >>>>>>> 1. What exactly is the behavior that relates to this cache
> > >>>>>>> coherency that would distinguish the use of const from the
> > >>>>>>> absence
> > >>>>>>> of const?
> > >>>>>>>
> > >>>>>>> 2. Why would the standard accommodate such a propriety behavior?
> > >>>>>>>
> > >>>>>>> If the send buffer is read from memory and causes a cache flush
> > >>>>>>> that has not maintained coherency isn't this a problem in
> > >>>>>>> anycase?
> > >>>>>>> It appears to suggest that any value in the cache is irrelevant.
> > >>>>>>>
> > >>>>>>> Perhaps you mean it the other way around: that is is the
> > >>>>>>> value in
> > >>>>>>> the cache that is the effective value of the send buffer, and
> > >>>>>>> that
> > >>>>>>> this value may be flushed if a read causes a cache update? This
> > >>>>>>> *would* potentially cause the effective value of the send
> > >>>>>>> buffer to
> > >>>>>>> change during the send.
> > >>>>>>>
> > >>>>>>> With respect,
> > >>>>>>> Steven
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Nov 18, 2007, at 4:55 PM, Erez Haba wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Can you elaborate or send a pointer related to that
> > >>>>>>>> architecture?
> > >>>>>>>>
> > >>>>>>>> The const keyword is a contract between the caller and the
> > >>>>>>>> caller
> > >>>>>>>> and the callee. It means in our example that the MPI_Send
> > >>>>>>>> function
> > >>>>>>>> will not change the user supplied buffer. Is this the case with
> > >>>>>>>> the SX systems?
> > >>>>>>>>
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi-
> > >>>>>>>> forum.org] On Behalf Of Hubert Ritzdorf
> > >>>>>>>> Sent: Sunday, November 18, 2007 11:27 AM
> > >>>>>>>> To: mpi-21@xxxxxxxxxxxxx
> > >>>>>>>> Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> > >>>>>>>>
> > >>>>>>>> This is correct. The SX systems are not cache coherent. The
> > >>>>>>>> send
> > >>>>>>>> buffer
> > >>>>>>>> is read directly from
> > >>>>>>>> memory and may cause a cache flush. Thus, the const  keyword
> > >>>>>>>> would
> > >>>>>>>> not  be  correct.
> > >>>>>>>>
> > >>>>>>>> Hubert Ritzdorf
> > >>>>>>>> NEC Laboratories Europe
> > >>>>>>>> IT Research Division
> > >>>>>>>> NEC Europe Ltd.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Snir, Marc wrote:
> > >>>>>>>>
> > >>>>>>>>> If I remember correctly, NEC or some other Japanese vendor
> > >>>>>>>>> had a
> > >>>>>>>>> system with non-choherent shared memory were this added
> > >>>>>>>>> restriction was important. I guess that a read could cause a
> > >>>>>>>>> flush to be canceled.
> > >>>>>>>>>
> > >>>>>>>>> On 11/17/07 1:45 AM, "Steven Ericsson-Zenith"
> > >>>>>>>>> <steven@xxxxxxxxxxxxx> wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> I think there is an intersect here with Erez's interest in
> > >>>>>>>>> const.
> > >>>>>>>>>
> > >>>>>>>>> In a parallel language it should be possible to send a
> > >>>>>>>>> constant
> > >>>>>>>>> value
> > >>>>>>>>> and read it at the same time (formally that would be fine).
> > >>>>>>>>> Indeed, it
> > >>>>>>>>> is possible in Occam and Ease/Carnap. I don't know the
> > >>>>>>>>> semantics of
> > >>>>>>>>> const in C functions, I assume it requires either a constant
> > >>>>>>>>> parameter, a pass by value copy operation or some other
> > >>>>>>>>> guarantee
> > >>>>>>>>> that
> > >>>>>>>>> the const is not written to (even by concurrent processes).
> > >>>>>>>>>
> > >>>>>>>>> The issue in MPI has to do with the termination properties,
> > >>>>>>>>> as I
> > >>>>>>>>> recall; i.e., the validity of the send buffer is not
> > >>>>>>>>> deterministic
> > >>>>>>>>> since the read is not synchronized with the send (so you can't
> > >>>>>>>>> know in
> > >>>>>>>>> an interleaving that a subsequent alteration - including
> > >>>>>>>>> deallocation
> > >>>>>>>>> - to the buffer value has not occurred). Futzing with the send
> > >>>>>>>>> buffer
> > >>>>>>>>> during a send operation seems to me to be an unforgivable
> > >>>>>>>>> rational.
> > >>>>>>>>>
> > >>>>>>>>> It really is equivalent to the usage rules one finds in a
> > >>>>>>>>> process
> > >>>>>>>>> oriented language (like CSP/Occam or Ease/Carnap), an
> > >>>>>>>>> asynchronous
> > >>>>>>>>> operation is, in effect, a distinct process and the buffer
> > >>>>>>>>> (if it
> > >>>>>>>>> is a
> > >>>>>>>>> variable) ceases to be in scope in all other processes
> > >>>>>>>>> during the
> > >>>>>>>>> period of concurrency. Indeed, such variables can only come
> > >>>>>>>>> back
> > >>>>>>>>> into
> > >>>>>>>>> scope if there is a cooperative barrier, they cannot come
> > >>>>>>>>> back into
> > >>>>>>>>> scope from  subordinate processes. Occam overcomes this by not
> > >>>>>>>>> allowing subordinates, Ease/Carnap overcomes this by only
> > >>>>>>>>> allowing
> > >>>>>>>>> constants (i.e., pass by value) to be passed to subordinates.
> > >>>>>>>>>
> > >>>>>>>>> With respect,
> > >>>>>>>>> Steven
> > >>>>>>>>> --
> > >>>>>>>>> Dr. Steven Ericsson-Zenith
> > >>>>>>>>> Institute for Advanced Science & Engineering
> > >>>>>>>>> http://iase.info
> > >>>>>>>>> http://senses.info
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Nov 16, 2007, at 3:25 PM, Erez Haba wrote:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> Thanks for the pointer!!!
> > >>>>>>>>>>
> > >>>>>>>>>> Can someone elaborate on the rational to prohibit read
> > >>>>>>>>>> access to a
> > >>>>>>>>>> buffer being sent?
> > >>>>>>>>>>
> > >>>>>>>>>> I think that it's more than a little loss of functionally.
> > >>>>>>>>>> Consider
> > >>>>>>>>>> a multi threaded application that wishes to send some data to
> > >>>>>>>>>> other
> > >>>>>>>>>> ranks and at the same time process this data locally?
> > >>>>>>>>>> Also consider the scenario when rank x sends the same data to
> > >>>>>>>>>> rank y
> > >>>>>>>>>> and z concurrently? does the text below prohibit that?
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Quote
> > >>>>>>>>>> --------------
> > >>>>>>>>>>
> > >>>>>>>>>> In a multi-threaded implementation of MPI, the system may de-
> > >>>>>>>>>> schedule a thread that is blocked on a send or receive
> > >>>>>>>>>> operation,
> > >>>>>>>>>> and schedule another thread for execution in the same address
> > >>>>>>>>>> space.
> > >>>>>>>>>> In such a case it is the user's responsibility not to
> > >>>>>>>>>> access or
> > >>>>>>>>>> modify a communication buffer until the communication
> > >>>>>>>>>> completes.
> > >>>>>>>>>> Otherwise, the outcome of the computation is undefined.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> [] Rationale.
> > >>>>>>>>>>
> > >>>>>>>>>> We prohibit read accesses to a send buffer while it is
> > >>>>>>>>>> being used,
> > >>>>>>>>>> even though the send operation is not supposed to alter the
> > >>>>>>>>>> content
> > >>>>>>>>>> of this buffer. This may seem more stringent than
> > >>>>>>>>>> necessary, but
> > >>>>>>>>>> the
> > >>>>>>>>>> additional restriction causes little loss of functionality
> > >>>>>>>>>> and
> > >>>>>>>>>> allows better performance on some systems --- consider the
> > >>>>>>>>>> case
> > >>>>>>>>>> where data transfer is done by a DMA engine that is not
> > >>>>>>>>>> cache-
> > >>>>>>>>>> coherent with the main processor. ( End of rationale.)
> > >>>>>>>>>>
> > >>>>>>>>>> ------------
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -----Original Message-----
> > >>>>>>>>>> From: Lisandro Dalcin [mailto:dalcinl@xxxxxxxxx]
> > >>>>>>>>>> Sent: Friday, November 16, 2007 3:03 PM
> > >>>>>>>>>> To: mpi-21@xxxxxxxxxxxxx
> > >>>>>>>>>> Cc: Erez Haba
> > >>>>>>>>>> Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> > >>>>>>>>>>
> > >>>>>>>>>> On 11/16/07, Erez Haba <erezh@xxxxxxxxxxxxx> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Such implementations would be really bad for muti-threaded
> > >>>>>>>>>>> applications, or those using async send. Is there an MPI
> > >>>>>>>>>>> contract
> > >>>>>>>>>>> that the application should not access the send buffer for
> > >>>>>>>>>>> read
> > >>>>>>>>>>> while the send is not complete?
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> AFAIK, there is. Please, see:
> > >>>>>>>>>>
> > >>>>>>>>>> http://www.mpi-forum.org/docs/mpi-11-html/node40.html
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>> From: owner-mpi-21@xxxxxxxxxxxxx [mailto:owner-mpi-21@mpi-
> > >>>>>>>>>>> forum.org] On Behalf Of Darius Buntinas
> > >>>>>>>>>>> Sent: Friday, November 16, 2007 1:11 PM
> > >>>>>>>>>>> To: mpi-21@xxxxxxxxxxxxx
> > >>>>>>>>>>> Subject: Re: [mpi-21] MPI Forum suggestions 11/14
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> I believe the standard allows an implementation to modify
> > >>>>>>>>>>> the
> > >>>>>>>>>>> buffer in
> > >>>>>>>>>>> a send call, as long as it's changed back before it
> > >>>>>>>>>>> completes.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I heard (don't remember from who), that there was an
> > >>>>>>>>>>> implementation
> > >>>>>>>>>>> that
> > >>>>>>>>>>> would do endian conversion in place on the source buffer
> > >>>>>>>>>>> before
> > >>>>>>>>>>> sending,
> > >>>>>>>>>>> then change it back.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Changing the standard to require the buffer pointer to be
> > >>>>>>>>>>> const
> > >>>>>>>>>>> could
> > >>>>>>>>>>> theoretically disallow that implementation.  So adding
> > >>>>>>>>>>> consts
> > >>>>>>>>>>> wouldn't
> > >>>>>>>>>>> be a minor change.
> > >>>>>>>>>>>
> > >>>>>>>>>>> -d
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 11/16/2007 02:06 PM, David Gingold wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Nov 16, 2007, at 2:42 PM, Jeff Squyres wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> 1. const: Sure, this seems like a good idea.  But it
> > >>>>>>>>>>>>> *is* an
> > >>>>>>>>>>>>> API
> > >>>>>>>>>>>>> change, and I am in agreement with Bill Gropp that there
> > >>>>>>>>>>>>> should
> > >>>>>>>>>>>>> be no
> > >>>>>>>>>>>>> API changes for MPI 2.1 (except for bug fixes).  Even if
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>> API
> > >>>>>>>>>>>>> changes supposedly won't matter (because someone will
> > >>>>>>>>>>>>> find a
> > >>>>>>>>>>>>> case
> > >>>>>>>>>>>>> where it does matter).  More specifically: this is a
> > >>>>>>>>>>>>> slippery
> > >>>>>>>>>>>>> slope
> > >>>>>>>>>>>>> for 2.1.  Making one "minor" change will lead to more
> > >>>>>>>>>>>>> minor
> > >>>>>>>>>>>>> changes
> > >>>>>>>>>>>>> which will lead to ...
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> If an implementation declares in its mpi.h:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> int MPI_Send(const void *, int, MPI_Datatype, int, int,
> > >>>>>>>>>>>> MPI_Comm);
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> does this violate the existing standard?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I don't see how this would break code that calls MPI_Send
> > >>>>>>>>>>>> ().  C
> > >>>>>>>>>>>> libraries made a similar transition with the advent of
> > >>>>>>>>>>>> ANSI C; I
> > >>>>>>>>>>>> believe
> > >>>>>>>>>>>> the const semantics are defined in a way that lets one do
> > >>>>>>>>>>>> just
> > >>>>>>>>>>>> this.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The const declaration allows an important optimization when
> > >>>>>>>>>>>> compiling
> > >>>>>>>>>>>> the calling functions.  But if an implementation is
> > >>>>>>>>>>>> allowed to
> > >>>>>>>>>>>> do
> > >>>>>>>>>>>> this,
> > >>>>>>>>>>>> then perhaps there is no need to change to the standard.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -dg
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> Lisandro Dalcín
> > >>>>>>>>>> ---------------
> > >>>>>>>>>> Centro Internacional de Métodos Computacionales en Ingeniería
> > >>>>>>>>>> (CIMEC)
> > >>>>>>>>>> Instituto de Desarrollo Tecnológico para la Industria Química
> > >>>>>>>>>> (INTEC)
> > >>>>>>>>>> Consejo Nacional de Investigaciones Científicas y Técnicas
> > >>>>>>>>>> (CONICET)
> > >>>>>>>>>> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> > >>>>>>>>>> Tel/Fax: +54-(0)342-451.1594
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Marc Snir
> > >>>>>>>>> Director, Illinois Informatics Initiative
> > >>>>>>>>> U Illinois at Urbana-Champaign
> > >>>>>>>>> LIS Bldg., Room 123
> > >>>>>>>>> 501 E Daniel St
> > >>>>>>>>> Champaign, IL 61820
> > >>>>>>>>> (217) 244 6568
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>> --
> > >>>>>>> Dr. Steven Ericsson-Zenith
> > >>>>>>> Institute for Advanced Science & Engineering
> > >>>>>>> http://iase.info
> > >>>>>>> http://senses.info
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>
> > >>
> > >>
> > >
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>