RE: [mpi-21] Follow up on the MPI Forum meeting

["Rajeev Thakur" <thakur@xxxxxxxxxxx>]

Probably not for the case you mention, but if you had a single array of size
> 2 GB, you could scatterv it using datatypes.

Rajeev 

> -----Original Message-----
> From: owner-mpi-21@xxxxxxxxxxxxx 
> [mailto:owner-mpi-21@xxxxxxxxxxxxx] On Behalf Of Lisandro Dalcin
> Sent: Tuesday, November 20, 2007 3:39 PM
> To: mpi-21@xxxxxxxxxxxxx
> Subject: Re: [mpi-21] Follow up on the MPI Forum meeting
> 
> Sorry, Rajeev, but perhaps I was not clean enough, or I 
> missunderstood you.
> 
> Suppose you have a comm of size 2, and at some designated root process
> (rank 0 or 1 in this case) you have a lot of memory (much more than
> 2GB), and you have allocated to arrays of some type (let say, arrays
> of char's) both with size below 2GB. Now suppose the first array is
> 'near the begining' of your memory, and the second is 'near the end'
> of your memory.
> 
> Is there any way to do MPI_Scatterv() or those two arrays avoiding a
> 'concatenation'/'packing' into a temporary buffer? In 32 bits (and if
> your kernel maps no more than 2GB) you can do this by using the
> address of one of the arrays as 'buf', and computing a displacement as
> the difference between the addresses for the two arrays.
> 
> I use this for implementing collectives in my Python bingings for MPI,
> were I need to communicate a list of arbitrary Python objects (where
> len(list) == size(comm)), those objects are actually serialized in
> independent contiguous buffers using standard Python 'pickle'
> serialization. All this work fine in 32 bits, but it does not work in
> 64 bits.
> 
> I believe that if 'disp' arguments to vector variant collectives were
> of type MPI_Aint, that would be possible and easy. But the are
> integers, and then you cannot 'scatterv' if the start adress of the
> different pieces are more than 2GB appart each other.
> 
> 
> On 11/20/07, Rajeev Thakur <thakur@xxxxxxxxxxx> wrote:
> > > > 2G is not a problem -- there *are* well-known workarounds.
> > > E.g., you
> > > > can make a datatype and send a suitable count such that the
> > > aggregate
> > > > size is >>2GB.
> > >
> > > You are right, for p2p and bcast/gather/scatter that way 
> would work.
> > >
> > > However, I cannot find the way to use the vector variants 
> collectives
> > > like scatterv/gatherv/alltoallv when the chunks you want to
> > > communicate can be at any location in memory, and and integer
> > > displacements cannot span more than 2GB. I'm missing something? Of
> > > course, I know alltoallw, but I would prefer to use the 
> 'cannonical'
> > > MPI collectives for this.
> >
> > The displacements in the "v" versions are in units of the 
> datatype, so if
> > you can construct a bigger datatype, it will work.
> >
> > Rajeev
> >
> >
> 
> 
> -- 
> Lisandro Dalcín
> ---------------
> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> PTLC - Güemes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> 
>