draft of chapter 4

• Messages sorted by: [ date ][ thread ][ subject ][ author ]
• Next message: Shane Hebert: "RE: we should wait for 1sided implementations"
• Previous message: Marc Snir: "Re: we should wait for 1sided implementations"

--boundary=_0
Content-Type: text/plain

-:) -:) -:) *** (:- (:- (:-

The postscript for the current version of the chapter is appended below. The
main changes from the previous version, sent out two weeks ago, is that I
included many corrections pointed out by Rolf, and I changed the semantics so
as to accomadate better systems where remote accesses cannot be done easily at
byte granularity (e.g., Cray). To allow such systems to execute put/get
transfers without requiring locks I left the outcome of conflicting put/get
accesses undefined (this is for "true" conflicts, not "false conflicts" due to
false sharing).

--boundary=_0
Content-Type: application/postscript
Content-Description: one-sided.ps

%!PS-Adobe-2.0
%%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software
%%Title: temp.dvi
%%Pages: 43 1
%%BoundingBox: 0 0 612 792
%%EndComments
%%BeginProcSet: tex.pro
/TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch
load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{
isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale
Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get
round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10
N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{
/vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{
statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N
/FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin
/FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array
/BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2
array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}
B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont
setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup
length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{
ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B
/ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0
N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S
dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0
ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice
ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}
imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr
put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf
div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook
known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore
showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook
known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256
array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N
/RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X
/rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0
7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1
TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1
-.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{
moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{
S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B
/j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w
}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p
a}B /bos{/SS save N}B /eos{clear SS restore}B end
%%EndProcSet
TeXDict begin 1000 300 300 @start /Fa 1 59 df<70F8F8F87005057D840C>58
D E /Fb 1 1 df<FFFFC0FFFFC012027D871A>0 D E /Fc 4 62 df<03060C1C183830706060E0
E0E0E0E0E0E0E0E0E06060703038181C0C0603081E7E950D>40 D<C0603038181C0C0E06060707
070707070707070706060E0C1C18383060C0081E7E950D>I<1F003FC061E0C0E0E070E0700070
007000E000C001C0030006000C30183030707FE0FFE0FFE00C137E9211>50
D<7FFFE0FFFFF0000000000000000000000000000000000000FFFFF07FFFE0140A7E8B19>61
D E /Fd 6 113 df<0C000C001C003C00FC004C000C000C000C000C000C000C000C000C000C00
0C007FC07FC00A127E910F>49 D<7C00FF00E380418001801F807F80E180C180C180E3807FF03C
F00C0D7E8C0F>97 D<0E000E000E000000000000007E007E000600060006000600060006000600
06000600FFE0FFE00B137E920F>105 D<0380038003800000000000003F803F80018001800180
01800180018001800180018001800180018001804180E380FF007C0009197F920F>I<FEE0FFF0
3BB033303330333033303330333033303330FFFCFFFC0E0D808C0F>109
D<F780FFC038E0387030303030303030303030387038E03FC037003000300030003000FC00FC00
0C137F8C0F>112 D E /Fe 1 81 df<FFFFFFFFE0FFFFFFFFF07800001FF07C000003F03E0000
00F81F000000380F0000001C0F8000000C07C000000C03C000000603E000000001F000000000F8
0000000078000000007C000000003E000000001E000000001F000000000F8000000007C0000000
03C000000003C000000001800000000380000000030000000006000000000C0000000018000000
00380000000030000000006000000000C0000006018000000C038000000C030000001C06000000
380C000000F818000001F83800001FF03FFFFFFFF07FFFFFFFF0FFFFFFFFE0272A7E7F2C>80
D E /Ff 34 121 df<FFE0FFE0FFE00B037F8C10>45 D<F0F0F0F004047B830E>I<00C001C007
C0FFC0FFC0FBC003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0
03C003C003C003C003C003C003C003C003C0FFFFFFFFFFFF10227CA118>49
D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E001E06001E02001E00001E00001
E00001E00003C00003C0000780000780000F00001E00003C0000780000F00001E00001C0000380
000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001F
FF003E0F003807807003C02003C02003C00003C00003C00003C0000780000780000F00001E0003
FC0003F80003FE00000F000007800003C00003C00001E00001E00001E00001E00001E08001E0C0
03C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00
006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00
3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00000F00
000F00000F00000F0015217FA018>I<FFFFE0FFFFE0FFFFE00001E00003C0000380000780000F
00000E00001E00001C00003C0000380000780000780000F00000F00000E00001E00001E00001E0
0003C00003C00003C00003C00003C0000780000780000780000780000780000780000780001321
7EA018>55 D<FFFC00FFFF80FFFFC0F007F0F001F0F00078F0003CF0003CF0003CF0003CF0003C
F00038F00078F000F0F003E0FFFFC0FFFF00FFFFC0F00FE0F001F8F00078F0003CF0001CF0001E
F0001EF0001EF0001EF0001EF0003CF0007CF000F8F003F0FFFFE0FFFFC0FFFE0017237BA220>
66 D<FFFFF0FFFFF0FFFFF0F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000FFFFE0FFFFE0FFFFE0F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000F00000F00000F00000F00000FFFFF8FFFFF8FFFFF815237BA21D>69
D<000FF000003FFE0000FFFF8001F81F8003E00780078001000F0000001E0000001E0000003C00
00003C000000780000007800000078000000F0000000F0000000F0000000F0000000F0000000F0
000000F000FFC0F000FFC0F000FFC0780003C0780003C0780003C03C0003C03C0003C01E0003C0
1E0003C00F0003C0078003C003E003C001F80FC000FFFFC0003FFF00000FF8001A257DA321>71
D<FC00007EFE0000FEFE0000FEFE0000FEF70001DEF70001DEF70001DEF78003DEF380039EF380
039EF3C0079EF3C0079EF1C0071EF1C0071EF1E00F1EF0E00E1EF0E00E1EF0F01E1EF0F01E1EF0
701C1EF0783C1EF0783C1EF038381EF03C781EF03C781EF01C701EF01C701EF01EF01EF00EE01E
F00EE01EF00FE01EF007C01EF007C01EF007C01EF000001E1F237BA22A>77
D<FC001EFE001EFE001EFE001EF7001EF7001EF3801EF3801EF3C01EF1C01EF1C01EF1E01EF0E0
1EF0F01EF0F01EF0781EF0781EF0381EF03C1EF03C1EF01E1EF01E1EF00E1EF00F1EF0071EF007
1EF0079EF0039EF0039EF001DEF001DEF000FEF000FEF000FEF0007E17237BA222>I<001FC000
007FF00001FFFC0003F07E0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000
F0780000F0780000F070000070F0000078F0000078F0000078F0000078F0000078F0000078F000
0078F0000078F0000078780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F
0007C00F800F8007C01F0003F07E0001FFFC00007FF000001FC0001D257DA324>I<FFFE00FFFF
80FFFFC0F003E0F000F0F00078F00078F0003CF0003CF0003CF0003CF0003CF0003CF00078F000
78F000F0F003E0FFFFC0FFFF80FFFE00F00000F00000F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000F0000016237BA21F>I<FFFC00FFFF80FFFFC0F0
03E0F000F0F00078F00038F0003CF0003CF0003CF0003CF0003CF00038F00078F000F0F003E0FF
FFC0FFFF80FFFE00F01E00F00F00F00700F00780F00380F003C0F001E0F001E0F000F0F000F0F0
0078F00038F0003CF0001EF0001EF0000F18237BA21F>82 D<F0001F0001E0F0003F8001E0F000
3F8001E078003F8001C078003B8003C078007BC003C078007BC003C03C0073C007803C00F1C007
803C00F1E007803C00F1E007801E00E1E00F001E01E0E00F001E01E0F00F001E01E0F00F000F01
C0F01E000F03C0701E000F03C0781E000703C0781C00078380383C00078380383C000787803C3C
000387003C38000387001C380003C7001C780003CF001C780001CE001E700001CE000E700001CE
000E700001EE000EF00000EC000EE00000FC0007E00000FC0007E00000FC0007E00000780007C0
002B237FA22E>87 D<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780FF00F
F00FF00FF80F7C3F7FFF3FFF1F8F10167E9517>97 D<F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000F00000F1F000F7FC00FFFE00FC1F00F80F00F007
80F00780F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F00780F00780F80F00FC3E
00FFFE00F7F800F3F00012237CA219>I<01FC0007FF000FFF801F07803C018078000078000070
0000F00000F00000F00000F00000F00000F000007800007800007800403C00C01F07C00FFFC007
FF8001FC0012167E9516>I<0003C00003C00003C00003C00003C00003C00003C00003C00003C0
0003C00003C00003C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F003C0
F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003E3C0
12237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0FFFF
C0F00000F00000F000007000007800007800003C00801F07800FFF8007FF0001F80012167E9516
>I<01F0F807FFF80FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F001F
FE001FFC0039F0003800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078F0
0078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>103
D<F0F0F0F0000000000000000000F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004237D
A20B>105 D<F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F01F80F03F00F03E00F07C00F0F800F1F000F3E000F7C000FFC000FFC000FFE000
FFE000FDF000F8F800F07800F07C00F03E00F01E00F01F00F00F00F00F80F007C012237CA218>
107 D<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004
237DA20B>I<F0F807C0F3FE1FF0FFFF7FF8FE0FF078FC0FE07CF807C03CF807C03CF007803CF0
07803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803C
F007803CF007803CF007803CF007803C1E167C9529>I<F1F8F7FCFFFEFE1EF80FF80FF00FF00F
F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00F10167C9519>I<01FC0007
FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F00078F00078F00078F0007878
00F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F9518>I<F1F000F7FC00FFFE00
FC3F00F80F00F00780F00780F007C0F003C0F003C0F003C0F003C0F003C0F003C0F007C0F00780
F00F80F80F00FC3E00FFFE00F7F800F3F000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F0000012207C9519>I<F0E0F3E0F7E0FFE0FF00FC00F800F800F000F000F000F0
00F000F000F000F000F000F000F000F000F000F0000B167C9511>114 D<07F01FFE3FFF3C0F78
03780078007C003F003FF01FF80FFC01FE001F000F000F400FE00FF81EFFFE3FFC0FF010167F95
13>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F000F000F000F000F000F000F00
0F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12>I<F00FF00FF00FF00FF00FF00F
F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF83F7FFF7FEF1F8F10167C9519>I<
7801F07C01E03E03C01E07C00F0780078F0007DE0003FC0001FC0000F80000700000F80001FC00
03DC00039E00078F000F07801E07801E03C03C01E07800F0F800F81516809516>120
D E /Fg 6 111 df<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C181838306040060E7C
840D>I<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E
00F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015147E9318>97
D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F
700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207E9F14>I<00007C0000FE
00019E00039E00030C000700000700000700000700000E00000E00000E0000FFF001FFF0000E00
001C00001C00001C00001C00001C00003800003800003800003800003800007000007000007000
00700000700000E00000E00000E00000E00000C00001C00031C000798000F300007E00003C0000
17297E9F16>102 D<0F07C0001F8FE00031D8700061F0380061E0380061C03800C3C070000380
700003807000038070000700E0000700E0000700E0000701C1800E01C3000E0383000E0386000E
018E001C01FC000C00F00019147F931B>110 D E /Fh 17 119 df<387CFEFEFE7C3800000000
387CFEFEFE7C3807127D910D>58 D<00038000000380000007C0000007C0000007C000000FE000
000FE000001FF000001BF000001BF0000031F8000031F8000061FC000060FC0000E0FE0000C07E
0000C07E0001803F0001FFFF0003FFFF8003001F8003001F8006000FC006000FC00E000FE00C00
07E0FFC07FFEFFC07FFE1F1C7E9B24>65 D<FFFFF800FFFFFF000FC01FC00FC007E00FC003F00F
C001F80FC001F80FC000FC0FC000FC0FC000FC0FC000FE0FC000FE0FC000FE0FC000FE0FC000FE
0FC000FE0FC000FE0FC000FE0FC000FC0FC000FC0FC000FC0FC001F80FC001F80FC003F00FC007
E00FC01FC0FFFFFF00FFFFF8001F1C7E9B25>68 D<FFC00003FFFFE00007FF0FE00007F00DF000
0DF00DF0000DF00DF0000DF00CF80019F00CF80019F00C7C0031F00C7C0031F00C3E0061F00C3E
0061F00C1F00C1F00C1F00C1F00C1F00C1F00C0F8181F00C0F8181F00C07C301F00C07C301F00C
03E601F00C03E601F00C01FC01F00C01FC01F00C01FC01F00C00F801F00C00F801F0FFC0701FFF
FFC0701FFF281C7E9B2D>77 D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFF
C03F87C07E07C0FC07C0FC07C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97
D<03FC000FFE001F1F003E1F007C1F007C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00
007E01803E03801F07000FFE0003F80011127E9115>99 D<01FC000FFF001F0F803E07C07C03C0
7C03E0FC03E0FFFFE0FFFFE0FC0000FC0000FC00007C00007E00603E00C01F81C00FFF0001FC00
13127F9116>101 D<03F0F00FFFF83E1F383C0F307C0F807C0F807C0F807C0F803C0F003E1F00
1FFC0033F0003000007000003800003FFF003FFFC01FFFE03FFFF07801F8F00078F00078F00078
7800F03E03E01FFFC003FE00151B7F9118>103 D<1E003F003F007F003F003F001E0000000000
000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FF
E0FFE00B1E7F9D0E>105 D<FF00FF001F001F001F001F001F001F001F001F001F001F001F001F
001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1D7F9C0E>108
D<FF1FC0FF7FE01FE1F01F80F81F80F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00
F81F00F81F00F81F00F8FFE7FFFFE7FF18127F911B>110 D<01FC000FFF801F07C03E03E07C01
F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F87C01F07C01F03E03E01F07C00FFF8001FC
0015127F9118>I<FE3E00FE7F001ECF801F8F801F8F801F07001F00001F00001F00001F00001F
00001F00001F00001F00001F00001F0000FFF000FFF00011127F9114>114
D<1FD83FF87038E018E018F000FF807FE07FF01FF807FC007CC01CC01CE01CF038FFF0CFC00E12
7E9113>I<030003000300070007000F000F003F00FFFCFFFC1F001F001F001F001F001F001F00
1F001F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913>I<FF07F8FF07F81F00F81F00F81F00
F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F01F80F03F80FFEFF03F8
FF18127F911B>I<FFC1FCFFC1FC1F00601F80E00F80C00FC0C007C18007C18003E30003E30001
F60001F60001FE0000FC0000FC0000780000780000300016127F9119>I
E /Fi 34 123 df<00003F03E00000FF87F00001C78CF00003879CF00003031860000700380000
070038000007003800000E003800000E007000000E007000000E00700000FFFFFF8001FFFFFF00
001C00E000001C00E000001C00E000001C00E000003800E000003801C000003801C000003801C0
00003801C000007001C0000070038000007003800000700380000070038000007003800000E007
000000E007000000E007000000E007000001C006000001C00E000001C00E000031860C00007B8F
180000F31E3800007E0FF000003C07C000002429829F1C>11 D<00003FCE0000FFEE0001C1FC00
0381FC000300DC0007001C0007003800070038000E0038000E0038000E0070000E007000FFFFF0
01FFFFF0001C00E0001C00E0001C00E0001C00E0001C01C0003801C0003801C0003801C0003803
8000380380007003800070038C00700718007007180070071800E0073000E003F000E001E000E0
000000C0000001C0000001C00000318000007B800000F30000007E0000003C0000001F29829F1B
>13 D<1C3C3C3C3C0C0C1818307060C080060E7D840E>44 D<70F8F8F0E005057B840E>46
D<000007000000070000000F0000000F0000001F0000003F0000003F0000006F0000006F000000
CF000000CF0000018F0000038F0000030F0000060F0000060F00000C0F80000C07800018078000
180780003FFF80007FFF800060078000C0078000C0078001800780018007800300078007000780
0F0007807FC07FF8FFC07FF81D207E9F22>65 D<01FFFFFE01FFFFFC001E003C001E001C001E00
1C003C000C003C000C003C000C003C001800780C1800780C1800780C0000781C0000F0380000FF
F80000FFF80000F0380001E0300001E0300001E0303001E0306003C0006003C0006003C000C003
C001C007800180078003800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69
D<01FFFFFC01FFFFF8001E0078001E0038001E0038003C0018003C0018003C0018003C00300078
0C3000780C3000780C0000781C0000F0380000FFF80000FFF80000F0380001E0300001E0300001
E0300001E0300003C0000003C0000003C0000003C0000007800000078000000780000007800000
FFFC0000FFF800001E1F7D9E1E>I<01FF0000FFC001FF0000FFC0001F0001F800001F0001F800
001F0003780000370003F00000370006F0000037000CF0000037000CF00000670019E000006380
19E00000638031E00000638031E00000C38063C00000C380C3C00000C380C3C00000C38183C000
01838187800001838307800001838307800001838607800003038C0F000003038C0F0000030398
0F00000301D80F00000601F01E00000601F01E00000601E01E00000E01C01E0000FFE1C3FFE000
FFC183FFC0002A1F7D9E29>77 D<0001FC000007FF00001F07C0003801E000F000E001E0007003
C0007007800078070000380F0000381E0000381E0000383C0000383C0000387800007878000078
7800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C070000780780007
0078000F0038001E003C003C001E00F0000F83E00007FF800001FC00001D217A9F23>79
D<01FFFF0001FFFFC0001E01E0001E00F0001E0078003C0078003C0078003C0078003C00780078
00F0007800F0007801E0007803C000F00F0000FFFE0000FFF80000F03C0001E01E0001E00E0001
E00F0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C1807803C1807803C30
FFF81E30FFF00FE0000007C01D207D9E21>82 D<0FFFFFF01FFFFFE01E0781E0180780E0380780
60300F0060300F0060600F0060600F00C0C01E00C0C01E00C0001E0000001E0000003C0000003C
0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0000000
F0000001E0000001E0000001E0000003E00000FFFF0000FFFF00001C1F789E21>84
D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00F00E
00F00E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F00070007
0007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01E
E01EE03CE03CE038607071E03FC01F0010207B9F15>I<007E0001FF000383800F07801E07801C
07003C0200780000780000780000F00000F00000F00000F00000F00000700200700700381E001F
F80007E00011147C9315>I<0000780003F80003F00000700000700000700000E00000E00000E0
0000E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700
F00E00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F078015207C9F17>I<007C
0001FF000783000F01801E01803C01803C0300780E007FFC007FE000F00000F00000F00000F000
007000007002007807003C1E001FF80007E00011147C9315>I<0000F80001FC0003BC00033C00
0718000700000700000E00000E00000E00000E00000E0001FFE001FFE0001C00001C00001C0000
380000380000380000380000380000700000700000700000700000700000700000E00000E00000
E00000E00001C00001C00001C0000180003380007B8000F300007E00003C00001629829F0E>I<
003C6000FF7001E3E00381E00701E00F01E00F01C01E01C01E01C01E01C03C03803C03803C0380
3C03803C07003C0F001C1F001E3F000FFE0003CE00000E00000E00001C00001C00301C00783800
F0F0007FE0003F8000141D7E9315>I<01E0000FE0000FC00001C00001C00001C0000380000380
00038000038000070000070000073E00077F000EC3800F81C00F01C00E01C01E03801C03801C03
801C0380380700380700380700380E18700E30700E30701C60700C60E00FC060078015207D9F17
>I<006000F000F000E000000000000000000000000000000F001F80318031C063806380C38007
00070007000E000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>I<03C01FC01F8003
800380038007000700070007000E000E000E000E001C001C001C001C0038003800380038007000
700070007180E300E300E300E6007E003C000A207C9F0C>108 D<1E07C0F8003F1FE1FC0033B8
730E0063E076070063C03C07006380380700C780780E000700700E000700700E000700700E000E
00E01C000E00E01C000E00E01C000E00E038601C01C038C01C01C038C01C01C071801C01C03180
3803803F001801801E0023147D9325>I<1E07C03F1FE033B87063E07063C038638038C7807007
00700700700700700E00E00E00E00E00E00E01C31C01C61C01C61C038C1C018C3801F81800F018
147D931A>I<007C0001FF000383800F01C01E01C01C01E03C01E07801E07801E07801E0F003C0
F003C0F003C0F00780F00700700F00701E003838001FF00007C00013147C9317>I<03C1E007E7
F8067E3C0C7C1C0C781E0C701E18E01E00E01E00E01E00E01E01C03C01C03C01C03C01C07803C0
7803C07003C0E003E3C0077F80071E000700000700000E00000E00000E00000E00001C0000FFC0
00FFC000171D809317>I<1E0F003F3F8033F1C063C1C063C3C06383C0C7838007000007000007
00000E00000E00000E00000E00001C00001C00001C00001C000038000018000012147D9313>
114 D<00FC03FE07070E0F0E0F0E0E1E000F800FF007F803FC003E001E701EF01CF01CE0386070
3FE01F8010147D9313>I<018001C0038003800380038007000700FFF0FFF00E000E000E000E00
1C001C001C001C0038003800380038307060706070C071803F001E000C1C7C9B0F>I<0F00601F
80703180E031C0E06380E06380E0C381C00701C00701C00701C00E03800E03800E03800E038C0E
07180E07180E0F180E1F3007F3F003E1E016147D9318>I<0F01C01F83C03183E031C1E06380E0
6380E0C380C00700C00700C00700C00E01800E01800E01800E03000E03000E06000E06000F0C00
07F80001E00013147D9315>I<0F0060E01F8071E03180E1F031C0E0F06380E0706380E070C381
C0600701C0600701C0600701C0600E0380C00E0380C00E0380C00E0381800E0381800E0781000E
078300070F860007F9FC0001F0F8001C147D931E>I<0787800FCFC018F8E03070E06071E06071
E0C0E1C000E00000E00000E00001C00001C00001C00071C060F380C0F380C0E38180C7C3007CFE
00387C0013147D9315>I<0F00601F80703180E031C0E06380E06380E0C381C00701C00701C007
01C00E03800E03800E03800E03800E07000E07000E0F000E1F0007FE0003EE00000E00000E0000
1C00781C0078380070700060E0003FC0001F0000141D7D9316>I<00E06003F0C007FFC00E1F80
0C0300000600000C0000180000300000600000C0000180000300000600C00C01801801803F8700
7FFE0060FC00C0780013147E9313>I E /Fj 44 121 df<01F1C003F1C007F1C00701C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C0FFF1C0FFF1C0FFF1C00E01C00E01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0121D809C16>
13 D<00E001C0038007000E000E001C001C003800380038007000700070007000E000E000E000
E000E000E000E000E000E000E000E000E00070007000700070003800380038001C001C000E000E
000700038001C000E00B2A7E9E10>40 D<E000700038001C000E000E0007000700038003800380
01C001C001C001C000E000E000E000E000E000E000E000E000E000E000E000E001C001C001C001
C0038003800380070007000E000E001C0038007000E0000B2A7E9E10>I<7878787838307060E0
05097D830C>44 D<010007003F00FF00FF00C70007000700070007000700070007000700070007
00070007000700070007000700070007000700FFF8FFF8FFF80D1C7C9B15>49
D<001C0000003E0000003E0000002E0000006700000067000000E7800000C7800000C3800001C3
C0000183C0000181C0000381E0000381E0000700F0000700F0000600F0000E0078000FFFF8000F
FFF8001FFFFC001C003C0018003C0038001E0038001E0070001F0070000F0070000F00E0000780
191D7F9C1C>65 D<FFF800FFFF00FFFF80F00FC0F003E0F000F0F000F0F000F0F000F0F000F0F0
01E0F007C0FFFF80FFFE00FFFF80F01FC0F003E0F000F0F00078F00078F00078F00078F00078F0
00F0F001F0F007E0FFFFC0FFFF80FFFC00151D7C9C1C>I<003FC000FFF003FFF007C0700F8000
1E00003E00003C0000780000780000780000F00000F00000F00000F00000F00000F00000F00000
F00000F000007800007800007800003C00003E00001E00000F800807C07803FFF800FFF0003F80
151F7D9D1B>I<FFFC00FFFF00FFFF80F00FE0F003F0F000F0F00078F00078F0003CF0003CF000
1CF0001EF0001EF0001EF0001EF0001EF0001EF0001EF0001EF0003CF0003CF0007CF00078F000
F0F001F0F007E0FFFFC0FFFF00FFFC00171D7C9C1E>I<FFFFC0FFFFC0FFFFC0F00000F00000F0
0000F00000F00000F00000F00000F00000F00000FFFF80FFFF80FFFF80F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F00000FFFFC0FFFFC0FFFFC0121D7C9C19>I<
FFFF80FFFF80FFFF80F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
FFFF00FFFF00FFFF00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000111D7C9C18>I<003F8001FFF003FFF807C0F80F00181E00003E00003C00
00780000780000780000F00000F00000F00000F00000F00000F00000F007F8F007F8F007F87800
387800387800383C00383E00381E00380F003807C07803FFF801FFF0003F80151F7D9D1C>I<F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0FFFFF0FF
FFF0FFFFF0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F0
00F0F000F0F000F0141D7C9C1D>I<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0
F0F0F0F0F0041D7C9C0C>I<F000F000F000F000F000F000F000F000F000F000F000F000F000F0
00F000F000F000F000F000F000F000F000F000F000F000F000FFFEFFFEFFFE0F1D7C9C16>76
D<FC0007E0FC0007E0FC0007E0EE000DE0EE000DE0EE000DE0E70019E0E70019E0E70019E0E780
39E0E38031E0E3C071E0E3C071E0E1C061E0E1C061E0E1E0E1E0E1E0E1E0E0E0C1E0E0F1C1E0E0
7181E0E07181E0E07181E0E03B01E0E03B01E0E03B01E0E01E01E0E01E01E0E01E01E0E00001E0
1B1D7C9C24>I<FC0070FC0070FE0070EE0070EF0070E70070E70070E78070E38070E3C070E3C0
70E1E070E1E070E0E070E0F070E07070E07870E07870E03C70E03C70E01C70E01E70E00E70E00E
70E00F70E00770E007F0E003F0E003F0141D7C9C1D>I<003F000001FFE00003FFF00007C0F800
0F807C001E001E003E001F003C000F00780007807800078078000780F00003C0F00003C0F00003
C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C078000780780007807C000F803C00
0F003E001F001F003E000F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>I<FFFC
00FFFF00FFFF80F007C0F003E0F001E0F000F0F000F0F000F0F000F0F000F0F001E0F003E0F007
C0FFFF80FFFF00FFFC00F00000F00000F00000F00000F00000F00000F00000F00000F00000F000
00F00000F00000141D7C9C1B>I<FFF800FFFF00FFFF80F007C0F001E0F000E0F000F0F000F0F0
00F0F000F0F000E0F001E0F007C0FFFF80FFFF00FFF800F03C00F01C00F01E00F00F00F00F00F0
0780F00780F003C0F003C0F001E0F000F0F000F0F00078151D7C9C1B>82
D<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F00000F800007C00007F00
003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C00003C00003C00003C0C007
80E00780FC1F007FFE001FFC0007F000121F7E9D17>I<FFFFFF80FFFFFF80FFFFFF80001E0000
001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E00
00001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E
0000001E0000001E0000001E0000001E0000001E0000191D7F9C1C>I<F00070F00070F00070F0
0070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F0
0070F00070F00070F00070F00070F00070F000F07800E07801E03C03C01F07800FFF8007FE0001
F800141E7C9C1D>I<F0000380F00003807800070078000700780007003C000E003C000E003E00
0E001E001C001E001C000F0038000F0038000F003800078070000780700003C0600003C0E00003
C0E00001E0C00001E1C00001E1C00000F1800000F380000073000000730000007B0000003E0000
003E0000001C0000191D7F9C1C>I<F000F00070F000F80070F000F800707800B8006078019C00
E078019C00E078019C00E03C039C01C03C031E01C03C030E01C01E030E01801E070F03801E070F
03801E060F03800F060703000F0E0787000F0C078700070C038600070C038600079C038E000798
03CE00039801CC00039801CC00039801CC0001D001C80001D000D80001F000F80001E000F80000
E0007000241D7F9C27>I<78000E007C001E003C003C001E0038000F0070000F00F0000781E000
03C1C00001C3C00001E7800000F70000007E0000003E0000003C0000003C0000007E0000007700
0000E7800001E3800003C1C0000381E0000700F0000F00F8000E0078001C003C003C003E007800
1F0070000F00F0000F80191D7F9C1C>I<F80001E07C0001C03E0003801E0007801F0007000F80
0E0007801E0007C01C0003E03C0001E0380001F0700000F0F0000078E000007DC000003FC00000
1F8000001F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000
000F0000000F0000000F0000000F00001B1D809C1C>I<0FC03FF07FF87038401C001C001C00FC
0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 D<E000E000E000E000E000E0
00E000E000E000E000E000E3E0EFF0FFF8F87CF01CE01EE00EE00EE00EE00EE00EE00EE01CF01C
F87CFFF8EFF0E3C00F1D7D9C15>I<07E00FF81FFC3C1C70047000E000E000E000E000E000E000
700070043C1C1FFC0FF807E00E127E9112>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000
E000E000700070043C1C3FFC1FF807E00E127E9112>101 D<03C3C00FFFC01FFFC01C3800381C
00381C00381C00381C00381C001C38001FF8001FF0003BC0003800003800001FFC001FFF003FFF
80700780E001C0E001C0E001C0F003C07C0F803FFF001FFE0007F800121B7F9115>103
D<F0F0F0F000000000000000707070707070707070707070707070707070041D7E9C0A>105
D<E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0031D7D9C0A>108
D<E3F03F00EFF8FF80FFFDFFC0F81F81E0F00F00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E
00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E01B127D9124
>I<E3E0EFF0FFF8F83CF01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01C0E
127D9115>I<03F0000FFC001FFE003C0F00780780700380E001C0E001C0E001C0E001C0E001C0
F003C07003807807803C0F001FFE000FFC0003F00012127F9115>I<078E1FEE3FFE7C3E781E70
0EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E000E000E000E000E000E000E000E
000E0F1A7E9115>113 D<E380E780EF80FC00F800F000F000E000E000E000E000E000E000E000
E000E000E000E00009127D910E>I<1FC03FF07FF0F030E000E000F0007F003FC01FE000F00038
00388038F078FFF07FE01FC00D127F9110>I<1C001C001C001C001C001C00FFE0FFE0FFE01C00
1C001C001C001C001C001C001C001C001C001C001C201FF00FF007C00C187F970F>I<E01CE01C
E01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE07CFFFC7FDC3F1C0E127D9115>I<
E007E0077006700E700E381C381C381C1C381C381C380E700E700660076007E003C003C010127F
9113>I<7003807807003C0E001C1C000E1C0007380003F00001E00001C00001E00003F0000738
000E18000E1C001C0E00380700700380F003C01212809113>120 D E /Fk
61 123 df<003F0F0000FFBF8003C3F3C00703E3C00703C1800E01C0000E01C0000E01C0000E01
C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E
01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0007F87FC00
7F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C00703C00E01800E00000E00000E0000
0E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D809C17>I<003FC000FFC003C3C00703
C00701C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0FFFFC00E01C00E01C00E01C00E01
C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF87FCFF8151D
809C17>I<003F03F00000FFCFF80003C0FC1C000701F03C000701F03C000E00E018000E00E000
000E00E000000E00E000000E00E000000E00E00000FFFFFFFC00FFFFFFFC000E00E01C000E00E0
1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00
E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF807FC7FCFF80211D809C23>
I<7070F8F8FCFCFCFC7C7C0C0C0C0C0C0C181818183030606040400E0D7F9C15>34
D<00F0000001F8000003980000030C0000070C0000070C0000070C0000070C0000070C00000718
0000073800000730000007607FE003C07FE003801F0003800C0007C00C000DC0180019E0180038
E0300070F06000F0706000F038C000F03CC000F01F8000F00F006078070060780F80C03C39E1C0
1FF0FF8007C03E001B1F7E9D20>38 D<00C00180030006000E000C001C00180038003000300070
00700060006000E000E000E000E000E000E000E000E000E000E000E000E0006000600070007000
30003000380018001C000C000E0006000300018000C00A2A7D9E10>40 D<C0006000300018001C
000C000E000600070003000300038003800180018001C001C001C001C001C001C001C001C001C0
01C001C001C0018001800380038003000300070006000E000C001C00180030006000C0000A2A7E
9E10>I<70F0F8F8781818183030706040050D7D840C>44 D<FFE0FFE0FFE00B0380890E>I<70F8
F8F87005057D840C>I<030007003F00FF00C70007000700070007000700070007000700070007
0007000700070007000700070007000700070007000700FFF8FFF80D1C7C9B15>49
D<07C01FF03878603C601EF01EF80FF80FF80F700F000F000E001E001C003C0078007000E001C0
038007000E030C03180330067FFEFFFEFFFE101C7E9B15>I<70F8F8F870000000000000000070
F8F8F87005127D910C>58 D<70F8F8F870000000000000000070F0F8F878181818303070604005
1A7D910C>I<0FE03FF8703C601CF01EF01EF01E001E003C007800E001C0018003800300030003
00030003000200000000000000000007000F800F800F8007000F1D7E9C14>63
D<00060000000F0000000F0000000F0000001F8000001F8000001F8000001F80000033C0000033
C0000033C0000061E0000061E0000061E00000C0F00000C0F00000C0F000018078000180780001
FFF80003FFFC0003003C0003003C0006001E0006001E0006001E001F001F00FFC0FFF0FFC0FFF0
1C1D7F9C1F>65 D<FFFFC0FFFFF00F00F80F003C0F001C0F001E0F001E0F001E0F001E0F001C0F
003C0F00780FFFF00FFFE00F00F80F003C0F001E0F001E0F000F0F000F0F000F0F000F0F000F0F
001E0F003E0F007CFFFFF8FFFFC0181C7E9B1D>I<001F808000FFE18003F0338007801B800F00
0F801E0007801C0003803C000380780003807800018070000180F0000180F0000000F0000000F0
000000F0000000F0000000F0000000F00000007000018078000180780001803C0001801C000300
1E0003000F00060007800C0003F0380000FFF000001F8000191E7E9C1E>I<FFFFC000FFFFF000
0F007C000F001E000F000F000F0007000F0003800F0003C00F0003C00F0001C00F0001E00F0001
E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001C00F0001C00F0003C00F00
03800F0007800F000F000F001E000F007C00FFFFF000FFFFC0001B1C7E9B20>I<FFFFFCFFFFFC
0F007C0F001C0F000C0F000E0F00060F03060F03060F03060F03000F07000FFF000FFF000F0700
0F03000F03000F03030F03030F00030F00060F00060F00060F000E0F001E0F007CFFFFFCFFFFFC
181C7E9B1C>I<FFF3FFC0FFF3FFC00F003C000F003C000F003C000F003C000F003C000F003C00
0F003C000F003C000F003C000F003C000FFFFC000FFFFC000F003C000F003C000F003C000F003C
000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C00FFF3FFC0FFF3
FFC01A1C7E9B1F>72 D<FFF0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F
000F000F000F000F000F000F000F000F000F000F000F00FFF0FFF00C1C7F9B0F>I<FFF800FFF8
000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00
000F00000F00000F00180F00180F00180F00180F00380F00300F00700F00F00F01F0FFFFF0FFFF
F0151C7E9B1A>76 D<FF8000FF80FFC001FF800FC001F8000FC001F8000DE00378000DE0037800
0DE00378000CF00678000CF00678000CF00678000C780C78000C780C78000C780C78000C3C1878
000C3C1878000C3C1878000C1E3078000C1E3078000C1E3078000C0F6078000C0F6078000C0F60
78000C07C078000C07C078000C07C078001E03807800FFC387FF80FFC387FF80211C7E9B26>I<
FF00FFC0FF80FFC00F801E000FC00C000FC00C000DE00C000CF00C000CF00C000C780C000C780C
000C3C0C000C1E0C000C1E0C000C0F0C000C0F0C000C078C000C07CC000C03CC000C01EC000C01
EC000C00FC000C00FC000C007C000C003C000C003C001E001C00FFC01C00FFC00C001A1C7E9B1F
>I<003F800000FFE00003E0F80007803C000E000E001E000F003C00078038000380780003C078
0003C0700001C0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0
780003C0780003C0780003C03C0007803C0007801E000F000F001E0007803C0003E0F80000FFE0
00003F80001B1E7E9C20>I<FFFF80FFFFE00F00F00F00380F003C0F001E0F001E0F001E0F001E
0F001E0F001E0F003C0F00380F00F00FFFE00FFF800F00000F00000F00000F00000F00000F0000
0F00000F00000F00000F0000FFF000FFF000171C7E9B1C>I<FFFF0000FFFFE0000F00F0000F00
38000F003C000F001E000F001E000F001E000F001E000F001E000F003C000F0038000F00F0000F
FFE0000FFFC0000F01E0000F00F0000F0078000F0078000F0078000F0078000F0078000F007800
0F0078000F0078300F003830FFF03C60FFF01FE0000007C01C1D7E9B1F>82
D<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F00000F000007C00007FC0
003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001C0C001C0E00180E003
80F00300FC0E00CFFC0083F800121E7E9C17>I<7FFFFFC07FFFFFC0780F03C0700F01C0600F00
C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F0000000F0000000F0000000F
0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000
0F0000000F000003FFFC0003FFFC001B1C7F9B1E>I<FFF0FFC0FFF0FFC00F001E000F000C000F
000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C00
0F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C0007001800078018
000380300001C0300000E0E000007FC000001F00001A1D7E9B1F>I<FFE01FF0FFE01FF00F0007
800F0003000F000300078006000780060007C00E0003C00C0003C00C0003E01C0001E0180001E0
180000F0300000F0300000F03000007860000078600000786000003CC000003CC000003CC00000
1F8000001F8000001F8000000F0000000F0000000F0000000600001C1D7F9B1F>I<FFE0FFE1FF
FFE0FFE1FF1F001E007C0F001E00300F003F00300F003F00300F803F0070078067806007806780
60078067806003C0E780C003C0C3C0C003C0C3C0C001E0C3C18001E181E18001E181E18001E181
E18000F381F30000F300F30000F300F300007B00F600007E007E00007E007E00007E007E00003C
003C00003C003C00003C003C00001C0038000018001800281D7F9B2B>I<FFF00FFCFFF00FFC07
8003C007C0038003C0030003E0060001F0060000F00C0000F81C0000781800007C3800003C3000
001E6000001F6000000FC000000FC0000007800000078000000780000007800000078000000780
0000078000000780000007800000078000007FF800007FF8001E1C809B1F>89
D<08081818303060606060C0C0C0C0C0C0F8F8FCFCFCFC7C7C38380E0D7B9C15>92
D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE001FFE003E0E00780E00F00E00F00E
60F00E60F01E60783E603FFFC01F878013127F9115>97 D<FC0000FC00001C00001C00001C0000
1C00001C00001C00001C00001C00001C00001C7E001DFF001F87801E01C01C01E01C00E01C00F0
1C00F01C00F01C00F01C00F01C00F01C00E01C01E01E01C01F078019FF00187C00141D7F9C17>
I<03F00FF81E3C383C78187000F000F000F000F000F000F000780078063C061E0C0FF803E00F12
7F9112>I<001F80001F8000038000038000038000038000038000038000038000038000038003
E3800FFB801E0F80380780780380700380F00380F00380F00380F00380F00380F0038070038078
03803807801E1F800FFBF007E3F0141D7F9C17>I<03E00FF01C38381C781E700EFFFEFFFEF000
F000F000F000700078063C061E0C0FF803E00F127F9112>I<007801FC039E071E0E0C0E000E00
0E000E000E000E00FFE0FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E
007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3DC0381C00781E00781E00781E00781E00
381C003C3C003FF00037E0007000007000003000003FFC001FFF003FFF80700780E001C0E001C0
E001C0E001C07003803C0F001FFE0007F800121C7F9215>I<FC0000FC00001C00001C00001C00
001C00001C00001C00001C00001C00001C00001C7C001DFF001F07001E03801E03801C03801C03
801C03801C03801C03801C03801C03801C03801C03801C03801C0380FF9FF0FF9FF0141D7F9C17
>I<18003C007C003C001800000000000000000000000000FC00FC001C001C001C001C001C001C
001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<FC0000FC00001C00001C00
001C00001C00001C00001C00001C00001C00001C00001C7FC01C7FC01C3E001C18001C30001C60
001CC0001DE0001FE0001E70001C78001C38001C3C001C1C001C0E001C0F00FF9FE0FF9FE0131D
7F9C16>107 D<FC00FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00
1C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<FC7E07E000FD
FF9FF8001F83B838001E01E01C001E01E01C001C01C01C001C01C01C001C01C01C001C01C01C00
1C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C00FF8FF8FF
80FF8FF8FF8021127F9124>I<FC7C00FDFF001F07001E03801E03801C03801C03801C03801C03
801C03801C03801C03801C03801C03801C03801C0380FF9FF0FF9FF014127F9117>I<03F0000F
FC001E1E00380700780780700380F003C0F003C0F003C0F003C0F003C0F003C070038078078038
07001E1E000FFC0003F00012127F9115>I<FC7E00FDFF001F87801E03C01C01E01C01E01C00F0
1C00F01C00F01C00F01C00F01C00F01C01E01C01E01E03C01F07801DFF001C7C001C00001C0000
1C00001C00001C00001C0000FF8000FF8000141A7F9117>I<03E1800FF9801E1F803C07807807
80780380F00380F00380F00380F00380F00380F003807803807807803C07801E1F800FFB8007E3
80000380000380000380000380000380000380001FF0001FF0141A7F9116>I<FDE0FFF01F781E
781E301C001C001C001C001C001C001C001C001C001C001C00FFC0FFC00D127F9110>I<1F903F
F07070E030E030E030F8007F803FE00FF000F8C038C038E038E038F070DFE08FC00D127F9110>
I<0C000C000C000C000C001C001C003C00FFE0FFE01C001C001C001C001C001C001C001C001C30
1C301C301C301C301E600FC007800C1A7F9910>I<FC1F80FC1F801C03801C03801C03801C0380
1C03801C03801C03801C03801C03801C03801C03801C03801C07800C0F800FFBF003E3F014127F
9117>I<FF0FE0FF0FE01C07801C03000E06000E06000E0600070C00070C00071C000398000398
0003F80001F00001F00000E00000E00000E00013127F9116>I<FF3FCFE0FF3FCFE01C0F07801C
0F03001C1F03000E1B06000E1B86000E1B86000E318E000731CC000731CC000760CC0003E0F800
03E0F80003E0F80001C0700001C0700001C070001B127F911E>I<7F8FF07F8FF00F0780070600
038E0001DC0001D80000F00000700000780000F80001DC00038E00030E000607000F0380FF8FF8
FF8FF81512809116>I<FF0FE0FF0FE01C07801C03000E06000E06000E0600070C00070C00071C
0003980003980003F80001F00001F00000E00000E00000E00000C00000C00000C000F18000F180
00C700007E00003C0000131A7F9116>I<7FFC7FFC7838707060F060E061C063C00380070C0F0C
0E0C1C1C3C1838187078FFF8FFF80E127F9112>I E /Fl 54 123 df<00F8F001F8F003F8F007
80F00700F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F
00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F
00F00F00F00F00F01420809F18>13 D<007000E001C00380078007000E001E001E003C003C003C
0078007800780078007000F000F000F000F000F000F000F000F000F000F000F000F00070007800
7800780078003C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40
D<E000700038001C001E000E0007000780078003C003C003C001E001E001E001E000E000F000F0
00F000F000F000F000F000F000F000F000F000F000E001E001E001E001E003C003C003C0078007
8007000E001E001C0038007000E0000C2E7DA112>I<0006000000060000000600000006000000
060000000600000006000000060000000600000006000000060000000600000006000000060000
00060000FFFFFFF0FFFFFFF0000600000006000000060000000600000006000000060000000600
0000060000000600000006000000060000000600000006000000060000000600001C207D9A23>
43 D<787878781830306060E0050A7D830D>I<FFC0FFC0FFC00A037F8B0F>I<03F00007F8001F
FE001E1E003C0F00380700780780780780700380F003C0F003C0F003C0F003C0F003C0F003C0F0
03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07807807807807807803807003C0F001E
1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C0
03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0FFFEFF
FEFFFE0F1F7C9E17>I<FFFFFFF0FFFFFFF0000000000000000000000000000000000000000000
0000000000000000000000FFFFFFF0FFFFFFF01C0C7D9023>61 D<001F0000001F0000003F8000
003B8000003B8000007BC0000073C0000071C00000F1E00000E1E00000E0E00001E0F00001E0F0
0001C0F00003C0780003C078000380780007803C0007803C0007003C000FFFFE000FFFFE000FFF
FE001E000F001E000F003C000F803C0007803C000780780007C0780003C0780003C0F00003E01B
207F9F1E>65 D<FFF800FFFF00FFFF80F00FC0F003E0F001E0F000F0F000F0F000F0F000F0F000
F0F001E0F003C0F01F80FFFF00FFFF00FFFF80F007E0F001E0F000F0F00078F00078F00078F000
78F00078F00078F000F0F001F0F007E0FFFFC0FFFF80FFFC0015207B9F1E>I<001FC000FFF801
FFFC03E03C07800C0F00001E00003E00003C00007C0000780000780000780000F00000F00000F0
0000F00000F00000F00000F00000F000007800007800007800007C00003C00003E00001E00000F
000207800E03E03E01FFFC00FFF0001FC017227DA01D>I<FFFC00FFFF80FFFFC0F007E0F001F0
F000F8F00078F0003CF0003CF0001EF0001EF0000EF0000FF0000FF0000FF0000FF0000FF0000F
F0000FF0000FF0000FF0001EF0001EF0001EF0003CF0007CF000F8F001F0F007E0FFFFC0FFFF80
FFFC0018207B9F21>I<FFFFC0FFFFC0FFFFC0F00000F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000FFFF80FFFF80FFFF80F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000FFFFE0FFFFE0FFFFE013207B9F1B>I<FFFFC0FF
FFC0FFFFC0F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000FF
FF00FFFF00FFFF00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F0000012207B9F1A>I<001FE000FFF801FFFE03E03E07800E0F0000
1E00003E00003C00007C0000780000780000780000F00000F00000F00000F00000F00000F00000
F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E001E0F001E07801E03E03E01FFFE
00FFF8001FC017227DA01E>I<F00078F00078F00078F00078F00078F00078F00078F00078F000
78F00078F00078F00078F00078F00078FFFFF8FFFFF8FFFFF8F00078F00078F00078F00078F000
78F00078F00078F00078F00078F00078F00078F00078F00078F00078F0007815207B9F20>I<F0
F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207C9F0D>I<F0
0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000FFFF80FFFF80FFFF8011207B9F19>76 D<F80001F8FC0003F8FC0003F8F400
0378F6000778F6000778F6000778F7000F78F3000E78F3000E78F3801E78F3801E78F1801C78F1
C03C78F1C03C78F1C03C78F0C03878F0E07878F0E07878F0607078F070F078F070F078F030E078
F039E078F039E078F019C078F019C078F019C078F00F8078F00F8078F00F8078F00000781D207B
9F28>I<FC0078FE0078FE0078F60078F70078F70078F38078F38078F38078F3C078F1C078F1E0
78F1E078F0E078F0F078F07078F07078F07878F03878F03C78F03C78F01C78F01E78F00E78F00E
78F00E78F00778F00778F00378F003F8F003F8F001F815207B9F20>I<003F000000FFC00003FF
F00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800078078000780F0
0003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C0
7800078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FFF0
0000FFC000003F00001A227DA021>I<FFF800FFFF00FFFF80F00FC0F003E0F001E0F000F0F000
F0F000F0F000F0F000F0F000F0F000F0F001E0F003E0F00FC0FFFF80FFFF00FFF800F00000F000
00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0000014207B9F1D
>I<FFF800FFFF00FFFF80F007C0F003E0F001E0F000F0F000F0F000F0F000F0F000F0F001E0F0
03E0F007C0FFFF80FFFF00FFF800F03C00F01E00F01E00F00F00F00F00F00780F00780F003C0F0
01C0F001E0F000F0F000F0F00078F00078F0003C16207B9F1D>82 D<01FC0007FF800FFFC01F03
C03C00C03C00007800007800007800007800007800007C00003C00003F00001FE0000FFC0007FE
0001FF00003F800007C00003C00003E00001E00001E00001E00001E00001E00001C0C003C0F007
C0FC0F807FFF001FFE0003F80013227EA019>I<FFFFFFC0FFFFFFC0FFFFFFC0001E0000001E00
00001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E
0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E000000
1E0000001E0000001E0000001E0000001E0000001E0000001E0000001E00001A207E9F1F>I<F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F07801E078
01E03C03C03C03C01F0F800FFF0007FE0001F80014217B9F1F>I<F00001E0F00001E0780003C0
780003C0780003C03C0007803C0007803C0007801E000F001E000F001F000F000F001E000F001E
0007801C0007803C0007803C0003C0380003C0780003C0780001E0700001E0F00001E0F00000F0
E00000F1E00000F1E0000071C000007BC000003B8000003B8000003F8000001F0000001F00001B
207F9F1E>I<F0007C000FF0007E000FF0007E000F78006E000E7800EE001E7800E7001E7800E7
001E3C00E7003C3C01E7803C3C01C7803C3C01C3803C1E01C380781E03C3C0781E0383C0780E03
81C0700F0381C0F00F0781E0F00F0701E0F0070700E0E0078700E1E0078F00F1E0078E00F1E003
8E0071C0038E0071C003CE0073C001DC007B8001DC003B8001DC003B8001DC003B8000F8003F00
00F8001F0000F8001F0028207F9F2B>I<780007807C000F003E001F001E001E000F003C000F80
7C000780780003C0F00003E1F00001F1E00000F3C000007FC000007F8000003F0000001F000000
1E0000003F0000007F8000007FC00000F3C00001F1E00001E0F00003C0F80007C0780007803C00
0F003E001F001E001E000F003C000F807C0007C0780003C0F00003E01B207F9F1E>I<F80000F8
7C0001F03C0001E03E0003E01F0003C00F0007800F800F8007C00F0003C01F0003E01E0001F03C
0000F07C00007878000078F000003CF000001DE000001FE000000FC00000078000000780000007
800000078000000780000007800000078000000780000007800000078000000780000007800000
078000000780001D20809F1E>I<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF8
0FF00FF00FF00FF83F7FFF3FEF1F8F10147E9316>97 D<F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F1F000F7FC00FFFE00FC3E00F80F00F00F00F0
0780F00780F00780F00780F00780F00780F00780F00F00F00F00F81F00FC3E00FFFC00F7F800F1
E00011207D9F17>I<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F00078007800
3C013E0F1FFF0FFE03F010147E9314>I<00078000078000078000078000078000078000078000
078000078000078000078000078007C7800FF7801FFF803E1F807C0780780780F80780F00780F0
0780F00780F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C78011207E9F
17>I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000
F000007000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01FE03FE0780
07000F000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F
000F000F000F000F000F000F000F000F000F20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00
780F00780F00780F00780F00780F003C1E003E3E001FFC003FF80033E0003000003800003FFE00
3FFF801FFFC03FFFE07803F0F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F
9317>I<F0F0F0F00000000000000000F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207D
9F0B>105 D<F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F01F00F01E00F03C00F07800F0F000F1E000F3C000F78000FFC000FFC000FFE000F9F000
F8F000F0F800F07C00F07C00F03E00F01E00F01F00F00F8011207D9F16>107
D<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207D9F0B>
I<F0FC07E0F3FE1FF0F7FF3FF8FE0FF07CF807C03CF807C03CF007803CF007803CF007803CF007
803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF0
07803C1E147D9327>I<F1F8F3FCF7FEFC1FF80FF80FF00FF00FF00FF00FF00FF00FF00FF00FF0
0FF00FF00FF00FF00FF00F10147D9317>I<01F80007FE001FFF803F0FC03C03C07801E07801E0
F000F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F800
14147F9317>I<F1F000F7FC00FFFE00FC3E00F81F00F00F00F00F80F00780F00780F00780F007
80F00780F00780F00F00F00F00F81F00FC3E00FFFC00F7F800F1E000F00000F00000F00000F000
00F00000F00000F00000F00000F00000111D7D9317>I<03C7800FF7801FFF803E1F807C0F8078
0780780780F00780F00780F00780F00780F00780F00780F807807807807C0F803E1F801FFF800F
F78007C780000780000780000780000780000780000780000780000780000780111D7E9317>I<
F0E0F3E0F7E0FF00FC00FC00F800F800F000F000F000F000F000F000F000F000F000F000F000F0
000B147D9310>I<07F01FFC3FFC780C7800780078007C003FC01FF00FF803F8007C003C003CC0
3CF07CFFF87FF00FC00E147F9311>I<1E001E001E001E001E001E00FFF0FFF0FFF01E001E001E
001E001E001E001E001E001E001E001E001E001E001E201FF00FF007C00C1A7F9910>I<F00FF0
0FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF03FFFFF7FEF3F0F1014
7D9317>I<F003C0F003C07803807807807807803C0F003C0F003C0F001E0E001E1E001E1E000F
1C000F3C000F3C0007380007380007B80003F00003F00001E00012147F9315>I<F01F00F0F01F
80F0F01F80F0781B81E0783B81E0783BC1E07839C1E03C31C3C03C71C3C03C71E3C01C70E3801E
60E7801E60E7801EE0E7800EE077000EC077000EC0770007C07E0007C03E0007803E001C147F93
1F>I<F003C0F003C07807807807807C07803C0F003C0F001E0F001E1E000E1E000F1C000F1C00
073C0007380003B80003B80003B00001F00001F00000E00000E00001C00001C00003C000038000
0780007F00007E00007C0000121D7F9315>121 D<7FFF7FFF7FFF003E003C007800F800F001E0
03E007C007800F001F001E003C007C00FFFFFFFFFFFF10147F9314>I E
/Fm 16 118 df<060C183030606060C0C0C0C0C0C0C0C0C0C06060603030180C06071A7F920C>
40 D<C0603018180C0C0C060606060606060606060C0C0C18183060C0071A7E920C>I<1F003F80
60C04040C060C060C060C060C060C060C060C06060C060C03F801F000B107F8F0F>48
D<18007800F80098001800180018001800180018001800180018001800FF80FF8009107E8F0F>
I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FFC0FFC00B107F8F0F>
I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F801F000B107F8F0F>
I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001FE01FE00B107F8F0F>
I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F803F000B107F8F0F>
I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F801F000B107F8F0F>
I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C001C0008000B117E90
0F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060C03F801F000B107F8F
0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C071807F003E000B107F8F
0F>I<07F807F800E000E000E000E000E000E000E000E000E000E060E0F0E0F1E07FC03F000D11
809010>74 D<1F003F8060C0FFC0FFC0C000C000E0C070C03F801F000A0B7F8A0E>101
D<F7C0FFE03860306030603060306030603060FCF8FCF80D0B7F8A11>110
D<F1E0F1E030603060306030603060306030E03FF81F780D0B7F8A11>117
D E /Fn 5 64 df<C0000CE0001C7000383800701C00E00E01C006018007038003870001CE0000
FC0000780000780000FC0001CE000387000703800601800E01C01C00E0380070700038E0001CC0
000C16187A9623>2 D<C000000000E000000000700000000038000000001C000000000E000000
000700000000038000000001C000000000E000000000700000000038000000001C000000000E00
0000000700000000038000000001C000000000E000000000700000000038000000001C00000000
0E000000000700000000038000000001C000000000E0000000007004000000380C0000001C0C00
00000E0C0000000718000000039800000001D800000000F8000000007800000003F80000001FFC
0000003C0C26267D9C2D>38 D<000000000C000000001C0000000038000000007000000000E000
000001C0000000038000000007000000000E000000001C0000000038000000007000000000E000
000001C0000000038000000007000000000E000000001C0000000038000000007000000000E000
000001C0000000038000000007000000000E000000001C0000008038000000C070000000C0E000
0000C1C0000000638000000067000000006E000000007C0000000078000000007F00000000FFE0
000000C0F000000026267D9C2D>46 D<FFFFFFF0FFFFFFF0000600000006000000060000000600
000006000000060000000600000006000000060000000600000006000000060000000600000006
000000060000000600000006000000060000000600000006000000060000000600000006000000
06000000060000000600001C1C7D9B23>62 D<0006000000060000000600000006000000060000
000600000006000000060000000600000006000000060000000600000006000000060000000600
000006000000060000000600000006000000060000000600000006000000060000000600000006
000000060000FFFFFFF0FFFFFFF01C1C7D9B23>I E /Fo 32 91 df<3078F8787005057C840D>
46 D<007E0001FF0003C3800701C00E01C00E01E01C00E01C01E03C01E03C01E07801E07801E0
7801E07801E07801E0F003C0F003C0F003C0F003C0F003C0F00380F00780E00780E00700E00F00
E00E00701E00701C003878003FF0000FC000131F7C9D17>48 D<000C001C00FC0FF80F38003800
380038003800700070007000700070007000E000E000E000E000E000E001C001C001C001C001C0
01C003C07FFEFFFE0F1E7C9D17>I<003F0000FFC001C1E00300F00600780600780F007C0F807C
0F807C0F00780600780000F80000F00001F00001E00003C0000780000F00001C00003800007000
00C0000180000300600600601C00C03800C07FFFC0FFFF80FFFF80161E7E9D17>I<007F0001FF
C00383E00701E00700F00F00F00F01F00F01F00001E00001E00003C0000780000F0000FE0000F8
00000E000007000007800003C00003C00003C03007C07807C0F807C0F807C0F00F80C00F00601E
00383C001FF80007E000141F7D9D17>I<0001C00003C00003C0000780000F80001B80003B8000
738000638000C7000187000307000707000E07000C0700180E00300E00600E00E00E00FFFFF0FF
FFF0001C00001C00001C00001C00001C00001C00003C0003FFC003FFC0141E7D9D17>I<030070
03FFF003FFE003FF8003FE000600000600000600000600000600000600000C7C000DFF000F8780
0E03800C01C00C01C00001E00001E00001E00001E07003C0F003C0F003C0F00380C00780E00700
600E00383C001FF0000FC000141F7D9D17>I<000F80007FC000F0E001C0E00381E00701E00E00
C01E00001C00003C000038000079FC007BFE007E07007C0380F80380F803C0F003C0F003C0F003
C0F003C0F00780E00780E00780E00700E00F00700E00701C003838001FF00007C000131F7C9D17
>I<3000003FFFE07FFFE07FFFC0600180E00300C00600C00C0000180000180000300000600000
C00001C0000180000380000300000700000600000E00000E00001E00001C00001C00003C00003C
00003C0000780000780000780000300000131F799D17>I<003F0000FF8001C1E00380E0070070
0E00700E00700E00700E00E00F00E00F81C00FC38007F70003FC0001FC0003FF000F3F801C0F80
3807C03803C07001C07001C0E001C0E001C0E001C0E003807003007007003C1C001FF80007E000
141F7D9D17>I<007E0001FF000383800701C00E01C01E01C01C01E03C01E03C01E07801E07801
E07801E07801E07803E07803E03807C03807C01C0FC01FFBC00FE380000780000780000700000E
00600E00F01C00F03800E07000E0E0007FC0003F0000131F7C9D17>I<00001800000038000000
38000000780000007C000000FC000000FC000001BC000001BC0000033C0000033E0000061E0000
061E00000C1E00000C1E0000181E0000181F0000300F0000300F0000600F00007FFF0000FFFF00
00C00F000180078001800780030007800300078006000780060007801F0007C0FFC07FFCFFC07F
FC1E207E9F22>65 D<0003F808000FFE18003F0738007801F801F000F803C000F007C000700780
00700F0000701F0000701E0000703E0000603C0000607C0000007C0000007C0000007C000000F8
000000F8000000F8000000F8000000F80000C0780000C0780001807C0001803C0003003C000300
1E0006001E000C000F80180007E0700001FFC000007F00001D217B9F21>67
D<07FFFF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E0078001E0078001E0078
001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E01E0003E01E0003E01
E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003C003C00F8007C03F00
7FFFFC00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C00780078003800780038
00780038007800380078003800780C3000F0183000F0180000F0180000F0380000FFF80000FFF8
0001E0700001E0300001E0300001E0301801E0303001E0003003C0003003C0006003C0006003C0
00E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07FFFFF807FFFFF8003C
00F8003C0078007800380078003800780038007800380078003800780C3000F0183000F0180000
F0180000F0380000FFF80000FFF80001E0700001E0300001E0300001E0300001E0300001E00000
03C0000003C0000003C0000003C0000003C0000003C0000007C000007FFE0000FFFE00001D1F7E
9E1E>I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C00078078000380F0000
381F0000381E0000383E0000303E0000307C0000007C0000007C0000007C000000F8000000F800
0000F8007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C03C0003C01E0003C00F
0007C007800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FFC007FFC7FFC0003C00
7800003C007800007800F000007800F000007800F000007800F000007800F000007800F00000F0
01E00000F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E003C00001E003C00001
E003C00001E003C00001E003C00001E003C00003C007800003C007800003C007800003C0078000
03C007800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22>I<07FFE007FFE000
3C00003C0000780000780000780000780000780000780000F00000F00000F00000F00000F00000
F00001E00001E00001E00001E00001E00001E00003C00003C00003C00003C00003C00003C00007
C000FFFC00FFFC00131F7F9E10>I<07FFF00007FFF000003C0000003C00000078000000780000
0078000000780000007800000078000000F0000000F0000000F0000000F0000000F0000000F000
0001E0000001E0000001E0000001E0018001E0018001E0030003C0030003C0030003C0070003C0
060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>76 D<07FC0000FFC007FC00
01FFC0003E0001F800003E00037800006E0003F000006E0006F000006E0006F000006E000CF000
0067000CF00000670018F00000C70019E00000C70031E00000C70031E00000C70061E00000C380
61E00000C380C1E000018380C3C00001838183C00001838183C0000181C303C0000181C303C000
0181C603C0000301C60780000301CC0780000301CC0780000300F80780000300F80780000700F0
0780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07FC03FFC007FC03FFC000
3E007C00003E003800006F003000006F003000006F003000006780300000678030000063C03000
00C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F060000180F0C000018078C0
00018078C00001803CC00001803CC00001803CC00003001F800003001F800003000F800003000F
800003000F800007000780000F800700007FF0030000FFF0030000221F7E9E22>I<0003F80000
1FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F0000F01F0000F01E0000F8
3E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F80001F0F80001F0F80001
F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E001E003C000F00780007C1
F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0003C01F0007800F00078
00F8007800F8007800F8007800F8007800F800F001F000F001F000F001E000F003C000F00F8000
FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C0000003C0000003C00000
03C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F>I<07FFFC0007FFFF00
003C07C0003C03E0007801E0007801F0007801F0007801F0007801F0007801E000F003E000F003
C000F0078000F01F0000FFFC0000FFF00001E0380001E03C0001E01C0001E01C0001E01E0001E0
1E0003C03E0003C03E0003C03E0003C03E0003C03E0603C03E0607C03E0C7FFC1F18FFFC0FF800
0003E01F207E9E21>82 D<003F04007FCC01E0FC03807C03003C0700380600180E00180E00180E
00180E00000F00000F00000FE00007FE0003FF8001FFC0007FE00007E00001E00000E00000F000
00F06000E06000E06000E06000E07001C0700180F80380FE0F00C7FC0081F80016217D9F19>I<
1FFFFFF81FFFFFF81E03C0F83803C0383807803830078038700780186007801860078038600780
30C00F0030000F0000000F0000000F0000000F0000000F0000001E0000001E0000001E0000001E
0000001E0000001E0000003C0000003C0000003C0000003C0000003C0000003C0000007C00001F
FFF0003FFFF0001D1F7B9E21>I<FFFC7FF8FFFC7FF807800F80078007000F0006000F0006000F
0006000F0006000F0006000F0006001E000C001E000C001E000C001E000C001E000C001E000C00
3C0018003C0018003C0018003C0018003C0018003C001800380030003800300038003000380060
003C00C0001C00C0001E0180000F07000007FC000001F800001D20799E22>I<FFF07FF83FF0FF
F07FF83FF01F000F800F800F000F8007000F000F8006000F000F8006000F001F800C000F001F80
0C000F00378018000F00378018000F0063C03000078063C030000780C3C060000780C3C0600007
8183C0E000078183C0C000078383C0C000078303E1800007C701E1800003C601E3000003C601E3
000003CC01E6000003CC01E6000003D801EC000003D801EC000003F001F8000001F000F8000001
E000F8000001E000F0000001C000F0000001C000E0000001C000E000002C207A9E2F>87
D<03FFC1FFC003FFC1FFC0003E007C00001E007000001E006000000F00C000000F01C000000F83
800000078300000007C600000003CC00000003FC00000001F800000001F000000000F000000000
F800000001F800000003FC000000033C000000063C0000000C1E0000001C1E000000381F000000
300F000000600F800000C007800001C007C000038003C0000FC003E000FFF01FFE00FFE01FFE00
221F7F9E22>I<7FF803FF80FFF803FF0007C000F80007C000E00003C000C00003E001C00001E0
03800001F003000000F006000000F80E000000F80C00000078180000007C300000003C70000000
3E600000001EC00000001F800000000F800000000F000000000F000000000F000000000E000000
001E000000001E000000001E000000001E000000001E000000001C000000003C00000003FFE000
0007FFE00000211F7B9E22>I<03FFFFC003FFFFC003F0078007C00F0007001F0007001E000E00
3C000C0078000C00F8000C00F0001801E0000003C00000078000000F8000000F0000001E000000
3C0000007C00000078030000F0030001E0030003E0060003C00600078006000F000E001F001C00
1E001C003C007C007801FC00FFFFF800FFFFF8001A1F7D9E1C>I E /Fp
38 121 df<387CFEFEFE7C3807077C860F>46 D<00700000F0000FF000FFF000F3F00003F00003
F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003
F00003F00003F00003F00003F00003F00003F00003F000FFFF80FFFF80111D7C9C1A>49
D<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE0FE0FE07E07C07E03807E0000FE0000FC0000F
C0001F80001F00003E0000780000F00000E00001C0600380600700600C00E01FFFE01FFFC03FFF
C07FFFC0FFFFC0FFFFC0131D7D9C1A>I<01FC0007FF000E0F801E0FC03F07E03F87E03F87E03F
87E01F0FC00E0FC0000F80001F0001FC0001FC00000F800007C00003E00003F03C03F87E03F8FF
03F8FF03F8FF03F8FF03F07E03F07C07E03C0FC00FFF8003FC00151D7E9C1A>I<0001C00003C0
0007C00007C0000FC0001FC0003FC00077C00067C000C7C00187C00387C00707C00E07C00C07C0
1807C03807C07007C0E007C0FFFFFEFFFFFE000FC0000FC0000FC0000FC0000FC0000FC001FFFE
01FFFE171D7F9C1A>I<3803803FFF803FFF003FFE003FFC003FF0003F80003000003000003000
0030000033F8003FFE003C1F00380F80300FC0000FC0000FE0000FE0780FE0FC0FE0FE0FE0FE0F
E0FC0FC0780FC0601F80383F001FFC0007F000131D7D9C1A>I<003F0001FFC007E0E00F81E01F
03F01E03F03E03F07C03F07C01E07C0000FC1000FCFF00FDFFC0FF03E0FE01F0FE01F0FC01F8FC
01F8FC01F8FC01F87C01F87C01F87C01F83C01F03E01F01E03E00F07C007FF8001FE00151D7E9C
1A>I<6000007FFFF87FFFF87FFFF07FFFE07FFFE0FFFFC0E00380C00700C00E00C00C00001C00
00380000780000780000F00000F00000F00001F00001F00001F00003F00003F00003F00003F000
03F00003F00003F00003F00001E000151E7D9D1A>I<01FC0007FF000F07801C03C01C01E03C01
E03C01E03E01E03F81E03FE3C03FF3801FFF000FFF000FFFC007FFE01FFFE03C7FF0781FF87807
F8F001F8F000F8F00078F00078F000707800707C00E03E03C00FFF8003FC00151D7E9C1A>I<01
FC000FFF001F07803E03C07C03E07C01E0FC01F0FC01F0FC01F0FC01F8FC01F8FC01F8FC01F87C
03F87C03F83E07F81FFDF807F9F80041F80001F03C01F07E01F07E03E07E03E07E07C03C078038
1F001FFC0007F000151D7E9C1A>I<387CFEFEFE7C38000000000000387CFEFEFE7C3807147C93
0F>I<07F8001FFE00381F80780F80FC0FC0FC0FC0FC0FC0780FC0301F80001F00003E00007C00
00700000E00000E00000C00000C00000C00000C00000C00000C000000000000000000000000000
01C00007F00007F00007F00007F00007F00001C00012207D9F19>63 D<0000E000000000E00000
0001F000000001F000000001F000000003F800000003F800000007FC00000007FC0000000FFE00
00000CFE0000000CFE000000187F000000187F000000307F800000303F800000703FC00000601F
C00000601FC00000C01FE00000C00FE00001FFFFF00001FFFFF000018007F000030003F8000300
03F800060003FC00060001FC000E0001FE00FFE01FFFE0FFE01FFFE0231F7E9E28>65
D<FFFFFE00FFFFFFC007E007E007E003F007E001F807E001FC07E001FC07E001FC07E001FC07E0
01FC07E001F807E003F807E003F007E00FE007FFFF8007FFFFE007E003F007E001F807E001FC07
E000FC07E000FE07E000FE07E000FE07E000FE07E000FE07E000FC07E001FC07E001F807E007F0
FFFFFFE0FFFFFF001F1F7E9E25>I<FFFFFE0000FFFFFFC00007F007F00007F001F80007F000FC
0007F0007E0007F0003F0007F0003F0007F0001F8007F0001F8007F0001F8007F0001FC007F000
1FC007F0001FC007F0001FC007F0001FC007F0001FC007F0001FC007F0001FC007F0001FC007F0
001F8007F0001F8007F0001F8007F0003F0007F0003F0007F0007E0007F000FC0007F001F80007
F007F000FFFFFFC000FFFFFE0000221F7E9E28>68 D<FFFFFFE0FFFFFFE007F007E007F001E007
F000E007F0006007F0007007F0003007F0003007F0603007F0603007F0600007F0600007F1E000
07FFE00007FFE00007F1E00007F0600007F0600007F0600C07F0600C07F0000C07F0001807F000
1807F0001807F0003807F0007807F000F807F003F0FFFFFFF0FFFFFFF01E1F7E9E22>I<FFFF80
FFFF8007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000
07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000
07F00007F000FFFF80FFFF80111F7F9E14>73 D<FFF000007FF8FFF80000FFF807F80000FF0006
FC0001BF0006FC0001BF0006FC0001BF00067E00033F00067E00033F00063F00063F00063F0006
3F00061F800C3F00061F800C3F00061F800C3F00060FC0183F00060FC0183F000607E0303F0006
07E0303F000603F0603F000603F0603F000603F0603F000601F8C03F000601F8C03F000600FD80
3F000600FD803F0006007F003F0006007F003F0006003E003F0006003E003F0006003E003F00FF
F01C07FFF8FFF01C07FFF82D1F7E9E32>77 D<FFF800FFF0FFF800FFF007FC00060007FE000600
07FF00060006FF800600067F800600063FC00600061FE00600060FF00600060FF806000607FC06
000603FC06000601FE06000600FF060006007F860006007FC60006003FE60006001FE60006000F
F600060007FE00060003FE00060003FE00060001FE00060000FE000600007E000600003E000600
001E000600001E00FFF0000E00FFF0000600241F7E9E29>I<001FF80000FFFF0001F81F8007E0
07E00FC003F01F8001F81F0000F83F0000FC7F0000FE7E00007E7E00007EFE00007FFE00007FFE
00007FFE00007FFE00007FFE00007FFE00007FFE00007FFE00007F7E00007E7F0000FE7F0000FE
3F0000FC3F8001FC1F8001F80FC003F007E007E001F81F8000FFFF00001FF800201F7D9E27>I<
FFFFFE00FFFFFF8007F00FE007F003F007F001F807F001F807F001FC07F001FC07F001FC07F001
FC07F001FC07F001F807F001F807F003F007F00FE007FFFF8007FFFE0007F0000007F0000007F0
000007F0000007F0000007F0000007F0000007F0000007F0000007F0000007F0000007F00000FF
FF8000FFFF80001E1F7E9E24>I<03FC080FFF381E03F83C00F8780078780038F80038F80018FC
0018FC0000FF0000FFF8007FFF007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC0000FC00
00FCC0007CC0007CC0007CE00078E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83
D<7FFFFFFC7FFFFFFC7C0FE07C700FE01C600FE00C600FE00CE00FE00EC00FE006C00FE006C00F
E006C00FE006000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE00000
0FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE00007FFFFC0
07FFFFC01F1E7E9D24>I<FFFF81FFE0FFFF81FFE007F0000C0007F0000C0007F0000C0007F000
0C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0
000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007
F0000C0007F0000C0007F0001C0003F000180003F800380001F800300000FC007000007F03E000
001FFF80000003FC0000231F7E9E28>I<FFFF1FFFC0FFE0FFFF1FFFC0FFE00FE001FC000E0007
F001FC000C0007F001FE000C0007F800FE000C0003F800FE00180003F800FF00180003FC00FF00
380001FC01FF80300001FC01BF80300000FE01BF80600000FE031FC0600000FF031FC0E000007F
031FE0C000007F060FE0C000003F860FE18000003F8E0FF18000003FCC07F18000001FCC07F300
00001FD803FB0000001FF803FF0000000FF803FE0000000FF001FE00000007F001FC00000007E0
00FC00000007E000FC00000003E000F800000003C0007800000001C00070000000018000300000
331F7F9E36>87 D<3FFFFF803FFFFF803F807F803E00FF003801FE003801FE007003FC007003F8
006007F800600FF000600FF000001FE000003FC000003FC000007F8000007F000000FF000001FE
000001FE018003FC018007F8018007F801800FF003800FE003801FE003003FC007003FC00F007F
801F00FF007F00FFFFFF00FFFFFF00191F7D9E20>90 D<07FC001FFF003F0F803F07C03F03E03F
03E00C03E00003E001FFE00FFFE03F83E07E03E07C03E0F803E0F803E0F803E0FC07E07E0DE03F
F9FE07E07E17147F9319>97 D<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00F8FFFF
F8FFFFF8FC0000FC0000FC00007C00007E00003E00183F00380F807007FFE000FF8015147F9318
>101 D<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800FFF
0019FC001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C70003EF0001EF0001EF000
1E78003C78003C3F01F80FFFE001FF00171E7F931A>103 D<1C003F007F007F007F003F001C00
000000000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F
001F001F001F001F00FFE0FFE00B217EA00E>105 D<FF00FF001F001F001F001F001F001F001F
001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F00
1F001F00FFE0FFE00B207E9F0E>108 D<FE0FE03F80FE3FF0FFC01E70F9C3E01EC07F01F01F80
7E01F01F807E01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F01F
007C01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F0FFE3FF8FFEFFE3FF8FFE
27147D932C>I<FE0FC0FE3FE01E61F01EC0F81F80F81F00F81F00F81F00F81F00F81F00F81F00
F81F00F81F00F81F00F81F00F81F00F81F00F81F00F8FFE3FFFFE3FF18147D931D>I<01FF0007
FFC01F83F03E00F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C
007C3E00F83E00F81F83F007FFC001FF0017147F931A>I<FF1FC0FF7FF01FE1F81F80FC1F007E
1F007E1F003E1F003F1F003F1F003F1F003F1F003F1F003F1F003E1F007E1F007C1F80FC1FC1F8
1F7FE01F1F801F00001F00001F00001F00001F00001F00001F0000FFE000FFE000181D7E931D>
I<FE3E00FE7F801ECFC01F8FC01F8FC01F8FC01F03001F00001F00001F00001F00001F00001F00
001F00001F00001F00001F00001F0000FFF000FFF00012147E9316>114
D<01800180018003800380038007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F80
0F800F800F860F860F860F860F8607CC03F801F00F1D7F9C14>116 D<FFE1FF00FFE1FF000F80
700007C0E00007E0C00003E1800001F3800000FF0000007E0000003E0000003F0000007F800000
6F800000C7C0000183E0000381F0000701F8000E00FC00FF81FF80FF81FF8019147F931C>120
D E /Fq 81 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F870000000000070F8F8F870051C
779B18>33 D<3803007C07807C0F80FE0F80EE0F00EE1F00EE1F00EE1E00EE3E00FE3E007C3C00
7C7C00387C0000780000F80000F80000F00001F00001F00001E00003E00003E00003C00007C000
07C3800787C00F87C00F8FE00F0EE01F0EE01F0EE01E0EE03E0FE03E07C03C07C018038013247E
9F18>37 D<03C00007E0000FF0001E78001C38001C38001C38001C38001C7BF01CF3F01EE3F00F
E7800FC7000F87000F0F001F0E003F8E007B9E0073DC00F1DC00E1FC00E0F800E07870F0F87071
FE707FFFF03FCFE01F03C0141C7F9B18>I<387C7E7E3E0E0E0E1E3C78F8F0C0070E789B18>I<00
7000F001F003C007800F001E001C003C003800780070007000F000E000E000E000E000E000E000
E000E000F00070007000780038003C001C001E000F00078003C001F000F000700C24799F18>I<
6000F00078003C001E000F000780038003C001C001E000E000E000F00070007000700070007000
700070007000F000E000E001E001C003C0038007800F001E003C007800F00060000C247C9F18>
I<01C00001C00001C00001C000C1C180F1C780F9CF807FFF001FFC0007F00007F0001FFC007FFF
00F9CF80F1C780C1C18001C00001C00001C00001C00011147D9718>I<00600000F00000F00000
F00000F00000F00000F00000F0007FFFC0FFFFE0FFFFE07FFFC000F00000F00000F00000F00000
F00000F00000F00000600013147E9718>I<3C7E7F7F7F3F0F0E1E7CF870080C788518>I<7FFF00
FFFF80FFFF807FFF0011047D8F18>I<78FCFCFCFC780606778518>I<000300000780000F80000F
80000F00001F00001F00003E00003E00003C00007C00007C0000F80000F80000F00001F00001F0
0003E00003E00007C00007C0000780000F80000F80001F00001F00001E00003E00003E00007C00
007C0000780000F80000F80000F0000060000011247D9F18>I<01F00007FC000FFE001F1F001C
07003803807803C07001C07001C0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E000E0E0
00E0F001E07001C07001C07803C03803801C07001F1F000FFE0007FC0001F000131C7E9B18>I<
01800380078007800F803F80FF80FB806380038003800380038003800380038003800380038003
80038003800380038003807FFCFFFE7FFC0F1C7B9B18>I<07F8001FFE003FFF007C0F807003C0
F001E0F000E0F000E0F000E00000E00000E00001E00001C00003C0000780000F00001E00003C00
00780000F00003E00007C0000F80001E00E03C00E07FFFE0FFFFE07FFFE0131C7E9B18>I<07F8
001FFE003FFF007C0F807803C07801C03001C00001C00003C0000780000F8003FF0003FE0003FF
00000F800003C00001C00001E00000E00000E0F000E0F001E0F001C0F003C07C0F803FFF001FFE
0007F800131C7E9B18>I<001F00003F00007F0000770000F70001E70001C70003C70007870007
07000E07001E07003C0700380700780700F00700FFFFF8FFFFF8FFFFF800070000070000070000
0700000700000700007FF000FFF8007FF0151C7F9B18>I<007E0001FF0007FF800FC3C01F03C0
3C03C0380180780000700000700000F3F800EFFE00FFFF00FE0F80F803C0F001C0F001E0F000E0
F000E0F000E07000E07001E07801C03C03C01E0F800FFF0007FE0003F800131C7E9B18>54
D<E00000FFFFE0FFFFE0FFFFE0E003C0E00780000F00000E00001E00001C00003C000078000070
0000700000F00000E00000E00001E00001C00001C00001C00003C0000380000380000380000380
00038000038000038000131D7E9C18>I<03F8000FFE001FFF003E0F803803807001C07001C070
01C07001C03803803C07801FFF0007FC000FFE001F1F003C07807001C0F001E0E000E0E000E0E0
00E0E000E07001C07803C03E0F801FFF000FFE0003F800131C7E9B18>I<3C7E7E7E7E3C000000
0000000000387C7E7E7E3E0E1E3C78F060071A789318>59 D<000300000F80001F80003F0000FE
0001FC0003F00007E0001FC0003F80007E0000FC0000FC00007E00003F80001FC00007E00003F0
0001FC0000FE00003F00001F80000F8000030011187D9918>I<7FFFC0FFFFE0FFFFE0FFFFE000
0000000000000000000000FFFFE0FFFFE0FFFFE07FFFC0130C7E9318>I<600000F80000FC0000
7E00003F80001FC00007E00003F00001FC0000FE00003F00001F80001F80003F0000FE0001FC00
03F00007E0001FC0003F80007E0000FC0000F8000060000011187D9918>I<00700000F80000F8
0000D80000D80001DC0001DC0001DC00018C00038E00038E00038E00038E000306000707000707
000707000707000FFF800FFF800FFF800E03800E03801C01C01C01C07F07F0FF8FF87F07F0151C
7F9B18>65 D<FFFC00FFFF00FFFF801C03C01C01C01C00E01C00E01C00E01C00E01C01E01C01C0
1C07C01FFF801FFF001FFFC01C03C01C00E01C00F01C00701C00701C00701C00701C00F01C00E0
1C03E0FFFFC0FFFF80FFFE00141C7F9B18>I<01FCE003FEE007FFE00F07E01E03E03C01E07800
E07000E07000E0F00000E00000E00000E00000E00000E00000E00000E00000E00000F000007000
E07000E07800E03C01E01E01C00F07C007FF8003FF0001FC00131C7E9B18>I<7FF800FFFE007F
FF001C0F801C03C01C03C01C01E01C00E01C00E01C00F01C00701C00701C00701C00701C00701C
00701C00701C00701C00F01C00E01C00E01C01E01C01C01C03C01C0F807FFF00FFFE007FF80014
1C7F9B18>I<FFFFF0FFFFF0FFFFF01C00701C00701C00701C00701C00001C00001C0E001C0E00
1C0E001FFE001FFE001FFE001C0E001C0E001C0E001C00001C00001C00381C00381C00381C0038
1C0038FFFFF8FFFFF8FFFFF8151C7F9B18>I<FFFFE0FFFFE0FFFFE01C00E01C00E01C00E01C00
E01C00001C00001C1C001C1C001C1C001FFC001FFC001FFC001C1C001C1C001C1C001C00001C00
001C00001C00001C00001C00001C0000FFC000FFC000FFC000131C7E9B18>I<01F9C007FFC00F
FFC01F0FC01C03C03C03C07801C07001C07001C0F00000E00000E00000E00000E00000E00000E0
0FF0E01FF0E00FF0F001C07001C07003C07803C03C03C01C07C01F0FC00FFFC007FDC001F9C014
1C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C0
1C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C01C0
1C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00FFFF807FFF0001C00001C00001C00001C0
0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0
0001C00001C00001C00001C00001C0007FFF00FFFF807FFF00111C7D9B18>I<01FFC003FFC001
FFC0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0000
0E00000E00000E00000E00000E00000E00600E00F00E00F01E00F03C007FFC003FF8001FE00012
1C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C07001C0F001C1E001C3C001C38001C7800
1CF0001DF0001DF8001FF8001FBC001F1C001E1E001E0E001C0F001C07001C07801C03801C03C0
1C01C07F03F0FF87F87F03F0151C7F9B18>I<7FE000FFE0007FE0000E00000E00000E00000E00
000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00
000E00700E00700E00700E00700E00707FFFF0FFFFF07FFFF0141C7F9B18>I<FC01F8FE03F8FE
03F83B06E03B06E03B06E03B06E03B8EE03B8EE0398CE0398CE039DCE039DCE039DCE038D8E038
D8E038F8E03870E03870E03800E03800E03800E03800E03800E03800E0FE03F8FE03F8FE03F815
1C7F9B18>I<7E07F0FF0FF87F07F01D81C01D81C01D81C01DC1C01CC1C01CC1C01CE1C01CE1C0
1CE1C01C61C01C71C01C71C01C31C01C39C01C39C01C39C01C19C01C19C01C1DC01C0DC01C0DC0
1C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF8003FFE007FFF00780F00700700F00780E003
80E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E003
80E00380E00380F00780700700780F007FFF003FFE000FF800111C7D9B18>I<FFFE00FFFF80FF
FFC01C03C01C01E01C00E01C00701C00701C00701C00701C00701C00E01C01E01C03C01FFFC01F
FF801FFE001C00001C00001C00001C00001C00001C00001C00001C0000FF8000FF8000FF800014
1C7F9B18>I<0FF8003FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380
E00380E00380E00380E00380E00380E00380E00380E00380E00380E1E380E1E380F0E78070F700
787F007FFF003FFE000FFC00001C00001E00000E00000F0000070000070011227D9B18>I<7FF8
00FFFE007FFF001C0F801C03801C03C01C01C01C01C01C01C01C03C01C03801C0F801FFF001FFE
001FFE001C0F001C07801C03801C03801C03801C03801C03801C039C1C039C1C039C7F03FCFF81
F87F00F0161C7F9B18>I<07F3801FFF803FFF807C1F80700780F00380E00380E00380E00000F0
00007800003F00001FF0000FFE0001FF00001F800003C00001E00000E00000E06000E0E000E0E0
01E0F001C0FC07C0FFFF80FFFF00E7FC00131C7E9B18>I<7FFFF8FFFFF8FFFFF8E07038E07038
E07038E07038007000007000007000007000007000007000007000007000007000007000007000
00700000700000700000700000700000700000700007FF0007FF0007FF00151C7F9B18>I<FF83
FEFF83FEFF83FE1C00701C00701C00701C00701C00701C00701C00701C00701C00701C00701C00
701C00701C00701C00701C00701C00701C00701C00701E00F00E00E00F01E007C7C003FF8001FF
00007C00171C809B18>I<FF07F8FF07F8FF07F81C01C01C01C01C01C01C01C00E03800E03800E
03800E03800F0780070700070700070700070700038E00038E00038E00038E00018C0001DC0001
DC0001DC0000D80000F80000F800007000151C7F9B18>I<FE03F8FE03F8FE03F8700070700070
7000703800E03800E03800E03800E03800E038F8E038F8E039DCE039DCE019DCC019DCC019DCC0
198CC01D8DC01D8DC01D8DC01D8DC00D8D800D05800F07800F07800E0380151C7F9B18>I<7F8F
E07F9FE07F8FE00E07000F0700070E00078E00039C0003DC0001F80001F80000F00000F0000070
0000F00000F80001F80001DC00039E00038E00070F000707000E07800E03801E03C07F07F0FF8F
F87F07F0151C7F9B18>I<FF07F8FF07F8FF07F81C01C01E03C00E03800F078007070007070003
8E00038E0001DC0001DC0001FC0000F80000F80000700000700000700000700000700000700000
700000700000700001FC0003FE0001FC00151C7F9B18>I<3FFFE07FFFE07FFFE07001C07003C0
700780700700000F00001E00001C00003C0000780000700000F00001E00001C00003C000078000
0700000F00001E00E01C00E03C00E07800E07000E0FFFFE0FFFFE0FFFFE0131C7E9B18>I<FFF8
FFF8FFF8E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E0
00E000E000E000E000E000E000E000E000E000E000E000E000FFF8FFF8FFF80D24779F18>I<FF
F8FFF8FFF800380038003800380038003800380038003800380038003800380038003800380038
0038003800380038003800380038003800380038003800380038FFF8FFF8FFF80D247F9F18>93
D<7FFF00FFFF80FFFF807FFF0011047D7F18>95 D<1FE0003FF8007FFC00783E00300F00000700
00070001FF000FFF003FFF007F0700780700F00700E00700E00700F00F00783F007FFFF03FFBF0
0FE1F014147D9318>97 D<7E0000FE00007E00000E00000E00000E00000E00000E00000E3E000E
FF800FFFC00FE3E00F80F00F00700F00780E00380E00380E00380E00380E00380F00380F00780F
00700F80F00FC3E00FFFC00EFF80067E00151C809B18>I<01FE0007FF001FFF803F07803C0300
780000700000F00000E00000E00000E00000E00000F000007000007801C03C01C03F07C01FFF80
07FF0001FC0012147D9318>I<001F80003F80001F8000038000038000038000038000038003F3
800FFB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E00780F007
80700780780F803E3F801FFFF00FFBF803E3F0151C7E9B18>I<03F0000FFC001FFE003E1F0078
0780700380F003C0E001C0E001C0FFFFC0FFFFC0FFFFC0F000007000007801C03C01C03F07C01F
FF8007FF0001FC0012147D9318>I<001FC0007FE000FFE001F1E001C0C001C00001C00001C000
7FFFC0FFFFC0FFFFC001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000
01C00001C00001C00001C0007FFF007FFF007FFF00131C7F9B18>I<03F1F007FFF80FFFF81E1F
303C0F003807003807003807003807003807003C0F001E1E001FFC003FF8003BF0003800003C00
001FFF001FFFC03FFFE07801F0F00078E00038E00038E00038F000787800F07E03F03FFFE00FFF
8003FE00151F7F9318>I<7E0000FE00007E00000E00000E00000E00000E00000E00000E3F000E
FF800FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E
00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C809B18>I<03800007C00007C00007C000038000
0000000000000000000000007FC000FFC0007FC00001C00001C00001C00001C00001C00001C000
01C00001C00001C00001C00001C00001C00001C00001C000FFFF00FFFF80FFFF00111D7C9C18>
I<0038007C007C007C003800000000000000000FFC1FFC0FFC001C001C001C001C001C001C001C
001C001C001C001C001C001C001C001C001C001C001C001C001C001C003C6038F078FFF07FE03F
C00E277E9C18>I<FE0000FE0000FE00000E00000E00000E00000E00000E00000E3FF00E7FF00E
3FF00E07800E0F000E1E000E3C000E78000EF0000FF8000FFC000F9C000F1E000E0F000E07800E
03800E03C0FFC7F8FFC7F8FFC7F8151C7F9B18>I<7FE000FFE0007FE00000E00000E00000E000
00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000
00E00000E00000E00000E00000E00000E0007FFFC0FFFFE07FFFC0131C7E9B18>I<7DF1F000FF
FBF8007FFFFC001F1F1C001E1E1C001E1E1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C00
1C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C007F1F1F00FFBFBF807F1F1F00191481
9318>I<7E3F00FEFF807FFFC00FE1E00F80E00F00E00F00E00E00E00E00E00E00E00E00E00E00
E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC1714809318>I<01F0000FFE001F
FF003E0F803803807001C07001C0E000E0E000E0E000E0E000E0E000E0F001E07001C07803C03C
07803E0F801FFF000FFE0001F00013147E9318>I<7E3E00FEFF807FFFC00FE3E00F80F00F0070
0F00780E00380E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF80
0E7E000E00000E00000E00000E00000E00000E00000E00007FC000FFE0007FC000151E809318>
I<03F3800FFB801FFF803E1F80780F80700780F00780E00380E00380E00380E00380E00380E003
80F00780700780780F803E1F801FFF800FFB8003F3800003800003800003800003800003800003
80000380003FF8003FF8003FF8151E7E9318>I<7F87E0FF9FF87FBFF803FC7803F03003E00003
C00003C00003C0000380000380000380000380000380000380000380000380007FFE00FFFF007F
FE0015147F9318>I<0FF7003FFF007FFF00F81F00E00700E00700F007007C00007FF0001FFC00
07FE00001F00600780E00380F00380F00780FC0F00FFFF00FFFE00E7F80011147D9318>I<0180
000380000380000380000380007FFFC0FFFFC0FFFFC00380000380000380000380000380000380
000380000380000380000380400380E00380E00381E003C3C001FFC000FF80007E0013197F9818
>I<7E07E0FE0FE07E07E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E
00E00E00E00E00E00E01E00F07E007FFFC03FFFE01FCFC1714809318>I<7F8FF0FF8FF87F8FF0
1E03C00E03800E03800E0380070700070700070700038E00038E00038E00038E0001DC0001DC00
01DC0000F80000F80000700015147F9318>I<FF8FF8FF8FF8FF8FF83800E03800E03800E01C01
C01C01C01C71C01CF9C01CF9C01CD9C01CD9C00DDD800DDD800DDD800D8D800F8F800F8F800707
0015147F9318>I<7F8FF07F9FF07F8FF0070700078E00039E0001DC0001F80000F80000700000
F00000F80001DC00039E00038E000707000F07807F8FF0FF8FF87F8FF015147F9318>I<7F8FF0
FF8FF87F8FF00E01C00E03800E0380070380070700070700038700038600038E0001CE0001CE00
00CC0000CC0000DC0000780000780000780000700000700000700000F00000E00079E0007BC000
7F80003F00001E0000151E7F9318>I<3FFFF07FFFF07FFFF07001E07003C0700780000F00001E
00007C0000F80001F00003E0000780000F00701E00703C0070780070FFFFF0FFFFF0FFFFF01414
7F9318>I<0007E0003FE0007FE000FC0000E00000E00000E00000E00000E00000E00000E00000
E00000E00000E00000E00001E0007FC000FF8000FF80007FC00001E00000E00000E00000E00000
E00000E00000E00000E00000E00000E00000E00000E00000FC00007FE0003FE00007E013247E9F
18>I<7C0000FF8000FFC00007E00000E00000E00000E00000E00000E00000E00000E00000E000
00E00000E00000E00000F000007FC0003FE0003FE0007FC000F00000E00000E00000E00000E000
00E00000E00000E00000E00000E00000E00000E00007E000FFC000FF80007C000013247E9F18>
125 D E /Fr 44 123 df<F8F8F8F8F805057B8410>46 D<007E0001FF8007FFE00FFFF01FC3F8
1F00F83E007C3E007C3C003C7C003E7C003E78001E78001EF8001FF8001FF8001FF8001FF8001F
F8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001F78001E7C003E7C003E
7C003E7C003E3E007C3F00FC1F00F81FC3F80FFFF007FFE001FF80007E0018297EA71D>48
D<00180000380000F80007F800FFF800FFF800FFF800F8F80000F80000F80000F80000F80000F8
0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8
0000F80000F80000F80000F80000F80000F80000F80000F80000F80000F8007FFFF07FFFF07FFF
F07FFFF014287CA71D>I<00FE0003FF800FFFC01FFFE01F07F03C01F87800FC78007C70007CF0
003EF0003E60003E20003E00003E00003E00003E00007C00007C0000780000F80001F00001E000
03E00007C0000F80001F00003E00007C0000F80001F00003E00003C0000780000F00001E00003C
00007FFFFE7FFFFE7FFFFE7FFFFE17287EA71D>I<007E0003FF8007FFE00FFFF01F81F83E00F8
7C007C78007C30007C10007C00007C00007C0000780000F80000F80001F00007E000FFC000FF80
00FFC000FFF00001F80000FC00007C00003E00003E00001F00001F00001F00001F00001F80001F
C0003EE0003EF0007E7C00FC3F01F81FFFF00FFFE003FFC000FE0018297EA71D>I<0007E0000F
E0000BE0001BE0001BE0003BE0003BE00073E00073E000F3E000F3E001E3E001E3E003E3E003C3
E007C3E00F83E00F83E01F03E01F03E03E03E03E03E07C03E07C03E0F803E0FFFFFFFFFFFFFFFF
FFFFFFFF0003E00003E00003E00003E00003E00003E00003E00003E00003E00003E018277EA61D
>I<3FFFF83FFFF83FFFF83FFFF83E00003E00003E00003E00003E00003E00003E00003E00003E
00003E00003E3E003EFF803FFFE03FFFF03FC1F03F80F83F007C3E007C00007C00003E00003E00
003E00003E00003E00003E00003E20003E60007C70007CF800F87C01F87F07F03FFFE00FFFC007
FF0001FC0017287EA61D>I<000FE0003FF800FFF801FFF803F81807E0000F80000F80001F0000
3E00003E00003C00007C00007C02007C3FC0F8FFE0F9FFF0FBFFF8FF80FCFE007CFE003EFC003E
FC003FFC001FF8001FF8001FF8001FF8001F78001F7C001F7C001F7C001E3C003E3E003E1E007C
1F00FC0FC1F807FFF003FFE001FF80007E0018297EA71D>I<FFFFFFFFFFFFFFFFFFFFFFFF0000
0F00001E00003C0000780000780000F00001F00001E00003E00003C00007C0000F80000F80000F
00001F00001E00003E00003E00003C00007C00007C00007C0000F80000F80000F80000F80000F8
0000F00001F00001F00001F00001F00001F00001F00001F00018277EA61D>I<007E0003FFC007
FFE00FFFF01F81F83F00FC3E007C7E007E7C003E7C003E7C003E7C003E7C003E3E007C3E007C1F
00F80FC3F007FFE003FFC003FFC00FFFF01F81F83F00FC3E007C7C003E7C003EF8001FF8001FF8
001FF8001FF8001FF8001F7C003E7C003E7E007E3F00FC1F81F80FFFF007FFE003FFC0007E0018
297EA71D>I<007E0001FF8007FFC00FFFE01F83F01F00F83E00787C007C7C003C78003EF8003E
F8003EF8001EF8001FF8001FF8001FF8001FF8003FFC003F7C003F7C007F3E007F3F01FF1FFFDF
0FFF9F07FF1F03FC1E00403E00003E00003E00007C00007C0000F80000F80001F01803E01E0FC0
3FFF801FFF000FFE0003F80018297EA71D>I<0003F000000003F000000007F800000007F80000
0007780000000F7C0000000F3C0000000F3C0000001F3E0000001E3E0000001E1E0000003E1F00
00003C1F0000007C0F8000007C0F800000780F800000F807C00000F807C00000F007C00001F003
E00001F003E00001E003E00003E001F00003E001F00003C001F00007C000F80007FFFFF80007FF
FFF8000FFFFFFC000FFFFFFC000F00007C001F00003E001F00003E001E00003E003E00001F003E
00001F003C00001F007C00000F807C00000F80F800000FC0F8000007C0F8000007C0222A7EA927
>65 D<0003FE00000FFFC0003FFFF000FFFFF001FE03F003F8006007E0002007C000000F800000
1F8000001F0000003E0000003E0000007C0000007C0000007C0000007C000000F8000000F80000
00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000007C0000007C00
00007C0000007C0000003E0000003E0000001F0000001F8000000F80000007C0000007E0000803
F8003801FE01F800FFFFF8003FFFF0000FFFC00003FE001D2C7CAA25>67
D<FFFFFFC0FFFFFFC0FFFFFFC0FFFFFFC0F8000000F8000000F8000000F8000000F8000000F800
0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000FFFFFF00FF
FFFF00FFFFFF00FFFFFF00F8000000F8000000F8000000F8000000F8000000F8000000F8000000
F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000FFFFFF
E0FFFFFFE0FFFFFFE0FFFFFFE01B2A7BA923>69 D<FFFFFF80FFFFFF80FFFFFF80FFFFFF80F800
0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8
000000F8000000F8000000F8000000F8000000FFFFFC00FFFFFC00FFFFFC00FFFFFC00F8000000
F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000192A7BA921>
I<0003FE00000FFFC0003FFFF8007FFFFC01FE01FC03F8007C07E0001807C000000F8000001F80
00001F0000003E0000003E0000007E0000007C0000007C0000007C000000F8000000F8000000F8
000000F8000000F8000000F8000000F8000000F8003FFCF8003FFCF8003FFC7C003FFC7C00007C
7C00007C7E00007C3E00007C3E00007C1F00007C1F80007C0F80007C07C0007C07E0007C03F800
7C01FE01FC007FFFFC003FFFF8000FFFC00003FE001E2C7CAA27>I<F800007CF800007CF80000
7CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800
007CF800007CF800007CF800007CF800007CF800007CFFFFFFFCFFFFFFFCFFFFFFFCFFFFFFFCF8
00007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007C
F800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF80000
7C1E2A7BA929>I<F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8
F8F8F8F8F8F8F8F8F8F8F8052A7BA910>I<FE000000FFFE000000FFFE000000FFFF000001FFFB
000001DFFB800003DFFB800003DFFB800003DFF9C000079FF9C000079FF9C000079FF9E0000F9F
F8E0000F1FF8E0000F1FF8F0001F1FF870001E1FF870001E1FF878003E1FF838003C1FF83C007C
1FF83C007C1FF81C00781FF81E00F81FF81E00F81FF80E00F01FF80F01F01FF80F01F01FF80701
E01FF80783E01FF80783E01FF80383C01FF803C7C01FF803C7C01FF801C7801FF801EF801FF801
EF801FF800EF001FF800FF001FF8007E001FF8007E001FF8007E001FF80000001F282A7BA933>
77 D<0003FC0000000FFF0000003FFFC00000FFFFF00001FE07F80003F801FC0007E0007E000F
C0003F000F80001F001F00000F801F00000F803E000007C03E000007C07C000003E07C000003E0
7C000003E078000001E0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F0F8000001
F0F8000001F0F8000001F0F8000001F0F8000001F07C000003E07C000003E07C000003E07E0000
07E03E000007C03E000007C01F00000F801F80001F800F80001F000FC0003F0007E0007E0003F8
01FC0001FE07F80000FFFFF000003FFFC000001FFF80000003FC0000242C7DAA2B>79
D<FFFFC000FFFFF800FFFFFE00FFFFFF00F8007F80F8001FC0F80007C0F80003E0F80003E0F800
01F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80003E0F80003E0F80007C0F8
001FC0F8007F80FFFFFF00FFFFFE00FFFFF800FFFFC000F8000000F8000000F8000000F8000000
F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00F8000000F8000000F80000001C2A7BA925>I<FFFFE000FFFFFC00FFFFFE00FFFFFF80F8003F
C0F8000FE0F80003F0F80001F0F80000F0F80000F8F80000F8F80000F8F80000F8F80000F8F800
00F0F80001F0F80003F0F8000FE0F8003FC0FFFFFF80FFFFFE00FFFFFC00FFFFE000F801F000F8
01F000F800F800F800F800F8007C00F8003C00F8003E00F8001E00F8001F00F8000F80F8000F80
F80007C0F80007C0F80003E0F80001E0F80001F0F80000F8F80000F8F800007C1E2A7BA926>82
D<007FC00001FFF80007FFFE000FFFFF001FC07F003F000F007E0006007C0000007C000000F800
0000F8000000F8000000F8000000F8000000FC0000007E0000007F0000003F8000001FF800000F
FF000007FFE00003FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000000FC0
000007C0000007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00FC003F
00FF80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>I<F80001F80001F0F80003F8
0001F0FC0003FC0001F07C0003FC0003E07C0003BC0003E07C0007BC0003E03E00079E0003C03E
00079E0007C03E00079E0007C03E000F9F0007C01F000F1F0007801F000F0F000F801F000F0F00
0F801F001F0F800F800F801E0F800F000F801E07801F000F801E07801F000F803E07C01F0007C0
3C07C01E0007C03C03C03E0007C03C03C03E0007C07C03E03E0003E07803E03C0003E07801E07C
0003E07801E07C0001E0F801F0780001E0F001F0780001F0F000F0F80001F0F000F0F80000F0F0
00F0F00000F1E000F8F00000F9E00078F00000F9E00079F0000079E00079E0000079C00079E000
007BC0003DE000007BC0003DE000003BC0003DC000003F80003FC000003F80001FC000001F8000
1F8000001F00001F8000342A7FA937>87 D<01FC000FFF003FFF803FFFC03C07E03003E00001F0
0001F00001F00001F00001F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0F801F0F801F0
F801F0FC03F07E0FF07FFFF03FFFF01FF9F00FE1F0141B7E9A1C>97 D<007F8001FFE007FFF80F
FFF81FC0783F00103E00007C00007C00007C0000F80000F80000F80000F80000F80000F80000F8
00007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FFE0007F80151B7E9A1A>99
D<00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0000
7C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C7C007C7C007CFC007CF800
7CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C7E00FC3F00FC1F83FC1FFF
FC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFFC01F87E03F01E03E00F07C
00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800007800007C00007C00003E
00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<001FC0007FC000FFC001FFC0
03F00003E00007C00007C00007C00007C00007C00007C00007C00007C00007C000FFFE00FFFE00
FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000
07C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C000122A7F
A912>I<F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F800
00F80000F80000F80000F87E00F9FF00FBFF80FFFFC0FF0FC0FE07E0FC03E0FC03E0F803E0F803
E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803
E0F803E0F803E0F803E0F803E0132A7BA91E>104 D<F8F8F8F8F800000000000000000000F8F8
F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8052A7CA90E>I<F8F8F8F8F8F8F8
F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8052A7CA9
0E>108 D<F83F007E00F9FF83FF00FBFFC7FF80FFFFEFFFC0FF07FE0FC0FE03FC07E0FC01F803
E0FC01F803E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F0
03E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801
F003E0F801F003E0F801F003E0F801F003E0F801F003E0231B7B9A30>I<F87E00F9FF00FBFF80
FFFFC0FF0FC0FE07E0FC03E0FC03E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0
F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0131B7B9A1E>
I<007E0001FF8007FFE00FFFF01F81F83F00FC3E007C7C003E7C003E78001EF8001FF8001FF800
1FF8001FF8001FF8001FF8001F7C003E7C003E7C003E3E007C3F00FC1FC3F80FFFF007FFE003FF
C0007E00181B7E9A1D>I<F87E00F9FF80FFFFC0FFFFE0FF0FE0FC03F0F801F8F800F8F800F8F8
00FCF8007CF8007CF8007CF8007CF8007CF8007CF8007CF800FCF800F8F801F8F801F0FC03F0FE
0FE0FFFFC0FFFF80FBFF00F8FC00F80000F80000F80000F80000F80000F80000F80000F80000F8
0000F80000F80000F8000016277B9A1E>I<F838F8F8F9F8FBF8FFC0FF00FE00FE00FC00FC00F8
00F800F800F800F800F800F800F800F800F800F800F800F800F800F800F800F8000D1B7B9A14>
114 D<07F8001FFF003FFF807FFF807C0780F80100F80000F80000F80000FC00007F00007FF000
3FFC001FFE0007FF0000FF80001FC0000FC00007C00007C04007C0E007C0FC1F80FFFF807FFF00
1FFE0007F800121B7E9A16>I<07C00007C00007C00007C00007C00007C00007C000FFFF80FFFF
80FFFF80FFFF8007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0
0007C00007C00007C00007C00007C00007C00007C08007E38003FFC003FFC001FF0000FC001222
7FA115>I<F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F8
03E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F807E0F807E0FC1FE0FFFFE07F
FBE03FF3E01F83E0131B7B9A1E>I<F800F800F8F801F800F87C01FC00F07C01DC01F07C01DC01
F07C03DC01F03E039E03E03E039E03E03E079E03E03E079E03E01F078F07C01F070F07C01F070F
07C00F0F0F07800F8F078F800F8F078F800F8E078F80079E078F00079E03CF0007DE03DF0003DC
03DE0003DC03DE0003DC01DE0003D801DE0001F801FC0001F801FC0001F800FC00251B7F9A28>
119 D<7C001F803E003F003F003E001F807C000F80FC0007C0F80007E1F00003F3E00001F7E000
00FFC000007F8000007F0000003E0000003E0000003F0000007F800000FFC00001F3E00003E1E0
0003E1F00007C0F8000F807C001F807C001F003E003E001F007C001F80FC000FC01A1B809A1B>
I<F8001FFC001F7C003E7C003E3E003E3E007C3F007C1F007C1F00F80F80F80F80F00F81F007C1
F007C1E003C1E003E3E003E3C001E3C001E3C000F78000F78000F780007700007700003F00003E
00003E00003E00003C00003C0000780000780000780000F00001F0007FE0007FE0007FC0007F00
0018277F9A1B>I<FFFFF0FFFFF0FFFFF0FFFFE00007E00007C0000F80001F80001F00003E0000
7E00007C0000F80001F80001F00003E00007E00007C0000F80001F80001F00003E00007E00007F
FFF8FFFFF8FFFFF8FFFFF8151B7E9A19>I E /Fs 22 122 df<FFFFFFFFFFFFC00000FFFFFFFF
FFFFFC0000FFFFFFFFFFFFFF8000FFFFFFFFFFFFFFE000FFFFFFFFFFFFFFF800001FFF000003FF
FC00001FFF0000007FFE00001FFF0000003FFF00001FFF0000001FFF80001FFF0000000FFFC000
1FFF0000000FFFC0001FFF00000007FFE0001FFF00000007FFE0001FFF00000007FFE0001FFF00
000003FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000003
FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000007FFE000
1FFF00000007FFE0001FFF00000007FFC0001FFF0000000FFFC0001FFF0000000FFF80001FFF00
00001FFF00001FFF0000003FFE00001FFF0000007FFC00001FFF000001FFF800001FFF00000FFF
E000001FFFFFFFFFFF8000001FFFFFFFFFFC0000001FFFFFFFFFFF8000001FFFFFFFFFFFF00000
1FFF000000FFFC00001FFF0000003FFF00001FFF0000000FFF80001FFF00000007FFE0001FFF00
000003FFF0001FFF00000001FFF0001FFF00000001FFF8001FFF00000000FFFC001FFF00000000
FFFC001FFF000000007FFE001FFF000000007FFE001FFF000000007FFE001FFF000000007FFF00
1FFF000000007FFF001FFF000000007FFF001FFF000000007FFF001FFF000000007FFF001FFF00
0000007FFF001FFF000000007FFF001FFF000000007FFF001FFF000000007FFE001FFF00000000
FFFE001FFF00000000FFFE001FFF00000001FFFC001FFF00000001FFFC001FFF00000003FFF800
1FFF00000007FFF0001FFF0000000FFFE0001FFF0000003FFFC0001FFF000001FFFF80FFFFFFFF
FFFFFFFF00FFFFFFFFFFFFFFFC00FFFFFFFFFFFFFFF000FFFFFFFFFFFFFF8000FFFFFFFFFFFFF8
000048477CC653>66 D<000000007FFC00000E0000000FFFFFC0001E0000007FFFFFF8003E0000
03FFFFFFFE007E00000FFFFFFFFF80FE00003FFFF8007FC1FE0000FFFF80000FF3FE0003FFFC00
0003FFFE0007FFF0000001FFFE000FFFC00000007FFE001FFF800000003FFE003FFF000000001F
FE007FFC000000000FFE00FFFC000000000FFE01FFF80000000007FE03FFF00000000003FE03FF
E00000000003FE07FFE00000000001FE0FFFC00000000001FE0FFFC00000000000FE1FFF800000
000000FE1FFF8000000000007E3FFF8000000000007E3FFF0000000000007E3FFF000000000000
3E7FFF0000000000003E7FFF0000000000003E7FFF0000000000003E7FFE000000000000007FFE
00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000
00000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000000000
00FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE
000000000000007FFE000000000000007FFE000000000000007FFF000000000000007FFF000000
0000003E7FFF0000000000003E3FFF0000000000003E3FFF0000000000003E3FFF800000000000
3E1FFF8000000000007E1FFF8000000000007C0FFFC000000000007C0FFFC00000000000FC07FF
E00000000000FC03FFE00000000000F803FFF00000000001F801FFF80000000003F000FFFC0000
000003F0007FFE0000000007E0003FFF000000000FC0001FFF800000001F80000FFFC00000003F
000007FFF0000000FE000003FFFC000001FC000000FFFF80000FF80000003FFFF8007FE0000000
0FFFFFFFFFC000000003FFFFFFFF00000000007FFFFFFC00000000000FFFFFE00000000000007F
FE00000047497AC754>I<00000003FFF0000000000000007FFFFF80000000000003FFFFFFF000
000000000FFFFFFFFC00000000003FFF003FFF0000000000FFF80007FFC000000003FFE00001FF
F000000007FF8000007FF80000000FFF0000003FFC0000001FFE0000001FFE0000003FFC000000
0FFF0000007FF800000007FF800000FFF000000003FFC00001FFF000000003FFE00003FFE00000
0001FFF00003FFC000000000FFF00007FFC000000000FFF80007FFC000000000FFF8000FFF8000
0000007FFC000FFF80000000007FFC001FFF80000000007FFE001FFF00000000003FFE003FFF00
000000003FFF003FFF00000000003FFF003FFF00000000003FFF007FFF00000000003FFF807FFE
00000000001FFF807FFE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF80FF
FE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0
FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFF
C0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001F
FFC0FFFE00000000001FFFC0FFFE00000000001FFFC07FFE00000000001FFF807FFF0000000000
3FFF807FFF00000000003FFF807FFF00000000003FFF803FFF00000000003FFF003FFF00000000
003FFF003FFF80000000007FFF001FFF80000000007FFE001FFF80000000007FFE001FFF800000
00007FFE000FFFC000000000FFFC000FFFC000000000FFFC0007FFE000000001FFF80003FFE000
000001FFF00003FFF000000003FFF00001FFF000000003FFE00000FFF800000007FFC000007FFC
0000000FFF8000007FFE0000001FFF8000003FFF0000003FFF0000000FFFC00000FFFC00000007
FFE00001FFF800000003FFF80007FFF000000000FFFF003FFFC0000000007FFFFFFFFF80000000
000FFFFFFFFC000000000003FFFFFFF00000000000007FFFFF8000000000000003FFF000000000
4A497AC757>79 D<00007FF00007000007FFFF000F00001FFFFFE01F00007FFFFFF83F0000FFFF
FFFC7F0003FFC00FFFFF0007FE0000FFFF000FFC00003FFF000FF800001FFF001FF000000FFF00
3FE0000007FF003FE0000003FF007FE0000001FF007FC0000000FF007FC0000000FF00FFC00000
007F00FFC00000007F00FFC00000003F00FFE00000003F00FFE00000003F00FFE00000001F00FF
F00000001F00FFF80000001F00FFFC0000001F00FFFE00000000007FFF80000000007FFFF80000
00007FFFFF800000003FFFFFFC0000003FFFFFFFC000001FFFFFFFF800001FFFFFFFFF00000FFF
FFFFFFC00007FFFFFFFFE00003FFFFFFFFF00001FFFFFFFFFC0000FFFFFFFFFC00003FFFFFFFFE
00000FFFFFFFFF000003FFFFFFFF8000007FFFFFFF80000003FFFFFFC00000003FFFFFC0000000
01FFFFE0000000003FFFE0000000000FFFE00000000003FFF00000000001FFF00000000001FFF0
F800000000FFF0F8000000007FF0F8000000007FF0F8000000007FF0F8000000003FF0FC000000
003FF0FC000000003FF0FC000000003FE0FE000000003FE0FE000000003FE0FF000000007FE0FF
800000007FC0FF800000007FC0FFC0000000FF80FFF0000001FF80FFF8000001FF00FFFE000003
FE00FFFFC0000FFC00FF7FFC007FF800FE3FFFFFFFF000FC0FFFFFFFC000F803FFFFFF0000F000
7FFFFC0000E00003FFE0000034497AC741>83 D<0003FFF0000000003FFFFF00000000FFFFFFE0
000001FFFFFFF8000003FF003FFC000007FF800FFE000007FF8007FF000007FF8003FF800007FF
8003FFC00007FF8001FFC00007FF8001FFC00003FF0000FFE00001FE0000FFE00000FC0000FFE0
0000000000FFE00000000000FFE00000000000FFE00000000000FFE00000000000FFE000000000
FFFFE00000007FFFFFE0000007FFFFFFE000003FFFFCFFE00000FFFE00FFE00003FFF000FFE000
07FFC000FFE0000FFF0000FFE0001FFE0000FFE0003FFC0000FFE0007FFC0000FFE0007FF80000
FFE000FFF80000FFE000FFF00000FFE000FFF00000FFE000FFF00000FFE000FFF00001FFE000FF
F00001FFE000FFF80003FFE0007FFC0007FFE0007FFC000FFFE0003FFE001F7FF8001FFF80FE7F
FFF007FFFFFC3FFFF003FFFFF01FFFF0007FFFC00FFFF00007FE0003FFF0342E7DAD38>97
D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC0
0000000003FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC000000000
01FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC000
00000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001
FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00FFE000001FFC0FFFF
C00001FFC3FFFFF00001FFCFFFFFFC0001FFFFE01FFF0001FFFF0007FF8001FFFC0001FFC001FF
F80000FFE001FFF00000FFF001FFE000007FF001FFC000007FF801FFC000003FFC01FFC000003F
FC01FFC000003FFC01FFC000001FFE01FFC000001FFE01FFC000001FFE01FFC000001FFF01FFC0
00001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF
01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000
001FFE01FFC000001FFE01FFC000003FFE01FFC000003FFC01FFC000003FFC01FFC000003FF801
FFC000007FF801FFE000007FF001FFF00000FFE001FFF80001FFE001FFFC0003FFC001FF7F0007
FF0001FE3FC03FFE0001FC0FFFFFF80001FC03FFFFF00001F800FFFF800001F0001FF800003848
7CC741>I<00001FFF80000001FFFFF8000007FFFFFE00001FFFFFFF00007FF801FF8000FFE003
FFC003FFC003FFC007FF8003FFC007FF0003FFC00FFE0003FFC01FFE0003FFC01FFC0001FF803F
FC0000FF003FFC00007E007FFC000000007FF8000000007FF800000000FFF800000000FFF80000
0000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FF
F800000000FFF800000000FFF800000000FFF8000000007FFC000000007FFC000000007FFC0000
00003FFC000000003FFE000000001FFE000001F01FFF000001F00FFF000003F007FF800007E007
FFC00007C003FFE0000FC000FFF8003F80007FFF00FF00001FFFFFFC000007FFFFF0000001FFFF
C00000001FFC00002C2E7CAD34>I<0000000000FF8000000001FFFF8000000001FFFF80000000
01FFFF8000000001FFFF8000000001FFFF800000000007FF800000000003FF800000000003FF80
0000000003FF800000000003FF800000000003FF800000000003FF800000000003FF8000000000
03FF800000000003FF800000000003FF800000000003FF800000000003FF800000000003FF8000
00000003FF800000000003FF800000000003FF800000000003FF800000000003FF800000000003
FF8000001FF803FF800001FFFF03FF80000FFFFFE3FF80003FFFFFFBFF80007FFC03FFFF8001FF
E0007FFF8003FFC0003FFF8007FF80000FFF800FFF000007FF800FFE000003FF801FFE000003FF
801FFC000003FF803FFC000003FF803FFC000003FF807FFC000003FF807FF8000003FF807FF800
0003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80
FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF80000
03FF80FFF8000003FF807FF8000003FF807FF8000003FF807FF8000003FF803FFC000003FF803F
FC000003FF803FFC000003FF801FFE000007FF800FFE000007FF800FFF00000FFF8007FF00001F
FF8003FF80007FFF8001FFE000FFFFC000FFF807FBFFFF003FFFFFE3FFFF000FFFFFC3FFFF0003
FFFE03FFFF00003FF003FFFF38487CC741>I<00003FFE00000001FFFFC000000FFFFFF800003F
FFFFFC00007FFC0FFE0000FFE003FF0001FF8000FF8003FF0000FFC007FF00007FE00FFE00003F
F01FFE00003FF01FFC00003FF03FFC00001FF83FFC00001FF87FFC00001FF87FF800001FFC7FF8
00000FFC7FF800000FFCFFF800000FFCFFFFFFFFFFFCFFFFFFFFFFFCFFFFFFFFFFFCFFFFFFFFFF
FCFFF800000000FFF800000000FFF800000000FFF800000000FFF8000000007FF8000000007FF8
000000007FFC000000007FFC000000003FFC000000003FFC0000007C1FFE0000007C0FFE000000
FC0FFF000000F807FF800001F803FFC00003F001FFE00007E000FFF8001FC0007FFF00FF80001F
FFFFFF000007FFFFFC000000FFFFF00000000FFF00002E2E7DAD35>I<00000000003F000000FF
F000FFC00007FFFE03FFE0003FFFFFCFFFF0007FFFFFFFCFF001FFE07FFE0FF003FF801FFC1FF0
07FF000FFE0FF00FFE0007FF0FF00FFE0007FF07E01FFC0003FF81801FFC0003FF80003FFC0003
FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003F
FC0003FFC0003FFC0003FFC0003FFC0003FFC0001FFC0003FF80001FFC0003FF80000FFE0007FF
00000FFE0007FF000007FF000FFE000003FF801FFC000001FFE07FF8000003FFFFFFE0000003FF
FFFFC000000787FFFE0000000780FFF0000000078000000000000F8000000000000F8000000000
000FC000000000000FC000000000000FE000000000000FF8000000000007FFFFFFF0000007FFFF
FFFF000007FFFFFFFFE00003FFFFFFFFF80003FFFFFFFFFC0001FFFFFFFFFE0000FFFFFFFFFF00
03FFFFFFFFFF800FFFFFFFFFFF801FF000001FFFC03FC0000001FFC07F800000007FE07F800000
003FE0FF000000003FE0FF000000001FE0FF000000001FE0FF000000001FE0FF000000001FE0FF
800000003FE07F800000003FC07FC00000007FC03FE0000000FF801FF8000003FF000FFE00000F
FE0007FFE000FFFC0001FFFFFFFFF000003FFFFFFF8000000FFFFFFE000000007FFFC000003444
7DAE3A>103 D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC000
000000FFFFC00000000003FFC00000000001FFC00000000001FFC00000000001FFC00000000001
FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000
000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF
C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC007FE00
0001FFC03FFFC00001FFC0FFFFF00001FFC1FFFFFC0001FFC7F01FFC0001FFCFC00FFE0001FFDF
0007FF0001FFDE0007FF0001FFFC0007FF0001FFF80003FF8001FFF80003FF8001FFF00003FF80
01FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC00003FF8001FFC000
03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001
FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF
81FFFFFF38487BC741>I<00FC0001FF0003FF8007FF800FFFC00FFFC00FFFE00FFFE00FFFE00F
FFC00FFFC007FF8003FF8001FF0000FC0000000000000000000000000000000000000000000000
0000000000000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC8
20>I<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000FFFF
FF80FFFFFF80FFFFFF80FFFFFF80FFFFFF8019487CC720>108 D<00FF8003FF000003FF000000
FFFF801FFFE0001FFFE00000FFFF807FFFF8007FFFF80000FFFF81FFFFFC01FFFFFC0000FFFF83
F01FFE03F01FFE0000FFFF87C007FF07C007FF000003FF8F0007FF8F0007FF800001FF9E0003FF
9E0003FF800001FFBC0003FFBC0003FF800001FFF80001FFF80001FFC00001FFF80001FFF80001
FFC00001FFF00001FFF00001FFC00001FFE00001FFE00001FFC00001FFE00001FFE00001FFC000
01FFE00001FFE00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF
80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80592E7BAD62>
I<00FF8007FE0000FFFF803FFFC000FFFF80FFFFF000FFFF81FFFFFC00FFFF87F01FFC00FFFF8F
C00FFE0003FF9F0007FF0001FF9E0007FF0001FFBC0007FF0001FFF80003FF8001FFF80003FF80
01FFF00003FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC000
03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001
FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF
81FFFFFFFFFFFF81FFFFFF382E7BAD41>I<00000FFE0000000001FFFFF000000007FFFFFC0000
001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF00003FF00001FF80007FE0000
0FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC000007FF803FF8000003FF803F
F8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0FFF8000003FFE0FFF8000003
FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8
000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07FF8000003FFC07FF8000003FF
C07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007FF001FFC000007FF000FFE00
000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FFE000FFE000007FFC07FFC000
001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF000000332E7DAD3A>I<007FC00F
FE0000FFFFC0FFFFC000FFFFC3FFFFF000FFFFCFFFFFFC00FFFFFFE03FFF00FFFFFF000FFF8003
FFFC0007FFC001FFF80003FFE001FFF00001FFF001FFE00000FFF001FFC00000FFF801FFC00000
7FFC01FFC000007FFC01FFC000003FFC01FFC000003FFE01FFC000003FFE01FFC000003FFE01FF
C000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001F
FF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC0
00001FFF01FFC000003FFE01FFC000003FFE01FFC000003FFE01FFC000007FFC01FFC000007FFC
01FFC000007FF801FFC00000FFF801FFE00000FFF001FFF00001FFE001FFF80003FFE001FFFC00
07FFC001FFFF000FFF0001FFFFC07FFE0001FFCFFFFFF80001FFC3FFFFF00001FFC0FFFF800001
FFC01FF8000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000
000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF
C00000000001FFC00000000001FFC00000000001FFC00000000001FFC000000000FFFFFF800000
00FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF8000000038427CAD41>I<00FF80
7F00FFFF81FFC0FFFF87FFF0FFFF8FFFF8FFFF9F8FFCFFFF9F1FFE03FFBE1FFE01FFBC1FFE01FF
F81FFE01FFF81FFE01FFF01FFE01FFF00FFC01FFE007F801FFE003F001FFE0000001FFE0000001
FFE0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC00000
01FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC000
0001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0
000001FFC00000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFFE000272E7CAD2F>
114 D<000FFE01C000FFFFC7C003FFFFFFC007FFFFFFC01FF001FFC03FC0007FC03F80003FC07F
00001FC07E00000FC0FE00000FC0FE000007C0FE000007C0FF000007C0FF800007C0FFC0000000
FFF8000000FFFFC000007FFFFE00007FFFFFE0003FFFFFF8001FFFFFFE000FFFFFFF0007FFFFFF
8001FFFFFFC0007FFFFFE0001FFFFFF00000FFFFF0000007FFF00000007FF80000001FF8F80000
0FF8F8000007F8FC000007F8FC000003F8FC000003F8FE000003F8FF000003F0FF000007F0FF80
0007F0FFC0000FE0FFF0001FC0FFFC00FF80FFFFFFFF00FC7FFFFC00F01FFFF000E003FF800025
2E7CAD2E>I<0001F000000001F000000001F000000001F000000001F000000001F000000003F0
00000003F000000003F000000007F000000007F000000007F00000000FF00000000FF00000001F
F00000003FF00000003FF00000007FF0000001FFF0000003FFF000000FFFFFFFC0FFFFFFFFC0FF
FFFFFFC0FFFFFFFFC0FFFFFFFFC0007FF00000007FF00000007FF00000007FF00000007FF00000
007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF000
00007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF0
0000007FF00000007FF00000007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007F
F001F0007FF001F0007FF001F0007FF001F0007FF003F0003FF803E0003FF807E0001FFC07C000
1FFE0F80000FFFFF800003FFFE000000FFFC0000001FE00024427EC12D>I<007FC00000FF80FF
FFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF8003FFC00007
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0
0003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80
01FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00007FF8001FFC000
07FF8001FFC00007FF8001FFC0000FFF8001FFC0001FFF8000FFC0003FFF8000FFE0007FFF8000
7FE000FBFFC0007FF803F3FFFF003FFFFFE3FFFF000FFFFF83FFFF0003FFFE03FFFF00007FF003
FFFF382E7BAD41>I<FFFFFF0003FFFFFFFFFF0003FFFFFFFFFF0003FFFFFFFFFF0003FFFFFFFF
FF0003FFFF01FFE000001F8000FFE000001F0000FFF000003F00007FF000003E00007FF800007E
00003FF800007C00003FFC0000FC00001FFC0000F800001FFC0000F800001FFE0001F800000FFE
0001F000000FFF0003F0000007FF0003E0000007FF8007E0000003FF8007C0000003FFC007C000
0003FFC00FC0000001FFC00F80000001FFE01F80000000FFE01F00000000FFF03F000000007FF0
3E000000007FF87E000000003FF87C000000003FF87C000000003FFCFC000000001FFCF8000000
001FFFF8000000000FFFF0000000000FFFF00000000007FFE00000000007FFE00000000003FFC0
0000000003FFC00000000003FFC00000000001FF800000000001FF800000000000FF0000000000
00FF0000000000007E0000000000007E0000000000007E0000000000007C000000000000FC0000
00000000F8000000000001F8000000000001F00000001F0003F00000003F8003E00000007FC003
E0000000FFE007C0000000FFE00FC0000000FFE00F80000000FFE01F80000000FFE03F00000000
7FC0FE000000007F83FC000000003FFFF8000000001FFFE00000000007FF800000000001FE0000
00000038427EAD3D>121 D E /Ft 8 117 df<0000000F80000000000F80000000001F80000000
003F80000000007F80000000007F8000000000FF8000000001FF8000000003FF8000000003FF80
00000007FF800000000FFF800000001FFF800000001FFF800000003DFF8000000079FF80000000
F9FF80000000F1FF80000001E1FF80000003C1FF8000000781FF8000000F81FF8000000F01FF80
00001E01FF8000003C01FF8000007C01FF8000007801FF800000F001FF800001E001FF800003E0
01FF800003C001FF8000078001FF80000F0001FF80001F0001FF80001E0001FF80003C0001FF80
00780001FF8000F80001FF8000FFFFFFFFFF80FFFFFFFFFF80FFFFFFFFFF80FFFFFFFFFF800000
03FF8000000003FF8000000003FF8000000003FF8000000003FF8000000003FF8000000003FF80
00000003FF8000000003FF8000000003FF80000007FFFFFF800007FFFFFF800007FFFFFF800007
FFFFFF8029387DB730>52 D<0000001FFE0000E0000003FFFFE001E000001FFFFFF803E000007F
FFFFFE07E00001FFFC00FF0FE00007FFC0001FDFE0000FFF000007FFE0003FFC000001FFE0007F
F0000000FFE000FFE00000007FE001FFC00000003FE003FF800000001FE007FF800000001FE007
FF000000000FE00FFE0000000007E00FFE0000000007E01FFC0000000007E01FFC0000000003E0
3FFC0000000003E03FF80000000003E07FF80000000001E07FF80000000001E07FF80000000001
E07FF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF000000000
0000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000
000000FFF0000000000000FFF0000000000000FFF0000000000000FFF00000000000007FF00000
000000007FF80000000000007FF80000000001E07FF80000000001E03FF80000000001E03FFC00
00000001E01FFC0000000001E01FFC0000000003E00FFE0000000003C00FFE0000000007C007FF
0000000007C007FF800000000F8003FF800000000F8001FFC00000001F0000FFE00000003E0000
7FF00000007C00003FFC000000F800000FFF000003F0000007FFC0000FE0000001FFFC007FC000
00007FFFFFFF000000001FFFFFFC0000000003FFFFE000000000001FFE0000003B3D7BBB46>67
D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF801FF80001FF800FFC0001FF8
007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF00003C0003FF0000000003FF0
000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF000007FF83FF00003FF
803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0007FE0003FF000FFE0003FF0
00FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0007FF000FFE000FFF0007FE0
01DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FFE0003FE001FFE02B267DA52F
>97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF8007FC003F800FF8003FC01F
F8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE0000FF87FE0000FF8FFE0000FF8
FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0000000FFE0000000FFE00000
00FFE00000007FE00000007FE00000007FF00000003FF00000783FF00000781FF80000F80FF800
00F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00000FFFFC000001FFC0002526
7DA52C>101 D<00FF0000000000FFFF0000000000FFFF0000000000FFFF0000000000FFFF0000
00000007FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003
FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000
000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF
000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003FF0F81FF800003FF3C00FF80
0003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007FE00003FFC0007FE00003FF80
007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000
03FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF0000
7FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003
FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007F
E00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80313C
7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFFFE03FF00FFFFF000FFC007FF
E0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001FFC03FF00000FFC03FF00000F
FE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF
000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF00000F
FE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF00001FF803FF80003FF003FF
C0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC0003FF0FFFF00003FF01FF00
0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF
0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000
00FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC00000030377EA536>112
D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FEE07FE003FEE0
7FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003FF00000003FF
00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003
FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000
03FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE000023267EA5
28>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000001F0000001F
0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00001FFFFFF0FF
FFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF000003FF0000
03FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF00
0003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF
003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377EB626>116
D E /Fu 78 124 df<000FC3E0007FEFF000E07C7801C0F8780380F83007007000070070000700
700007007000070070000700700007007000FFFFFF80FFFFFF8007007000070070000700700007
007000070070000700700007007000070070000700700007007000070070000700700007007000
0700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381
E00381E00701E0070000070000070000070000070000070000FFFFE0FFFFE00700E00700E00700
E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700
E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E00380E00700E00700E00700E007
00E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E00700E00700E00700E00700E007
00E00700E00700E00700E00700E00700E00700E00700E00700E07FE7FE7FE7FE1720809F19>I<
000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700F00F00070070000007007000
000700700000070070000007007000000700700000FFFFFFFF00FFFFFFFF000700700700070070
070007007007000700700700070070070007007007000700700700070070070007007007000700
7007000700700700070070070007007007000700700700070070070007007007007FE3FE3FF07F
E3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06180C180C381C301860304020
0F0E7E9F17>34 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D>39 D<006000C00180030007
0006000C001C00180038003800300070007000700060006000E000E000E000E000E000E000E000
E000E000E000E000E0006000600070007000700030003800380018001C000C0006000700030001
8000C000600B2E7DA112>I<C0006000300018001C000C0006000700030003800380018001C001
C001C000C000C000E000E000E000E000E000E000E000E000E000E000E000E000C000C001C001C0
01C00180038003800300070006000C001C00180030006000C0000B2E7DA112>I<70F8FCFC7C0C
0C0C181838306040060E7C840D>44 D<FFC0FFC0FFC00A037F8A0F>I<70F8F8F87005057C840D>
I<00030003000700060006000E000C000C001C0018001800380030003000700060006000E000C0
00C001C00180018001800380030003000700060006000E000C000C001C00180018003800300030
00700060006000E000C000C000102D7DA117>I<03F0000FFC001E1E001C0E0038070078078070
0380700380700380F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F0
03C0F003C0F003C07003807003807003807807803807001C0E001E1E000FFC0003F000121F7E9D
17>I<018003801F80FF80E3800380038003800380038003800380038003800380038003800380
0380038003800380038003800380038003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E
00300F00600F80C00780F007C0F807C0F803C0F803C02003C00007C0000780000780000F00000E
00001C0000380000700000E00001C0000380000300000600C00C00C01800C03001807FFF80FFFF
80FFFF80121E7E9D17>I<03F0000FFC001C1E00300F00780F80780F80780780380F80000F8000
0F00000F00001E00003C0003F80003F000001C00000E00000F000007800007800007C02007C0F8
07C0F807C0F807C0F00780600F80700F003C1E001FFC0003F000121F7E9D17>I<000E00000E00
001E00003E00003E00006E0000EE0000CE00018E00038E00030E00060E000E0E000C0E00180E00
380E00300E00600E00E00E00FFFFF0FFFFF0000E00000E00000E00000E00000E00000E00000E00
00FFE000FFE0141E7F9D17>I<3803003FFF003FFE003FF8003FE0003000003000003000003000
0030000030000031F00037FC003E0E003C07003807803003800003800003C00003C00003C06003
C0F003C0F003C0F00380C00780600700700E003C1E000FF80007E000121F7E9D17>I<007C0001
FE000783000E03800C07801C0780380300380000780000700000700000F3F800F7FE00FC0E00F8
0700F80380F80380F003C0F003C0F003C0F003C0F003C07003C07003C07003803803803807001C
07000E1E0007FC0001F000121F7E9D17>I<6000007FFFC07FFFC07FFF80600180C00300C00600
C00600000C0000180000180000300000600000600000E00000C00001C00001C00001C000038000
038000038000038000078000078000078000078000078000078000078000030000121F7D9D17>
I<03F0000FFC001E1E003807003007007003807003807003807003807807003E07003F0E001FDC
000FF80003F80007FC001E7E00383F00700F80700780E003C0E001C0E001C0E001C0E001C0F001
807003803807001E0E000FFC0003F000121F7E9D17>I<03F00007F8001E1C00380E0038070070
0700700380F00380F00380F003C0F003C0F003C0F003C0F003C07007C07007C0380FC01C0FC01F
FBC007F3C0000380000380000780000700300700780E00780E00701C003078001FF0000FC00012
1F7E9D17>I<70F8F8F8700000000000000000000070F8F8F87005147C930D>I<70F8F8F8700000
000000000000000070F8F8F878181818183030606040051D7C930D>I<7FFFFFE0FFFFFFF00000
000000000000000000000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C
0C7D9023>61 D<0FC03FF070786038E03CF03CF03C603C003C007800F000E001C0018003800300
0300030003000300030003000000000000000000000007000F800F800F8007000E207D9F15>63
D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018
F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000
C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0
0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 D<FFFFE000FFFFF80007803E0007801F0007800F
0007800F8007800F8007800F8007800F8007800F8007800F0007801F0007803E0007807C0007FF
F80007FFFC0007801E0007800F0007800F8007800780078007C0078007C0078007C0078007C007
8007C00780078007800F8007801F0007803E00FFFFFC00FFFFF0001A1F7E9E20>I<001FC04000
7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0
780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8
1C00007FF000001FC0001A217D9F21>I<FFFFE000FFFFFC0007803E0007800F00078007800780
03C0078003E0078001E0078001F0078000F0078000F0078000F8078000F8078000F8078000F807
8000F8078000F8078000F8078000F8078000F8078000F0078000F0078001F0078001E0078001E0
078003C00780078007800F0007803E00FFFFFC00FFFFE0001D1F7E9E23>I<FFFFFF00FFFFFF00
07801F000780070007800300078003000780038007800180078001800780C1800780C1800780C0
000780C0000781C00007FFC00007FFC0000781C0000780C0000780C0000780C0600780C0600780
0060078000C0078000C0078000C0078001C0078001C0078003C007800F80FFFFFF80FFFFFF801B
1F7E9E1F>I<FFFFFF00FFFFFF0007801F00078007000780030007800300078003800780018007
8001800780C1800780C1800780C0000780C0000781C00007FFC00007FFC0000781C0000780C000
0780C0000780C0000780C000078000000780000007800000078000000780000007800000078000
0007800000FFFE0000FFFE0000191F7E9E1E>I<000FC020007FF86001F81CE003E006E0078003
E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800
0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C
0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F
24>I<FFF8FFF8FFF8FFF807800F0007800F0007800F0007800F0007800F0007800F0007800F00
07800F0007800F0007800F0007800F0007800F0007FFFF0007FFFF0007800F0007800F0007800F
0007800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F000780
0F00FFF8FFF8FFF8FFF81D1F7E9E22>I<FFFCFFFC078007800780078007800780078007800780
078007800780078007800780078007800780078007800780078007800780078007800780FFFCFF
FC0E1F7F9E10>I<FFFC1FFCFFFC1FFC078007C0078007000780060007800C0007801800078030
00078070000780E0000781C000078380000787000007878000078F8000079BC00007B1E00007E1
E00007C0F0000780F0000780780007803C0007803C0007801E0007801E0007800F000780078007
800780078007C0FFFC3FFCFFFC3FFC1E1F7E9E23>75 D<FFFE00FFFE0007800007800007800007
800007800007800007800007800007800007800007800007800007800007800007800007800007
800007800607800607800607800607800E07800E07800C07801C07803C0780FCFFFFFCFFFFFC17
1F7E9E1C>I<FF80001FF8FFC0003FF807C0003F0007C0003F0006E0006F0006E0006F0006E000
6F00067000CF00067000CF00067000CF000638018F000638018F000638018F00061C030F00061C
030F00061C030F00060E060F00060E060F00060E060F0006070C0F0006070C0F0006070C0F0006
03980F000603980F000603980F000601F00F000601F00F000601F00F000F00E00F00FFF0E1FFF8
FFF0E1FFF8251F7E9E2A>I<FF807FF8FFC07FF807C0078007E0030007E0030006F0030006F003
0006780300063C0300063C0300061E0300061E0300060F0300060F0300060783000607C3000603
C3000601E3000601E3000600F3000600F30006007B0006007B0006003F0006001F0006001F0006
000F0006000F000F000700FFF00700FFF003001D1F7E9E22>I<001F800000FFF00001E0780007
C03E000F801F000F000F001E0007803C0003C03C0003C07C0003E07C0003E0780001E0F80001F0
F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E07C0003
E07C0003E03C0003C03E0007C01E0007800F000F000F801F0007C03E0001F0F80000FFF000001F
80001C217D9F23>I<FFFFE000FFFFF80007807C0007801E0007801F0007800F0007800F800780
0F8007800F8007800F8007800F8007800F0007801F0007801E0007807C0007FFF80007FFE00007
800000078000000780000007800000078000000780000007800000078000000780000007800000
0780000007800000FFFC0000FFFC0000191F7E9E1F>I<FFFF8000FFFFF0000780F80007803C00
07803E0007801E0007801F0007801F0007801F0007801F0007801E0007803E0007803C000780F8
0007FFF00007FF80000781C0000780E00007807000078070000780780007807800078078000780
7C0007807C0007807C0007807C0C07807E0C07803E0CFFFC1F18FFFC0FF0000003E01E207E9E21
>82 D<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F00000
7800007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0
C001C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE0
780F01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F00
00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F
0000000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C
1F7E9E21>I<FFFC7FF8FFFC7FF807800780078003000780030007800300078003000780030007
800300078003000780030007800300078003000780030007800300078003000780030007800300
0780030007800300078003000780030007800300078003000380020003C0060003C0060001C00C
0000E0180000783000003FE000000FC0001D207E9E22>I<FFF007FEFFF007FE0F8001F00F0000
E0078000C0078000C007C001C003C0018003C0018003E0038001E0030001E0030000F0060000F0
060000F0060000780C0000780C0000780C00003C1800003C1800003C1800001E3000001E300000
1F7000000F6000000F6000000FE0000007C0000007C0000003800000038000000380001F207F9E
22>I<FFF07FF83FF0FFF07FF83FF00F0007800F800F00078003000F000780030007800FC00600
07800FC0060007800FC00600078019E00E0003C019E00C0003C019E00C0003C039E00C0001E030
F0180001E030F0180001E030F0180000F06078300000F06078300000F06078300000F0E07C3000
0078C03C60000078C03C60000078C03C6000003D801EC000003D801EC000003D801EC000003F00
0FC000001F000F8000001F000F8000001F000F8000000E00070000000E00070000000E00070000
2C207F9E2F>I<7FF83FF87FF83FF807C00F8003C0060001E00E0001F00C0000F0180000783800
007C3000003C7000003E6000001EC000000FC000000F8000000780000007C0000007E000000DE0
00001DF0000018F8000038780000307C0000603C0000E01E0000C01F0001800F00038007800300
07C00F8007C0FFE01FFEFFE01FFE1F1F7F9E22>I<FEFEC0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0
C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0FEFE072D7CA10D>91
D<0804180C3018703860306030C060C060C060F87CFC7EFC7E7C3E381C0F0E7B9F17>I<FEFE06
060606060606060606060606060606060606060606060606060606060606060606060606060606
06FEFE072D7FA10D>I<1FE0003FF000783800781C00300E00000E00000E00000E0003FE001FFE
003E0E00780E00F80E00F00E30F00E30F00E30F01E30783F703FEFE00F878014147E9317>97
D<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3F
000EFF800FC3C00F00E00E00F00E00700E00780E00780E00780E00780E00780E00780E00780E00
700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19>I<03F80FFC1E1E3C1E380C780070
00F000F000F000F000F000F0007000780038033C031F0E0FFC03F010147E9314>I<000380003F
80003F8000038000038000038000038000038000038000038000038000038003E3800FFB801E0F
803C0780780380780380700380F00380F00380F00380F00380F00380F003807003807003807807
803807801E1F800FFBF803E3F815207E9F19>I<03F0000FFC001E1E003C0F0038070078070070
0380F00380FFFF80FFFF80F00000F00000F000007000007800003801801C03800F070007FE0001
F80011147F9314>I<003C00FE01CF038F03060700070007000700070007000700FFF0FFF00700
0700070007000700070007000700070007000700070007000700070007007FF07FF01020809F0E
>I<0001E003E3F00FFF701C1C703C1E00380E00780F00780F00780F00780F00380E003C1E001C
1C003FF80033E0003000003000003800003FFE001FFF803FFFC07003E07000F0E00070E00070E0
0070F000F07801E03E07C00FFF0003FC00141F7F9417>I<0E0000FE0000FE00000E00000E0000
0E00000E00000E00000E00000E00000E00000E00000E3E000EFF000FC3800F01C00F01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC
FFE7FC16207F9F19>I<1C001E003E001E001C000000000000000000000000000E007E007E000E
000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A1F809E0C>
I<00E001F001F001F000E0000000000000000000000000007007F007F000F00070007000700070
00700070007000700070007000700070007000700070007000700070007000706070F0E0F1C07F
803E000C28829E0E>I<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00
000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18000E30000E70000EF0000FF8000F38
000E1C000E1E000E0E000E07000E07800E03800E03C0FFCFF8FFCFF815207F9F18>I<0E00FE00
FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E
000E000E000E000E000E000E000E000E00FFE0FFE00B20809F0C>I<0E3F03F000FEFF8FF800FF
C1DC1C000F80F80E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00
0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E
00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00FFC3800F01C00F01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FCFFE7FC
16147F9319>I<01F80007FE001E07803C03C03801C07000E07000E0F000F0F000F0F000F0F000
F0F000F0F000F07000E07801E03801C03C03C01E078007FE0001F80014147F9317>I<0E3F00FE
FF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E00780E00780E00780E00780E00700E
00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E00000E00000E00000E00000E0000FF
E000FFE000151D7F9319>I<03E1800FF9801E1F803C0780780780780380700380F00380F00380
F00380F00380F00380F003807003807803807807803C0F801E1F800FF38003E380000380000380
000380000380000380000380000380003FF8003FF8151D7E9318>I<0E78FEFCFF9E0F1E0F0C0F
000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00F147F9312>I<1F903F
F07070E030E030E030F00078007F803FE00FF000F0C078C038C038E038E038F070DFE08F800D14
7E9312>I<06000600060006000E000E001E003E00FFF8FFF80E000E000E000E000E000E000E00
0E000E000E000E180E180E180E180E18073007E003C00D1C7F9B12>I<0E01C0FE1FC0FE1FC00E
01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00E
03C0070FC007FDFC01F1FC16147F9319>I<FF87F8FF87F81E01E00E01C00E01800E0180070300
070300070300038600038600038E0001CC0001CC0001FC0000F80000F800007000007000007000
15147F9318>I<FF9FF3FCFF9FF3FC1C0780F01C0780601C07C0E00E0FC0C00E0CC0C00E0CE0C0
070CE18007186180071871800398730003B0330003B0330003B03F0001F03E0001E01E0001E01E
0000E01C0000C00C001E147F9321>I<7FC7FC7FC7FC0703E007038003830001C70000EE0000EC
00007800003800003C00007C0000EE0001C7000187000303800701C00F01E0FF87FEFF87FE1714
809318>I<FF87F8FF87F81E01E00E01C00E01800E018007030007030007030003860003860003
8E0001CC0001CC0001FC0000F80000F80000700000700000700000600000600000600000C000F0
C000F18000F380007F00003C0000151D7F9318>I<3FFF3FFF380E301C703C6038607060F001E0
01C0038007830F030E031C073C063806701EFFFEFFFE10147F9314>I<FFFFFCFFFFFC1602808C
17>I E /Fv 46 122 df<70F8FCFC7C0C0C0C0C181830306040060F7C840E>44
D<FFE0FFE0FFE00B037F8B10>I<70F8F8F87005057C840E>I<01F00007FC000E0E001C07003803
803803807803C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001
E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07
000E0E0007FC0001F00013227EA018>48 D<018003800F80FF80F3800380038003800380038003
800380038003800380038003800380038003800380038003800380038003800380038003800380
0380FFFEFFFE0F217CA018>I<03F0000FFC001C1F00300F806007806003C0C003C0F003E0F801
E0F801E0F801E02003E00003E00003C00003C0000780000780000F00001E00001C000038000070
0000E00001C0000380000700600E00600C00601800E03000C07FFFC0FFFFC0FFFFC013217EA018
>I<03F0000FFC001C1F003007807007807807C07807C07803C03807C00007C000078000078000
0F00000E00003C0003F80003F800001E00000F000007800007C00003C00003E02003E07003E0F8
03E0F803E0F003C06003C06007803007801C1F000FFC0003F00013227EA018>I<000E00000E00
001E00001E00003E00003E00006E0000EE0000CE0001CE00018E00030E00070E00060E000E0E00
0C0E00180E00180E00300E00700E00600E00E00E00FFFFF8FFFFF8000E00000E00000E00000E00
000E00000E00000E0001FFF001FFF015217FA018>I<1000801C07801FFF001FFE001FFC001FF0
0018000018000018000018000018000018000018F8001BFE001F0F001C07801803801803C00001
C00001E00001E00001E00001E06001E0F001E0F001E0F001C0C003C06003C06003803007001C1E
000FFC0003F00013227EA018>I<007E0001FF0003C1800701C00E03C01C03C01C018038000038
0000780000700000702000F1FC00F3FE00F60700FC0380F803C0F801C0F001C0F001E0F001E0F0
01E0F001E0F001E07001E07001E07001E03801C03803C01803801C07000E0E0007FC0001F00013
227EA018>I<01F00007FC000E0F001807803803803001C07001C07001C07001C07801C0780380
3E03803F07001FCE000FFC0003F80003FC000F7F001C3F80380F807007C07003C0E001E0E000E0
E000E0E000E0E000E0E000E07001C07001C03803801E0F000FFE0003F00013227EA018>56
D<01F00007FC000E0E001C0700380380780380700380F001C0F001C0F001C0F001E0F001E0F001
E0F001E0F001E07001E07003E07803E03807E01C0DE00FF9E007F1E00081C00001C00003C00003
80000380300700780700780E00701C003838001FF0000FC00013227EA018>I<000180000003C0
000003C0000003C0000007E0000007E0000007E000000FF000000CF000000CF000001CF8000018
78000018780000383C0000303C0000303C0000601E0000601E0000601E0000C00F0000C00F0000
C00F0001FFFF8001FFFF8001800780030003C0030003C0030003C0060001E0060001E0060001E0
0E0000F01F0001F0FFC00FFFFFC00FFF20237EA225>65 D<000FE010003FF83000F81C7001E006
7003C003F0078001F00F0000F01E0000F03E0000703C0000707C0000707C000030780000307800
0030F8000030F8000000F8000000F8000000F8000000F8000000F8000000F80000007800003078
0000307C0000307C0000303C0000603E0000601E0000600F0000C0078000C003C0018001E00300
00F80E00003FF800000FE0001C247DA223>67 D<FFFFFFC0FFFFFFC007800FC0078003C0078001
C0078000C0078000E0078000E007800060078060600780606007806060078060000780E0000781
E00007FFE00007FFE0000781E0000780E000078060000780600007806018078060180780001807
80003007800030078000300780003007800070078000F0078001F0078007E0FFFFFFE0FFFFFFE0
1D227EA121>69 D<FFFFFFC0FFFFFFC007800FC0078003C0078001C0078000C0078000E0078000
E00780006007800060078060600780606007806000078060000780E0000781E00007FFE00007FF
E0000781E0000780E0000780600007806000078060000780600007800000078000000780000007
80000007800000078000000780000007800000FFFE0000FFFE00001B227EA120>I<FFFCFFFC07
800780078007800780078007800780078007800780078007800780078007800780078007800780
0780078007800780078007800780078007800780FFFCFFFC0E227EA112>73
D<FFC00003FFFFE00007FF07E00007E007E00007E006F0000DE006F0000DE006F0000DE0067800
19E006780019E006780019E0063C0031E0063C0031E0063C0031E0061E0061E0061E0061E0061E
0061E0060F00C1E0060F00C1E006078181E006078181E006078181E00603C301E00603C301E006
03C301E00601E601E00601E601E00601E601E00600FC01E00600FC01E00600FC01E006007801E0
1F807801E0FFF0783FFFFFF0303FFF28227EA12D>77 D<FF800FFFFFC00FFF07C001F807E00060
07F0006006F000600678006006780060063C0060063E0060061E0060060F0060060F0060060780
600607C0600603C0600601E0600601E0600600F060060078600600786006003C6006003C600600
1E6006000F6006000F60060007E0060007E0060003E0060001E0060001E01F8000E0FFF000E0FF
F0006020227EA125>I<FFFFF000FFFFFC0007803F0007800F8007800780078003C0078003C007
8003E0078003E0078003E0078003E0078003E0078003C0078003C00780078007800F8007803F00
07FFFC0007FFF00007800000078000000780000007800000078000000780000007800000078000
000780000007800000078000000780000007800000FFFC0000FFFC00001B227EA121>80
D<FFFFE00000FFFFF8000007803E000007800F00000780078000078007C000078003E000078003
E000078003E000078003E000078003E000078003E000078007C000078007800007800F00000780
3E000007FFF8000007FFF00000078078000007803C000007801E000007800E000007800F000007
800F000007800F000007800F000007800F800007800F800007800F800007800F818007800FC180
078007C180FFFC03E300FFFC01FE000000007C0021237EA124>82 D<03F0200FFC601C0EE03803
E07001E07001E0E000E0E000E0E00060E00060E00060F00000F000007800007F00003FF0001FFE
000FFF0003FF80003FC00007E00001E00000F00000F0000070C00070C00070C00070C00070E000
60E000E0F000C0F801C0EF0380C7FF0081FC0014247DA21B>I<7FFFFFF87FFFFFF87C0780F870
0780386007801860078018E007801CC007800CC007800CC007800CC007800CC007800C00078000
000780000007800000078000000780000007800000078000000780000007800000078000000780
0000078000000780000007800000078000000780000007800000078000000780000007800003FF
FF0003FFFF001E227EA123>I<0FE0001FF8003C1C003C0E00180700000700000700000F0003FF
000FFF003F07007C0700780700F00700F00718F00718F00F18780F187C3FB83FF3F00FC3C01515
7E9418>97 D<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00000E0000
0E00000E00000E00000E1F800E7FE00FC0F00F00780E00380E003C0E001C0E001E0E001E0E001E
0E001E0E001E0E001E0E001E0E001C0E003C0F00380F80700FC1F00C7FC00C1F0017237FA21B>
I<01FE0007FF000F07801C0780380300780000700000F00000F00000F00000F00000F00000F000
00F000007800007800C03C00C01E01800F030007FE0001F80012157E9416>I<0000E0000FE000
0FE00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E003F0E007
FEE01F07E03C01E03800E07800E07000E0F000E0F000E0F000E0F000E0F000E0F000E0F000E070
00E07800E03801E03C03E01E0EF00FFCFE03F0FE17237EA21B>I<01FC0007FF000F07801C03C0
3801C07801E07000E0FFFFE0FFFFE0F00000F00000F00000F00000F000007800007800603C0060
1E00C00F838007FF0000FC0013157F9416>I<003C00FE01CF038F038F07000700070007000700
0700070007000700FFF8FFF8070007000700070007000700070007000700070007000700070007
000700070007007FF87FF8102380A20F>I<0000F001F1F807FFB80F1F381E0F001C07003C0780
3C07803C07803C07803C07801C07001E0F000F1E001FFC0019F0001800001800001C00001FFF00
0FFFC01FFFE03801F0700070E00038E00038E00038E000387000707800F01E03C00FFF8001FC00
15217F9518>I<0E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00000E00
000E00000E00000E00000E1F800E7FC00FC1E00F80F00F00700E00700E00700E00700E00700E00
700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7FFFFE7FF18237FA21B
>I<1C001E003E001E001C00000000000000000000000000000000000E00FE00FE001E000E000E
000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A227FA10E>I<01
C003E003E003E001C00000000000000000000000000000000001E00FE00FE001E000E000E000E0
00E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E060
C0F1C0F1807F003E000B2C82A10F>I<0E0000FE0000FE00001E00000E00000E00000E00000E00
000E00000E00000E00000E00000E00000E00000E0FFC0E0FFC0E07E00E03800E07000E0E000E18
000E30000E78000EF8000F9C000F1E000E0E000E07000E07800E03C00E01C00E01E00E01F0FFE3
FEFFE3FE17237FA21A>I<0E00FE00FE001E000E000E000E000E000E000E000E000E000E000E00
0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FF
E0FFE00B237FA20E>I<0E1FC07F00FE7FE1FF80FEC0F303C01F807E01E00F003C00E00E003800
E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038
00E00E003800E00E003800E00E003800E00E003800E00E003800E0FFE3FF8FFEFFE3FF8FFE2715
7F942A>I<0E1F80FE7FC0FFC1E01F80F00F00700E00700E00700E00700E00700E00700E00700E
00700E00700E00700E00700E00700E00700E00700E0070FFE7FFFFE7FF18157F941B>I<01FC00
07FF000F07801C01C03800E07800F0700070F00078F00078F00078F00078F00078F00078F00078
7000707800F03800E01C01C00F078007FF0001FC0015157F9418>I<0E1F80FE7FE0FFC1F00F00
780E00780E003C0E003C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E003C0E003C0F00
780F80700FC1F00E7FC00E1F000E00000E00000E00000E00000E00000E00000E00000E0000FFE0
00FFE000171F7F941B>I<0E3CFEFEFFCF1F8F0F060F000E000E000E000E000E000E000E000E00
0E000E000E000E000E00FFF0FFF010157F9413>114 D<0F883FF87078E038E018E018E018F000
7F003FE01FF001F8003CC01CC01CE01CE01CF018F878DFF08FC00E157E9413>I<060006000600
060006000E000E000E001E003E00FFF8FFF80E000E000E000E000E000E000E000E000E000E000E
0C0E0C0E0C0E0C0E0C0E08071803F001E00E1F7F9E13>I<0E0070FE07F0FE07F01E00F00E0070
0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00F00E01F0
07037803FE7F01F87F18157F941B>I<FFC3FEFFC3FE1E00F80E00600E00600700C00700C00700
C003818003818003C38001C30001C30000E60000E60000E600007C00007C00007C000038000038
0017157F941A>I<FF8FF9FFFF8FF9FF1E01C07C1C03C0380E03E0300E03E0300E066030070670
6007067060070E7060038C30C0038C38C0039C38C001D8198001D81D8001D81D8001F00F8000F0
0F0000F00F0000E007000060060020157F9423>I<FFC3FEFFC3FE1E00F80E00600E00600700C0
0700C00700C003818003818003C38001C30001C30000E60000E60000E600007C00007C00007C00
003800003800003000003000007000006000006000F0C000F1C000F380007F00003E0000171F7F
941A>121 D E /Fw 20 118 df<FFFF80FFFF80FFFF8011037F9016>45
D<FFFFFFE00000FFFFFFFC000007E0007F000003E0000F800003E00007C00003E00001E00003E0
0000F00003E00000780003E000007C0003E000003E0003E000003E0003E000001F0003E000001F
0003E000000F8003E000000F8003E000000FC003E000000FC003E0000007C003E0000007C003E0
000007E003E0000007E003E0000007E003E0000007E003E0000007E003E0000007E003E0000007
E003E0000007E003E0000007E003E0000007E003E0000007E003E0000007C003E0000007C003E0
000007C003E000000FC003E000000F8003E000000F8003E000000F8003E000001F0003E000001F
0003E000003E0003E000007C0003E00000780003E00000F80003E00001F00003E00007E00003E0
000F800007E0007F0000FFFFFFFC0000FFFFFFE000002B317CB033>68 D<FFFF80FFFF8007F000
03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000
03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000
03E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E000
03E00003E00003E00003E00007F000FFFF80FFFF8011317DB017>73 D<FFF00000007FF8FFF800
0000FFF807F8000000FF0003F8000000FE00037C000001BE00037C000001BE00037C000001BE00
033E0000033E00033E0000033E00033E0000033E00031F0000063E00031F0000063E00031F0000
063E00030F80000C3E00030F80000C3E00030F80000C3E000307C000183E000307C000183E0003
07C000183E000303E000303E000303E000303E000303E000303E000301F000603E000301F00060
3E000301F000C03E000300F800C03E000300F800C03E0003007C01803E0003007C01803E000300
7C01803E0003003E03003E0003003E03003E0003003E03003E0003001F06003E0003001F06003E
0003001F06003E0003000F8C003E0003000F8C003E0003000F8C003E00030007D8003E00030007
D8003E00030007D8003E00030003F0003E00030003F0003E00030003F0003E00078001E0003E00
0FC001E0007F00FFFC01E00FFFF8FFFC00C00FFFF835317CB03D>77 D<FFFFFFC000FFFFFFF800
07E000FE0003E0001F0003E0000F8003E00007C003E00003E003E00003F003E00001F003E00001
F003E00001F803E00001F803E00001F803E00001F803E00001F803E00001F803E00001F003E000
01F003E00003F003E00003E003E00007C003E0000F8003E0001F0003E000FE0003FFFFF80003FF
FFC00003E000000003E000000003E000000003E000000003E000000003E000000003E000000003
E000000003E000000003E000000003E000000003E000000003E000000003E000000003E0000000
03E000000003E000000003E000000003E000000003E000000007F0000000FFFF800000FFFF8000
0025317CB02D>80 D<007F802001FFE06007C0F8600F001CE01E000FE03C0007E0380003E07800
01E0700001E0700000E0F00000E0F00000E0F0000060F0000060F0000060F8000060F80000007C
0000007E0000003F0000003FC000001FF800000FFF800007FFF80003FFFC0000FFFF00000FFF80
0000FFC000001FE0000007E0000003F0000001F0000000F0000000F8000000F8C0000078C00000
78C0000078C0000078C0000078E0000078E0000070F00000F0F00000E0F80000E0FC0001C0FE00
0380E7800700C1F01E00C07FFC00800FF0001D337CB125>83 D<01FE00000FFFC0001C03E0003C
00F0003E0078003E003C003E001C0008001E0000001E0000001E0000001E0000001E000001FE00
003FFE0001FF1E0007F01E000F801E001F001E003E001E007C001E007C001E00F8001E0CF8001E
0CF8001E0CF8003E0CF8003E0C7C007E0C3E00DF1C1F038FB80FFF07F003F803C01E1F7D9E21>
97 D<003FC000FFF803E01C07801E0F003E1E003E1E003E3C00087C00007C0000780000F80000
F80000F80000F80000F80000F80000F80000F80000F800007C00007C00007C00003E00031E0003
1F00060F800E07C01C03F03800FFE0003F80181F7D9E1D>99 D<000001E000003FE000003FE000
0003E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0
000001E0000001E0000001E0000001E0000001E0000001E0003F81E000FFE1E003E079E007800D
E00F0007E01E0003E01E0001E03C0001E07C0001E07C0001E0780001E0F80001E0F80001E0F800
01E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0780001E07C0001E07C0001E03C
0001E03E0003E01E0007E00F000FE007801DE003E071F001FFE1FF003F01FF20327DB125>I<00
3F800000FFE00003E0F80007803C000F001E001E001E001E000F003C000F007C000F007C000F80
78000780F8000780FFFFFF80FFFFFF80F8000000F8000000F8000000F8000000F8000000F80000
00780000007C0000003C0000003C0001801E0001800F0003000F80060007C00E0001F03C0000FF
F000001FC000191F7E9E1D>I<0003E0001FF8003C3800707C00F07C01E07C01E03803C00003C0
0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000FFFFC0FFFFC003C0
0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0
0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00007E0
007FFF007FFF0016327FB114>I<000000F0007F03F801FFCF1C03C1FC1C0780F8180F0078001F
007C001E003C003E003E003E003E003E003E003E003E003E003E003E003E003E003E001E003C00
1F007C000F0078000780F0000FC1E0000DFFC0001C7F000018000000180000001C0000001C0000
001E0000000FFFF8000FFFFF0007FFFF800FFFFFC01E0007E0380001F07000007070000078E000
0038E0000038E0000038E0000038E000003870000070780000F03C0001E01E0003C007C01F0001
FFFC00003FE0001E2F7E9F21>I<07000F801F801F800F80070000000000000000000000000000
0000000000000000000780FF80FF800F8007800780078007800780078007800780078007800780
07800780078007800780078007800780078007800780078007800FC0FFF8FFF80D307EAF12>
105 D<0781FE003FC000FF87FF80FFF000FF8E07C1C0F8000F9803E3007C0007B001E6003C0007
E000FC001E0007C000F8001E0007C000F8001E0007C000F8001E00078000F0001E00078000F000
1E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E000780
00F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E
00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000
F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF0341F7E9E38>109
D<0781FE0000FF87FF8000FF8E07C0000F9803E00007B001E00007E000F00007C000F00007C000
F00007C000F000078000F000078000F000078000F000078000F000078000F000078000F0000780
00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007
8000F000078000F000078000F000078000F000078000F0000FC001F800FFFC1FFF80FFFC1FFF80
211F7E9E25>I<001FC00000FFF80001E03C0007800F000F0007801E0003C01E0003C03C0001E0
3C0001E0780000F0780000F0780000F0F80000F8F80000F8F80000F8F80000F8F80000F8F80000
F8F80000F8F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003C00F00078007C0
1F0001F07C0000FFF800001FC0001D1F7E9E21>I<0783E0FF8FF8FF9C7C0FB07C07F07C07E038
07C00007C00007C000078000078000078000078000078000078000078000078000078000078000
0780000780000780000780000780000780000780000780000780000FC000FFFE00FFFE00161F7E
9E19>114 D<01FC100FFF301E03F03800F0700070E00070E00030E00030E00030F00030F80000
7E00003FF0001FFF000FFF8003FFE0003FF00003F8000078C0003CC0003CC0001CE0001CE0001C
F0001CF00018F80038FC0070EF01E0C3FFC080FE00161F7E9E1A>I<00C00000C00000C00000C0
0000C00001C00001C00001C00003C00003C00007C0000FC0001FC000FFFFE0FFFFE003C00003C0
0003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C0
0003C00003C03003C03003C03003C03003C03003C03003C03003C03001E06001E06000F0C0007F
80001F00142C7FAB19>I<078000F000FF801FF000FF801FF0000F8001F000078000F000078000
F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0000780
00F000078000F000078000F000078000F000078000F000078000F000078000F000078000F00007
8000F000078000F000078001F000078001F000078003F000038007F00003C00EF00001F03CF800
00FFF0FF80003FC0FF80211F7E9E25>I E /Fx 5 85 df<00000000600000000000E000000000
00E00000000001E00000000001F00000000003F00000000003F00000000007F0000000000FF000
0000000FF0000000001BF0000000001BF00000000033F00000000033F00000000063F000000000
63F000000000C3F000000001C3F00000000183F00000000303F00000000303F00000000603F800
00000603F80000000C01F80000000C01F80000001801F80000003801F80000003001F800000060
01F80000006001F8000000C001F8000000C001F80000018001F8000001FFFFF8000003FFFFF800
00070001F80000060001F800000C0001F800000C0001F80000180001FC0000180001FC00003000
00FC0000300000FC0000600000FC0000E00000FC0000C00000FC0001C00000FC0003C00000FC00
0FE00003FC00FFFC003FFFE0FFF8003FFFE02B337CB234>65 D<000FFFFFFC00000FFFFFFF8000
007F000FC000007E0003E000007E0001F000007E0000F800007E00007C0000FC00007C0000FC00
003E0000FC00003E0000FC00003E0001F800001E0001F800001F0001F800001F0001F800001F00
03F000001F0003F000001F0003F000001F0003F000001F0007E000003F0007E000003F0007E000
003F0007E000003F000FC000003E000FC000007E000FC000007E000FC000007E001F8000007C00
1F800000FC001F800000FC001F800000F8003F000001F8003F000001F0003F000003F0003F0000
03E0007E000007E0007E000007C0007E00000F80007E00000F0000FC00001F0000FC00003E0000
FC00007C0000FC0000F80001F80001F00001F80007C00001F8000F800003F8007E0000FFFFFFF8
0000FFFFFFC0000030317BB035>68 D<000FFFFFFFF8000FFFFFFFF800007F0003F800007E0000
F800007E00007800007E00007800007E0000300000FC0000300000FC0000300000FC0000300000
FC0000300001F80000300001F80000300001F80000300001F80000600003F00180600003F00180
000003F00180000003F00180000007E00300000007E00300000007E00700000007E01F0000000F
FFFE0000000FFFFE0000000FC01E0000000FC00E0000001F800C0000001F800C0000001F800C00
00001F800C0000003F00180000003F00180000003F00000000003F00000000007E00000000007E
00000000007E00000000007E0000000000FC0000000000FC0000000000FC0000000000FC000000
0001F80000000001F80000000001F80000000003F800000000FFFFF0000000FFFFF00000002D31
7BB02E>70 D<000FFFFFE000000FFFFFFC0000007F003F0000007E000F8000007E0007C000007E
0003E000007E0003F00000FC0001F00000FC0001F00000FC0001F00000FC0001F00001F80003F0
0001F80003F00001F80003F00001F80003E00003F00007E00003F00007C00003F0000FC00003F0
001F800007E0003F000007E0007C000007E000F8000007E007E000000FFFFF8000000FFFFF0000
000FC00FC000000FC007E000001F8003F000001F8001F000001F8001F800001F8001F800003F00
01F800003F0001F800003F0001F800003F0001F800007E0003F000007E0003F000007E0003F000
007E0003F00000FC0007E00000FC0007E00000FC0007E00000FC0007E01801F80007E03001F800
07E03001F80007E06003F80003E060FFFFE001F1C0FFFFC000FF80000000003E002D327BB033>
82 D<07FFFFFFFFF007FFFFFFFFF00FE007F007F00F8007E001F00E0007E000E00C0007E000E0
1C0007E000E018000FC000E018000FC000E030000FC000C030000FC000C030001F8000C060001F
8000C060001F8000C060001F8000C0C0003F00018000003F00000000003F00000000003F000000
00007E00000000007E00000000007E00000000007E0000000000FC0000000000FC0000000000FC
0000000000FC0000000001F80000000001F80000000001F80000000001F80000000003F0000000
0003F00000000003F00000000003F00000000007E00000000007E00000000007E00000000007E0
000000000FC0000000000FC0000000000FC0000000000FC0000000001F80000000001F80000000
001F80000000003F800000007FFFFF8000007FFFFF8000002C3174B032>84
D E end
%%EndProlog
%%BeginSetup
%%Feature: *Resolution 300
TeXDict begin
%%EndSetup
%%Page: 0 1
bop 799 911 a Fx(D)25 b(R)g(A)g(F)g(T)225 1002 y Fw(Do)r(cumen)n(t)20
b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)621
1196 y Fv(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum)832
1323 y(Ma)o(y)h(22,)g(1996)77 1381 y(This)h(w)o(ork)f(w)o(as)h(supp)q(orted)g
(in)f(part)h(b)o(y)e(ARP)l(A)h(and)h(NSF)e(under)h(gran)o(t)h(ASC-9310330,)i
(the)192 1439 y(National)d(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o
(hnology)f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)76 1497 y(Agreemen)o(t)e(No.)22
b(CCR-8809615,)d(and)e(b)o(y)e(the)h(Commission)e(of)j(the)f(Europ)q(ean)i
(Comm)o(unit)n(y)654 1555 y(through)f(Esprit)f(pro)s(ject)g(P6643.)p
eop
%%Page: 1 2
bop 166 49 a Fu(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g(run)g(of)g
(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF)f(Final)75
106 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop
%%Page: 1 3
bop 75 361 a Ft(Chapter)31 b(4)75 568 y Fs(One)39 b(Sided)e(Comm)m
(unications)75 809 y Fr(4.1)59 b(Intro)r(duction)75 910 y Fu(Remote)11
b(Memory)f(Access)i(\(RMA\))e(extends)i(the)f(comm)o(unication)g(mec)o
(hanisms)h(of)e(MPI)h(b)o(y)g(allo)o(wing)75 967 y(transfer)i(of)g(data)f
(from)h(the)h(memory)f(of)g(one)g(pro)q(cess)h(to)f(the)g(memory)g(of)g
(another)g(pro)q(cess)h(to)f(o)q(ccur)75 1023 y(with)19 b(the)g(explicit)h
(in)o(v)o(olv)o(emen)o(t)f(of)f(one)h(of)f(the)h(t)o(w)o(o)e(pro)q(cesses)i
(only:)27 b(either)19 b(the)g(source)g(pro)q(cess)75 1080 y(\(put\))c(or)f
(the)i(destination)g(pro)q(cess)f(\(get\).)166 1136 y(This)h(mo)q(de)f(of)g
(comm)o(unication)h(facilitates)g(the)f(co)q(ding)h(of)f(some)g(applications)
i(with)e(dynami-)75 1193 y(cally)f(c)o(hanging)f(data)e(access)i(patterns)f
(but)h(where)f(the)h(data)f(distribution)i(is)f(\014xed)g(or)f(slo)o(wly)h(c)
o(hang-)75 1249 y(ing.)23 b(Eac)o(h)16 b(pro)q(cess)g(can)g(compute)g(what)g
(data)f(it)h(needs)h(to)f(access)g(or)f(up)q(date)i(at)e(other)h(pro)q
(cesses.)75 1306 y(Ho)o(w)o(ev)o(er,)j(pro)q(cesses)g(ma)o(y)g(not)f(kno)o(w)
h(whic)o(h)h(data)e(in)i(their)g(o)o(wn)e(memory)h(need)h(b)q(e)g(accessed)f
(or)75 1362 y(up)q(dated)g(b)o(y)g(remote)f(pro)q(cesses,)h(nor)f(do)g(they)h
(kno)o(w)f(the)g(iden)o(tit)o(y)i(of)e(these)g(pro)q(cesses.)30
b(Regular)75 1418 y(send/receiv)o(e)19 b(comm)o(unication)g(requires)f(matc)o
(hing)g(op)q(erations)g(b)o(y)g(sender)g(and)g(receiv)o(er.)29
b(In)18 b(our)75 1475 y(situation,)g(only)f(one)h(op)q(eration)f(out)g(of)g
(eac)o(h)g(matc)o(hing)g(pair)h(is)g(sp)q(eci\014ed.)28 b(In)17
b(order)g(to)g(issue)h(the)75 1531 y(matc)o(hing)i(op)q(erations,)h(one)g
(needs)f(to)g(compute)g(the)g(in)o(v)o(erse)h(of)f(the)g(mapping)h(sp)q
(eci\014ed)h(b)o(y)e(the)75 1588 y(transfer)g(requests.)37
b(This)21 b(requires)h(all)g(pro)q(cesses)f(to)f(participate)i(in)g(a)e(time)
i(consuming)f(global)75 1644 y(computation.)29 b(The)18 b(use)h(of)e(put)i
(or)e(get)h(calls)h(a)o(v)o(oids)f(the)g(need)i(for)d(the)h(global)h
(computation.)29 b(A)75 1701 y(generic)17 b(example)h(of)e(this)h(nature)f
(is)h(the)g(execution)g(of)f(an)h(assignmen)o(t)f(of)g(the)h(form)f
Fq(A)23 b(=)h(B\(map\))p Fu(,)75 1757 y(where)15 b Fq(map)g
Fu(is)h(a)f(p)q(erm)o(utation)g(v)o(ector,)f(and)h Fq(A,)24
b(B)15 b Fu(and)g Fq(map)g Fu(are)g(distributed)h(in)g(the)f(same)g(manner.)
166 1814 y(RMA)k(will)h(b)q(e)g(supp)q(orted)f(on)g(distributed)h(memory)e
(systems)g(b)o(y)h(an)f(\\RMA)h(agen)o(t")f(at)g(the)75 1870
y(target)e(no)q(de)j(that)e(accepts)h(RMA)g(requests)f(and)h(p)q(erforms)f
(the)h(required)h(read)f(or)f(write)h(accesses)75 1927 y(in)d(the)g(memory)f
(of)g(the)h(target)e(pro)q(cess.)20 b(A)15 b(p)q(ortable)g(implemen)o(tation)
h(ma)o(y)e(use)h(an)f(async)o(hronous)75 1983 y(receiv)o(e)k(handler)g(to)e
(implemen)o(t)i(this)f(RMA)g(agen)o(t.)24 b(Systems)16 b(with)h(dedicated)h
(put/get)f(hardw)o(are)75 2039 y(\(Cra)o(y)h(T3D\))g(should)i(b)q(e)f(to)g
(tak)o(e)f(adv)m(an)o(tage)h(of)f(that)h(hardw)o(are,)g(at)f(least)h(for)g
(simple)h(transfers.)75 2096 y(More)f(complex)i(transfers)e(ma)o(y)g(require)
h(a)g(soft)o(w)o(are)e(agen)o(t)h(at)g(the)g(target)g(no)q(de.)34
b(Systems)20 b(with)75 2152 y(comm)o(unication)h(copro)q(cessors)f(\(P)o
(aragon,)f(Meik)o(o\))h(should)h(b)q(e)g(able)f(to)g(tak)o(e)f(adv)m(an)o
(tage)h(of)g(that)75 2209 y(copro)q(cessor)13 b(in)h(order)f(to)g(run)h(the)f
(RMA)g(agen)o(t)g(without)g(in)o(terfering)h(with)g(the)f(application)i(pro)q
(ces-)75 2265 y(sor)i(at)f(the)i(target)e(no)q(de,)i(at)f(least)g(for)g
(common)g(transfers.)25 b(On)18 b(shared)f(memory)g(systems,)g(if)h(the)75
2322 y(caller)h(can)f(directly)h(access)f(the)g(memory)f(of)h(the)g(target)e
(pro)q(cess,)i(then)h(it)f(should)h(b)q(e)f(p)q(ossible)i(to)75
2378 y(implemen)o(t)f(RMA)g(without)f(an)g(RMA)g(agen)o(t:)25
b(the)18 b(caller)h(pro)q(cess)f(can)g(directly)i(cop)o(y)e(data)f(to)g(or)75
2435 y(from)d(the)i(memory)e(of)h(the)g(target)f(pro)q(cess.)166
2491 y(W)l(e)g(shall)h(denote)f(b)o(y)g Fp(origin)h Fu(the)f(pro)q(cess)g
(that)g(p)q(erforms)f(the)i(call,)g(and)f(b)o(y)g Fp(target)g
Fu(the)g(other)75 2548 y(in)o(v)o(olv)o(ed)i(pro)q(cess.)k(Th)o(us,)14
b(in)i(a)f(put)g(op)q(eration,)g(source=origin)h(and)f(destination=target;)g
(in)h(a)e(get)75 2604 y(op)q(eration,)h(source=target)f(and)i
(destination=origin.)964 2828 y(1)p eop
%%Page: 2 4
bop 75 -100 a Fu(2)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)75 49 y Fr(4.2)59 b(Initialization)75 151 y Fu(In)12
b(some)f(systems,)h(RMA)f(op)q(erations)h(will)h(run)f(faster)e(when)i
(accessing)g(sp)q(ecially)i(allo)q(cated)f(memory)75 207 y(\(e.g.,)19
b(memory)g(that)g(is)h(shared)g(b)o(y)g(the)f(other)h(pro)q(cesses)g(in)g
(the)g(comm)o(unicating)g(group,)g(on)g(an)75 264 y(SMP\).)14
b(MPI)g(pro)o(vides)h(a)f(mec)o(hanism)h(for)f(allo)q(cating)h(and)g(freeing)
g(suc)o(h)g(sp)q(ecial)h(memory)l(.)j(The)c(use)-1992 b Fn(>)15
b Fm(\(June\))75 320 y Fu(of)g(suc)o(h)g(memory)g(for)f(RMA)i(is)g(not)e
(mandatory)l(.)-117 322 y Fn(?)h Fm(\(June\))75 471 y Fl(MPI)p
160 471 14 2 v 16 w(RMA)p 275 471 V 17 w(MALLOC\(size,)f(comm)m(,)e(base\))
117 548 y Fk(IN)155 b Fl(size)519 b Fk(size)11 b(of)e(memory)e(segmen)o(t)i
(in)g(b)o(ytes)h(\(non)g(negativ)o(e)f(in)o(teger\))117 620
y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)13 b(for)i(group)h(where)g(RMA)f
(comm)o(unication)905 677 y(is)f(exp)q(ected)i(to)e(o)q(ccur)h(\(handle\))117
749 y(OUT)108 b Fl(base)505 b Fk(initial)12 b(address)j(of)f(memory)d(allo)q
(cated)i(\(in)o(teger\))75 873 y Fq(int)23 b(MPI)p 245 873
15 2 v 17 w(Rma)p 334 873 V 17 w(malloc\(int)f(size,)i(MPI)p
829 873 V 16 w(Comm)g(comm,)f(void)g(*base\))75 960 y(MPI)p
150 960 V 17 w(RMA)p 239 960 V 17 w(MALLOC\()f(SIZE,)i(COMM,)f(BASE,)g
(IERROR\))170 1016 y(INTEGER)g(SIZE,)h(COMM,)f(IERROR)170 1073
y(MPI)p 245 1073 V 17 w(AINT)h(BASE)75 1254 y Fl(MPI)p 160
1254 14 2 v 16 w(RMA)p 275 1254 V 17 w(FREE\(base\))117 1331
y Fk(IN)155 b Fl(base)505 b Fk(initial)12 b(address)j(of)f(memory)d(segmen)o
(t)i(allo)q(cated)h(b)o(y)905 1387 y Fj(MPI)p 982 1387 13 2
v 15 w(RMA)p 1088 1387 V 14 w(MALLOC)75 1512 y Fq(int)23 b(MPI)p
245 1512 15 2 v 17 w(Rma)p 334 1512 V 17 w(free\(void)g(*base\))75
1598 y(MPI)p 150 1598 V 17 w(RMA)p 239 1598 V 17 w(FREE\()g(BASE,)g(IERROR\))
170 1655 y(MPI)p 245 1655 V 17 w(AINT)h(BASE)166 1741 y Fu(The)13
b(F)l(ortran)f(t)o(yp)q(e)h Fj(MPI)p 597 1741 13 2 v 15 w(AINT)f
Fu(in)i(the)f(F)l(ortran)f(binding)j(is)f(used)g(to)e(indicate)j(a)d(\\p)q
(oin)o(ter")h(t)o(yp)q(e,)75 1797 y(or)20 b(an)h(address)f(sized)i(in)o
(teger.)36 b(This)21 b(is)g(useful)h(only)f(on)g(F)l(ortran)e(systems)h(with)
h(extensions)h(for)75 1854 y(\(Cra)o(y\))14 b(p)q(oin)o(ters.)189
1939 y Fi(R)n(ationale.)60 b Fu(The)19 b(function)h Fl(MPI)p
815 1939 14 2 v 15 w(RMA)p 929 1939 V 17 w(MALLOC)f Fu(pro)o(vides)g(the)f
(opp)q(ortunit)o(y)h(to)f(allo)q(cate)189 1995 y(memory)13
b(that)g(is)h(more)f(easily)h(shared)g(within)h(the)f(sp)q(eci\014ed)h(comm)o
(unicator)f(group.)19 b(E.g.,)12 b(on)189 2051 y(a)g(shared)h(memory)g(mac)o
(hine,)h(it)f(can)g(allo)q(cate)h(the)f(memory)g(out)f(of)h(a)f(segmen)o(t)h
(that)f(is)i(shared)189 2108 y(b)o(y)h(all)h(pro)q(cesses)f(in)i(the)e(group)
g(of)f Fl(comm)n Fu(.)189 2180 y(The)f(comm)o(unicator)h(argumen)o(t)e(is)i
(pro)o(vided)h(for)d(the)i(case)g(that)e(di\013eren)o(t)i(segmen)o(ts)f(are)g
(used)189 2236 y(for)k(comm)o(unication)h(with)g(di\013eren)o(t)g(pro)q
(cesses.)28 b(E.g.,)17 b(in)i(the)f(case)f(of)h(a)f(cluster)i(of)e(SMP's,)189
2293 y(a)h(segmen)o(t)h(shared)g(b)o(y)g(all)h(pro)q(cessors)f(in)h(the)f
(SMP)g(w)o(ould)g(b)q(e)h(used)f(if)h(comm)o(unication)f(is)189
2349 y(in)o(ternal)c(to)f(the)h(SMP)g(no)q(de,)g(but)g(an)f(arbitrary)g
(segmen)o(t)h(migh)o(t)f(b)q(e)i(used)f(if)g(comm)o(unication)189
2406 y(is)g(external.)189 2478 y(A)22 b(system)g(ma)o(y)g(ha)o(v)o(e)g
(compiler)h(\(or)f(prepro)q(cessor\))g(supp)q(orted)h(mec)o(hanisms)g(to)e
(allo)q(cate)189 2534 y(\\go)q(o)q(d")16 b(memory)l(,)g(that)g(w)o(ould)i
(allo)o(w)f(suc)o(h)g(memory)f(to)g(b)q(e)i(used)f(for)f(static)h(v)m
(ariables,)h(e.g.,)189 2591 y(static)12 b(F)l(ortran)f(v)m(ariables,)k
Fj(COMMON)d Fu(areas,)g(or)g Fj(extern)i Fu(C)f(v)m(ariables.)20
b(The)13 b(curren)o(t)f(design)i(can)189 2647 y(tak)o(e)h(adv)m(an)o(tage)g
(of)h(suc)o(h)h(mec)o(hanisms:)22 b(One)17 b(can)f(op)q(en)g(a)g(windo)o(w)g
(in)o(to)g(memory)g(allo)q(cated)189 2704 y(b)o(y)f(suc)o(h)g(mec)o(hanism,)h
(rather)e(than)h(memory)g(allo)q(cated)h(b)o(y)f Fl(MPI)p 1360
2704 V 16 w(RMA)p 1475 2704 V 17 w(MALLOC)p Fu(.)1967 46 y
Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 3 5
bop 75 -100 a Fo(4.2.)34 b(INITIALIZA)l(TION)1302 b Fu(3)189
49 y(The)14 b(function)h Fl(MPI)p 543 49 14 2 v 16 w(RMA)p
658 49 V 17 w(MALLOC)f Fu(is)h(pro)o(vided)g(for)e(p)q(ortabilit)o(y)l(.)21
b(On)15 b(systems)f(that)f(do)h(not)189 106 y(ha)o(v)o(e)k(sp)q(ecial)j
(memory)l(,)f(it)g(will)g(b)q(e)g(a)f(simple)i(wrapp)q(er)e(on)g(top)g(of)g
(the)g(C)g Fl(mallo)q(c)d Fu(function.)189 162 y(Systems)d(that)f(ha)o(v)o(e)
g(a)h(sp)q(ecial)i(mec)o(hanism)f(for)e(dynamic)i(allo)q(cation)g(of)f(\\go)q
(o)q(d")f(memory)h(can)189 219 y(easily)j(use)f(those)g(to)g(supp)q(ort)g
Fl(MPI)p 820 219 V 16 w(RMA)p 935 219 V 17 w(MALLOC)p Fu(.)f(\()p
Fi(End)i(of)g(r)n(ationale.)p Fu(\))189 319 y Fi(A)n(dvic)n(e)j(to)h(users.)
66 b Fu(The)19 b(comm)o(unicator)g(argumen)o(t)g(in)h(the)g(call)g(is)g(only)
g(a)f(\\p)q(erformance)189 375 y(hin)o(t":)k(the)17 b(allo)q(cated)g(memory)f
(ma)o(y)g(b)q(e)i(used)f(for)f(RMA)h(comm)o(unication)g(o)o(v)o(er)f(an)o(y)h
(group,)189 432 y(lik)o(e)f(an)o(y)f(other)g(memory)l(.)k(\()p
Fi(End)d(of)g(advic)n(e)g(to)h(users.)p Fu(\))189 532 y Fi(A)n(dvic)n(e)12
b(to)h(implementors.)38 b Fu(A)12 b(design)g(similar)h(to)e(the)h(design)h
(of)e(C)h(mallo)q(c)g(and)g(free)g(functions)189 588 y(has)h(to)h(b)q(e)g
(used,)g(so)g(as)f(to)h(b)q(e)g(able)h(to)e(\014nd)h(out)g(the)g(size)h(of)e
(a)g(segmen)o(t,)h(when)g(the)g(segmen)o(t)f(is)189 645 y(freed.)21
b(MPI)15 b(has)h(to)f(k)o(eep)h(a)f(list)h(of)f(curren)o(tly)h(allo)q(cated)h
(segmen)o(ts.)j(This)d(list)f(can)g(also)f(hold)189 701 y(information)10
b(on)h(the)f(t)o(yp)q(e)g(of)g(segmen)o(t)g(allo)q(cated,)i(whic)o(h)g(ma)o
(y)d(b)q(e)i(needed)h(b)o(y)e Fl(MPI)p 1645 701 V 16 w(RMA)p
1760 701 V 17 w(INIT)p Fu(.)189 758 y(\()p Fi(End)15 b(of)i(advic)n(e)f(to)g
(implementors.)p Fu(\))166 941 y Fh(Missing:)35 b Fk(Need)15
b(new)f(error)h(co)q(des.)75 1124 y Fp(Example)j(4.1)k Fu(Example)13
b(of)f(use)h(of)f Fl(MPI)p 825 1124 V 16 w(RMA)p 940 1124 V
16 w(MALLOC)p Fu(,)g(in)h(F)l(ortran)e(with)i(p)q(oin)o(ter)g(supp)q(ort.)19
b(W)l(e)75 1180 y(assume)c(4)g(b)o(yte)g Fj(REAL)p Fu(s.)75
1270 y Fq(REAL)23 b(A)75 1326 y(POINTER)g(\(P,)g(A\(100,100\)\))70
b(!)24 b(no)g(memory)f(is)g(allocated)75 1383 y(CALL)g
(MPI_RMA_MALLOC\(4*100*100,)e(COMM,)i(P,)h(IERR\))695 1439
y(!)g(memory)f(is)h(allocated)166 1528 y Fu(Since)16 b(standard)e(F)l(ortran)
f(77)h(do)q(es)g(not)g(supp)q(ort)h(p)q(oin)o(ters,)f(this)h(co)q(de)g(is)g
(not)f(F)l(ortran)f(77)h(co)q(de.)75 1585 y(It)h(is)h(exp)q(ected)g(that)f
(most)f(\(all?\))21 b(F)l(ortran)14 b(compilers)j(o\013er)d(similar)i
(facilities.)166 1685 y(The)g(initialization)j(op)q(eration)d(allo)o(ws)g
(eac)o(h)g(pro)q(cess)h(to)e(sp)q(ecify)l(,)i(in)g(a)f(collectiv)o(e)h(op)q
(eration,)f(a)75 1742 y(\\memory)11 b(windo)o(w")h(that)f(is)h(made)g
(accessible)h(to)f(accesses)g(b)o(y)f(remote)h(pro)q(cesses.)19
b(The)12 b(information)75 1798 y(is)18 b(attac)o(hed)e(to)h(a)g(comm)o
(unicator)g(returned)g(b)o(y)h(the)f(call.)27 b(The)18 b(initialization)i(op)
q(eration)d(sp)q(eci\014es)75 1855 y(the)e(lo)q(cation)h(of)f(the)g(windo)o
(w)h(in)g(memory)l(.)75 2005 y Fl(MPI)p 160 2005 V 16 w(RMA)p
275 2005 V 17 w(INIT)e(\(base,)i(size,)f(disp)p 694 2005 V
17 w(unit,)h(comm)m(,)c(new)o(comm)n(\))117 2083 y Fk(IN)155
b Fl(base)505 b Fk(initial)12 b(address)j(of)f(windo)o(w)f(\(c)o(hoice\))117
2157 y(IN)155 b Fl(size)519 b Fk(size)15 b(of)e(windo)o(w)g(in)h(b)o(ytes)g
(\(non)g(negativ)o(e)g(in)o(teger\))117 2231 y(IN)155 b Fl(disp)p
395 2231 V 18 w(unit)422 b Fk(unit)16 b(size)h(for)e(displacemen)o(ts)h(in)f
(RMA)h(op)q(erations)g(\(non-)905 2288 y(negativ)o(e)e(in)o(teger\))117
2362 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(\(handle\))117
2436 y(OUT)108 b Fl(new)o(comm)393 b Fk(new)15 b(comm)o(uni)o(cator)d(to)h(b)
q(e)i(used)g(for)e(RMA)h(\(handle\))75 2561 y Fq(int)23 b(MPI)p
245 2561 15 2 v 17 w(RMA)p 334 2561 V 17 w(init\(void*)f(base,)i(int)f(size,)
g(int)h(disp)p 1187 2561 V 16 w(unit,)g(MPI)p 1419 2561 V 16
w(Comm)g(comm,)393 2617 y(MPI)p 468 2617 V 17 w(Comm)f(*newcomm\))75
2704 y(MPI)p 150 2704 V 17 w(RMA)p 239 2704 V 17 w(INIT\(BASE,)f(SIZE,)h
(DISP)p 757 2704 V 17 w(UNIT,)g(COMM,)g(NEWCOMM,)g(IERROR\))-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 4 6
bop 75 -100 a Fu(4)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)170 49 y Fq(<type>)23 b(BASE\(*\))170 106 y(INTEGER)g(SIZE,)h(DISP)p
604 106 15 2 v 16 w(UNIT,)f(COMM,)h(NEWCOMM,)e(IERROR)166 192
y Fu(This)e(is)h(a)e(collectiv)o(e)j(call)f(executed)g(b)o(y)e(all)i(pro)q
(cesses)g(in)f(the)g(group)g(of)f Fl(comm)m Fu(.)31 b(It)20
b(returns)75 249 y(a)e(comm)o(unicator)h(that)f(can)h(b)q(e)g(used)g(b)o(y)g
(these)g(pro)q(cesses)g(to)f(p)q(erform)h(RMA)g(op)q(erations.)30
b(Eac)o(h)75 305 y(pro)q(cess)16 b(sp)q(eci\014es)i(a)e(windo)o(w)g(of)g
(existing)h(memory)e(that)g(it)i(exp)q(oses)f(to)g(remote)f(accesses)h(b)o(y)
g(other)75 362 y(pro)q(cesses,)j(for)e(RMA)h(op)q(eration)g(that)f(use)h
(comm)o(unicator)g Fl(new)o(comm)m Fu(.)25 b(The)18 b(windo)o(w)g(consists)g
(of)75 418 y Fl(size)13 b Fu(b)o(ytes,)g(starting)f(at)h(address)g
Fl(base)p Fu(.)20 b(A)13 b(pro)q(cess)g(ma)o(y)g(elect)h(to)e(exp)q(ose)i(no)
f(memory)f(b)o(y)h(sp)q(ecifying)75 474 y Fl(size)i(=)h(0)p
Fu(.)166 531 y(The)g(displacemen)o(t)h(unit)g(argumen)o(t)e(is)h(pro)o(vided)
g(to)f(facilitate)i(address)f(arithmetics)g(in)h(RMA)75 587
y(op)q(erations:)j(the)15 b(target)e(displacemen)o(t)j(argumen)o(t)e(of)h(an)
g(RMA)f(op)q(eration)h(is)h(scaled)f(b)o(y)g(the)g(factor)75
644 y Fl(disp)p 152 644 14 2 v 18 w(unit)h Fu(sp)q(eci\014ed)h(b)o(y)e(the)h
(target)e(pro)q(cess,)h(at)f(windo)o(w)i(creation.)-1396 b
Fn(>)15 b Fm(\(June\))166 700 y Fu(Eac)o(h)21 b(windo)o(w)h(is)f(asso)q
(ciated)h(with)f(a)g(coun)o(ter)h(that)e(can)i(b)q(e)g(used)f(to)g(sync)o
(hronize)i(remote)75 757 y(memory)15 b(access)g(op)q(erations.)20
b(This)c(is)f(describ)q(ed)j(in)e(more)e(detail)j(in)f(Section)g(4.6.)-1669
b Fn(?)15 b Fm(\(June\))166 813 y Fu(A)j(comm)o(unicator)g(can)g(b)q(e)h
(used)g(for)e(RMA)h(commands)g(only)h(after)e(the)i(RMA)f(initialization)75
870 y(call.)j(Suc)o(h)15 b(comm)o(unicator)f(can)h(also)f(b)q(e)h(used)g(for)
f(other)g(MPI)h(calls,)g(as)f(an)o(y)h(other)f(comm)o(unicator.)166
926 y(The)21 b(newly)h(created)f(comm)o(unicator)f Fl(new)o(comm)c
Fu(carries)21 b(all)h(the)f(information)g(cac)o(hed)h(with)75
983 y Fl(comm)m Fu(,)12 b(including)18 b(top)q(ology)d(information,)g(as)g
(if)g Fl(new)o(comm)10 b Fu(w)o(as)k(created)i(b)o(y)f(duplicating)i
Fl(comm)m Fu(.)166 1039 y(The)i(v)m(arious)h(pro)q(cesses)f(in)h(the)f(group)
g(of)f Fl(comm)13 b Fu(ma)o(y)18 b(sp)q(ecify)j(completely)f(di\013eren)o(t)f
(target)75 1095 y(windo)o(ws,)g(in)h(lo)q(cation,)g(size)f(and)g(displacemen)
o(t)h(units.)31 b(As)18 b(long)h(as)f(all)i(the)f(gets)f(and)h(puts)f(to)g(a)
75 1152 y(particular)e(pro)q(cess)f(\014t)g(its)h(sp)q(eci\014c)h(target)d
(windo)o(w)h(this)h(should)g(p)q(ose)g(no)f(problem.)166 1208
y(The)k(information)h(ab)q(out)f(the)g(set)h(of)e(windo)o(ws)i(asso)q(ciated)
f(with)h Fl(new)o(comm)14 b Fu(is)20 b(cac)o(hed)g(with)75
1265 y(the)13 b(newly)i(created)e(comm)o(unicator)g(\(in)h(the)g(same)f
(manner)g(top)q(ology)g(information)h(is)g(cac)o(hed)f(when)75
1321 y(a)k(top)q(ology)g(is)h(created\).)25 b(Th)o(us,)18 b(this)g
(information)f(is)h(transfered)f(to)f(the)i(newly)g(created)f(comm)o(u-)75
1378 y(nicator)i(if)g Fl(new)o(comm)14 b Fu(is)19 b(duplicated)i(b)o(y)e(a)f
(call)i(to)e Fl(MPI)p 1100 1378 V 16 w(COMM)p 1258 1378 V 17
w(DUP)p Fu(,)g(but)h(is)h(not)e(transfered)h(to)75 1434 y(comm)o(unicators)c
(created)g(b)o(y)g(other)g(calls.)166 1491 y(The)k(same)g(area)g(in)h(memory)
f(ma)o(y)f(app)q(ear)i(in)g(m)o(ultiple)h(windo)o(ws,)f(eac)o(h)f(asso)q
(ciated)h(with)f(a)75 1547 y(di\013eren)o(t)c(comm)o(unicator.)189
1653 y Fi(A)n(dvic)n(e)k(to)i(users.)70 b Fu(The)20 b(displacemen)o(t)h(unit)
g(argumen)o(t)f(facilitates)g(computation)g(of)g(dis-)189 1710
y(placemen)o(ts)d(in)h(a)e(heterogeneous)h(en)o(vironmen)o(t.)25
b(Supp)q(ose)17 b(that)f(eac)o(h)h(windo)o(w)g(con)o(tains)g(an)189
1766 y(arra)o(y)f(of)h(\015oats.)25 b(The)17 b(size)i(of)e
Fj(\015oat)g Fu(ma)o(y)f(di\013er)i(on)f(di\013eren)o(t)h(mac)o(hines.)27
b(Eac)o(h)17 b(pro)q(cess)g(will)189 1823 y(pro)o(vide)h(a)g(displacemen)o(t)
h(unit)g(whic)o(h)g(is)g(the)f(exten)o(t)f(of)h Fj(MPI)p 1303
1823 13 2 v 14 w(FLO)o(A)m(T)p Fu(.)f(Then,)i(en)o(tries)f(of)g(the)189
1879 y(arra)o(y)d(in)h(a)g(target)f(pro)q(cess)i(can)f(b)q(e)h(sp)q
(eci\014ed)h(b)o(y)e(their)h(index,)g(with)f(the)g(righ)o(t)g(scaling)i(done)
189 1936 y(automatically)d(b)o(y)g(MPI.)g(If)h Fl(disp)p 778
1936 14 2 v 17 w(unit)h(=)e(1)p Fu(,)g(then)g(b)o(yte)g(displacemen)o(ts)i
(are)e(used.)189 2011 y(A)d(\\go)q(o)q(d")g(windo)o(w)g(for)g(RMA)h(comm)o
(unication)g(is)g(created)f(b)o(y)h(\014rst)f(allo)q(cating)h(\\go)q(o)q(d")f
(mem-)189 2067 y(ory)g(with)i Fl(MPI)p 453 2067 V 16 w(RMA)p
568 2067 V 16 w(MALLOC)f Fu(next)h(creating)f(an)g(RMA)h(windo)o(w)f(in)h
(the)f(memory)g(allo)q(cated)189 2124 y(with)i Fl(MPI)p 377
2124 V 16 w(RMA)p 492 2124 V 17 w(INIT)p Fu(.)f(\()p Fi(End)h(of)i(advic)n(e)
f(to)h(users.)p Fu(\))189 2230 y Fi(A)n(dvic)n(e)f(to)i(implementors.)50
b Fu(In)17 b(cases)g(where)g(RMA)f(op)q(erations)h(use)g(di\013eren)o(t)g
(mec)o(hanisms)189 2286 y(in)d(di\013eren)o(t)f(memory)g(areas)f(\(e.g.,)g
(load/store)h(in)h(a)f(shared)g(memory)g(segmen)o(t,)f(and)i(an)f(asyn-)189
2343 y(c)o(hronous)21 b(handler)h(in)g(priv)m(ate)g(memory\),)g(the)f
Fl(MPI)p 1157 2343 V 16 w(RMA)p 1272 2343 V 17 w(INIT)g Fu(call)h(needs)g(to)
f(\014gure)g(out)189 2399 y(whic)o(h)c(t)o(yp)q(e)g(of)f(memory)g(is)h(used)h
(for)e(the)g(windo)o(w.)25 b(T)l(o)16 b(do)h(so,)f(MPI)h(main)o(tains,)g(in)o
(ternally)l(,)189 2456 y(the)j(list)h(of)f(memory)g(segmen)o(ts)g(allo)q
(cated)h(b)o(y)f Fl(MPI)p 1141 2456 V 16 w(RMA)p 1256 2456
V 17 w(MALLOC)p Fu(,)f(or)h(b)o(y)g(other,)h(imple-)189 2512
y(men)o(tation)e(sp)q(eci\014c,)k(mec)o(hanisms,)e(together)f(with)g
(information)g(on)g(the)g(t)o(yp)q(e)g(of)g(memory)189 2569
y(segmen)o(t)f(allo)q(cated.)33 b(When)20 b(a)g(call)g(to)f
Fl(MPI)p 1004 2569 V 16 w(RMA)p 1119 2569 V 17 w(INIT)g Fu(o)q(ccurs,)h(then)
g(MPI)f(c)o(hec)o(ks)h(whic)o(h)189 2625 y(segmen)o(t)15 b(con)o(tains)h(eac)
o(h)g(windo)o(w,)g(and)h(decide,)g(accordingly)l(,)g(whic)o(h)g(mec)o(hanism)
g(to)e(use)i(for)189 2682 y(RMA)e(op)q(erations.)20 b(\()p
Fi(End)c(of)g(advic)n(e)g(to)h(implementors.)p Fu(\))1967 46
y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 5 7
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(5)166 49 y(It)10 b(is)h(con)o(v)o(enien)o(t)g(to)f(b)q(e)h(able)g(to)f
(access)g(information)h(on)f(windo)o(w)h(parameters)e(from)h(the)g(comm)o(u-)
75 106 y(nicator)k(that)f(represen)o(ts)h(the)g(windo)o(w.)20
b(T)l(o)13 b(do)h(so,)g(three)g(prede\014ned)h(attribute)f(k)o(eys)g(are)g
(pro)o(vided.)75 205 y Fp(MPI)p 184 205 16 2 v 18 w(WINDO)o(W)p
449 205 V 19 w(BASE:)22 b Fu(Initial)17 b(address)e(of)g(lo)q(cal)h(address)
75 297 y Fp(MPI)p 184 297 V 18 w(WINDO)o(W)p 449 297 V 19 w(SIZE:)22
b Fu(size)16 b(of)f(lo)q(cal)h(windo)o(w,)f(in)h(b)o(ytes)75
389 y Fp(MPI)p 184 389 V 18 w(WINDO)o(W)p 449 389 V 19 w(DISP)p
593 389 V 18 w(UNIT:)23 b Fu(displacemen)o(t)16 b(unit)g(used)g(for)f(lo)q
(cal)h(windo)o(w)166 488 y(If)g(a)f(comm)o(unicator)h Fl(comm)10
b Fu(w)o(as)15 b(created)g(b)o(y)h(a)g(call)h(to)e Fl(MPI)p
1254 488 14 2 v 15 w(RMA)p 1368 488 V 17 w(INIT\(base,)h(size,)g(disp)p
1774 488 V 17 w(unit,)75 544 y Fg(:)8 b(:)g(:)e Fl(\))p Fu(,)20
b(then)g Fl(MPI)p 380 544 V 16 w(A)l(TTR)p 513 544 V 17 w(GET\(comm)m(,)d
(MPI)p 866 544 V 16 w(WINDO)o(W)p 1078 544 V 16 w(BASE,)j(val,)g(\015ag\))g
Fu(will)h(return)f Fl(val)f(=)h(base)75 601 y Fu(\(after)14
b(t)o(yp)q(e)h(con)o(v)o(ersion,)g(in)h(C\);)f(and)g(similarly)l(,)i(for)d
(the)i(other)f(t)o(w)o(o.)587 b Fn(>)16 b Fm(\(June\))189 700
y Fi(A)n(dvic)n(e)c(to)i(users.)38 b Fu(If)12 b Fl(comm)6 b
Fu(is)13 b(not)e(asso)q(ciated)h(with)h(RMA)f(windo)o(ws,)g(then)h
Fl(MPI)p 1640 700 V 16 w(A)l(TTR)p 1773 700 V 17 w(GET)189
757 y Fu(will)18 b(return)f Fl(\015ag)g(=)g(false)p Fu(.)25
b(Th)o(us,)17 b(these)g(attributes)f(also)h(pro)o(vide)g(a)g(mec)o(hanism)g
(for)g(\014nding)189 813 y(out)f(whether)g(a)g(comm)o(unicator)g(is)h(asso)q
(ciated)g(with)g(an)f(RMA)h(windo)o(w.)23 b(\()p Fi(End)17
b(of)g(advic)n(e)h(to)189 870 y(users.)p Fu(\))1890 924 y Fn(?)e
Fm(\(June\))75 1055 y Fr(4.3)59 b(Put)20 b(and)g(Get)f(Calls)75
1156 y Fu(The)c(execution)g(of)f(a)g(put)h(op)q(eration)g(is)g(similar)g(to)f
(the)h(execution)g(of)f(a)g(send)i(b)o(y)e(the)h(origin)g(pro)q(cess)75
1213 y(and)f(a)g(matc)o(hing)g(receiv)o(e)h(b)o(y)g(the)f(target)f(pro)q
(cess.)19 b(The)c(ob)o(vious)f(di\013erence)i(is)e(that)g(all)h(parameters)75
1269 y(are)g(pro)o(vided)h(b)o(y)f(one)g(call)i({)e(the)g(call)h(executed)g
(b)o(y)f(the)h(origin)g(pro)q(cess.)75 1390 y Ff(4.3.1)49 b(Non-blo)q(cking)
19 b(and)d(Blo)q(cking)i(Put)1890 1429 y Fn(>)e Fm(\(June\))75
1570 y Fl(MPI)p 160 1570 V 16 w(PUT\()11 b(o)o(rigin)p 402
1570 V 15 w(addr,)h(o)o(rigin)p 632 1570 V 15 w(count,)h(o)o(rigin)p
884 1570 V 15 w(datat)o(yp)q(e,)g(ta)o(rget)p 1200 1570 V 16
w(rank,)f(ta)o(rget)p 1436 1570 V 16 w(disp,)g(ta)o(rget)p
1663 1570 V 17 w(count,)g(ta)o(r-)75 1627 y(get)p 137 1627
V 17 w(datat)o(yp)q(e,)k(increment,)d(comm)n(\))117 1704 y
Fk(IN)155 b Fl(o)o(rigin)p 427 1704 V 16 w(addr)381 b Fk(address)15
b(of)f(origin)e(bu\013er)j(\(c)o(hoice\))117 1778 y(IN)155
b Fl(o)o(rigin)p 427 1778 V 16 w(count)360 b Fk(n)o(um)o(b)q(er)13
b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117 1852 y(IN)155
b Fl(o)o(rigin)p 427 1852 V 16 w(datat)o(yp)q(e)302 b Fk(datat)o(yp)q(e)14
b(of)g(elemen)o(ts)f(sen)o(t)i(\(handle\))117 1926 y(IN)155
b Fl(ta)o(rget)p 433 1926 V 17 w(rank)375 b Fk(rank)14 b(of)f(target)i(\(in)o
(teger\))117 2000 y(IN)155 b Fl(ta)o(rget)p 433 2000 V 17 w(disp)384
b Fk(displacemen)o(t)20 b(from)f(start)i(of)f(windo)o(w)g(to)h(target)g
(bu\013er)905 2057 y(\(in)o(teger\))117 2131 y(IN)155 b Fl(ta)o(rget)p
433 2131 V 17 w(count)353 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f
(receiv)o(ed)j(\(in)o(teger\))117 2205 y(IN)155 b Fl(ta)o(rget)p
433 2205 V 17 w(datat)o(yp)q(e)295 b Fk(datat)o(yp)q(e)14 b(used)h(at)f(the)g
(target)h(\(handle\))117 2279 y(IN)155 b Fl(increment)399 b
Fk(incremen)o(t)16 b(added)g(to)f(the)h(target)h(windo)o(w)d(coun)o(ter)j
(\(in)o(te-)905 2336 y(ger\))117 2410 y(IN)155 b Fl(comm)466
b Fk(comm)o(unicator)11 b(used)k(for)f(comm)o(unicati)o(on)d(\(handle\))75
2534 y Fq(int)47 b(MPI)p 269 2534 15 2 v 17 w(Put\()23 b(void)h(*origin)p
693 2534 V 16 w(addr,)f(int)h(origin)p 1092 2534 V 16 w(count,)f(MPI)p
1347 2534 V 17 w(Datatype)393 2591 y(origin)p 540 2591 V 17
w(datatype,)f(int)i(target)p 1035 2591 V 16 w(rank,)f(int)h(target)p
1434 2591 V 16 w(disp,)f(int)393 2647 y(target)p 540 2647 V
17 w(count,)g(MPI)p 796 2647 V 16 w(Datatype)g(target)p 1171
2647 V 16 w(datatype,)g(int)h(increment,)393 2704 y(MPI)p 468
2704 V 17 w(Comm)f(comm\))-32 46 y Fm(1)-32 103 y(2)-32 159
y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441
y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40
724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40
1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232
y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40
1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740
y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40
2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248
y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40
2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 6 8
bop 75 -100 a Fu(6)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)75 49 y Fq(MPI)p 150 49 15 2 v 17 w(PUT\()23 b(ORIGIN)p
430 49 V 16 w(ADDR,)h(ORIGIN)p 734 49 V 16 w(COUNT,)f(ORIGIN)p
1061 49 V 16 w(DATATYPE,)g(TARGET)p 1460 49 V 16 w(RANK,)393
106 y(TARGET)p 540 106 V 17 w(DISP,)g(TARGET)p 844 106 V 16
w(COUNT,)g(TARGET)p 1171 106 V 16 w(DATATYPE,)g(INCREMENT,)g(COMM,)393
162 y(IERROR\))170 219 y(<type>)g(ORIGIN)p 484 219 V 17 w(ADDR\(*\))170
275 y(INTEGER)g(ORIGIN)p 508 275 V 17 w(COUNT,)g(ORIGIN)p 836
275 V 16 w(DATATYPE,)g(TARGET)p 1235 275 V 16 w(RANK,)g(TARGET)p
1538 275 V 17 w(DISP,)170 332 y(TARGET)p 317 332 V 17 w(COUNT,)g(TARGET)p
645 332 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(IERROR)-117
368 y Fn(?)15 b Fm(\(June\))166 418 y Fu(T)l(ransfers)20 b
Fl(o)o(rigin)p 478 418 14 2 v 16 w(count)i Fu(successiv)o(e)g(en)o(tries)f
(of)g(the)g(t)o(yp)q(e)g(sp)q(eci\014ed)i(b)o(y)d(the)h Fl(o)o(rigin)p
1685 418 V 16 w(datat)o(yp)q(e)p Fu(,)75 474 y(starting)13
b(at)h(address)g Fl(o)o(rigin)p 568 474 V 15 w(addr)g Fu(on)g(the)g(origin)h
(no)q(de)f(to)g(the)g(target)f(no)q(de)h(sp)q(eci\014ed)i(b)o(y)e(the)g
Fl(comm)m Fu(,)75 531 y Fl(ta)o(rget)p 190 531 V 16 w(rank)19
b Fu(pair.)31 b(The)19 b(data)f(are)h(written)g(in)h(the)e(target)g(bu\013er)
h(at)f(address)h Fl(ta)o(rget)p 1624 531 V 16 w(addr)g(=)g(win-)75
587 y(do)o(w)p 154 587 V 16 w(base)c(+)e(ta)o(rget)p 427 587
V 16 w(disp)p Fn(\002)p Fl(disp)p 626 587 V 20 w(unit)p Fu(,)h(where)f
Fl(windo)o(w)p 1016 587 V 17 w(base)h Fu(and)g Fl(disp)p 1290
587 V 17 w(unit)h Fu(are)e(the)g(base)g(address)h(and)75 644
y(windo)o(w)h(displacemen)o(t)i(unit)f(sp)q(eci\014ed)h(at)e(windo)o(w)g
(initialization,)j(b)o(y)d(the)h(target)e(pro)q(cess.)166 700
y(The)h(target)f(bu\013er)h(is)h(sp)q(eci\014ed)i(b)o(y)d(the)g(argumen)o(ts)
f Fl(ta)o(rget)p 1230 700 V 17 w(count)i Fu(and)f Fl(ta)o(rget)p
1568 700 V 17 w(datat)o(yp)q(e)p Fu(.)166 757 y(The)25 b(outcome)f(is)h(as)f
(if)i(the)e(origin)i(pro)q(cess)f(executed)g(a)g(blo)q(c)o(king)h(send)f(op)q
(eration)g(with)75 813 y(argumen)o(ts)c Fl(o)o(rigin)p 412
813 V 15 w(addr,)i(o)o(rigin)p 653 813 V 15 w(count,)h(o)o(rigin)p
916 813 V 15 w(datat)o(yp)q(e,)g(ta)o(rget)p 1243 813 V 17
w(rank,)e(tag,)h(comm)m Fu(,)d(and)h(the)h(tar-)75 870 y(get)i(pro)q(cess)g
(executed)h(a)e(receiv)o(e)i(op)q(eration)f(with)h(argumen)o(ts)e
Fl(ta)o(rget)p 1398 870 V 16 w(addr,)j(ta)o(rget)p 1649 870
V 16 w(count,)h(ta)o(r-)75 926 y(get)p 137 926 V 17 w(datat)o(yp)q(e,)21
b(source,)g(tag,)f(comm)m Fu(,)d(where)i Fl(ta)o(rget)p 993
926 V 16 w(addr)h Fu(is)g(the)f(target)f(bu\013er)h(address)h(computed)75
983 y(as)15 b(explained)i(ab)q(o)o(v)o(e.)j(The)15 b(message)g(sen)o(t)g(m)o
(ust)f(\014t,)h(without)g(truncation,)g(in)h(the)g(target)d(bu\013er.)166
1039 y(The)19 b Fl(ta)o(rget)p 378 1039 V 16 w(datat)o(yp)q(e)i
Fu(argumen)o(t)d(is)h(a)g(handle)h(to)e(a)h(datat)o(yp)q(e)f(ob)s(ject)h
(de\014ned)h(at)e(the)h(origin)75 1095 y(pro)q(cess.)j(Ho)o(w)o(ev)o(er,)15
b(this)h(ob)s(ject)f(is)i(in)o(terpreted)f(at)f(the)h(target)f(pro)q(cess:)21
b(the)16 b(outcome)g(is)g(as)g(if)g(the)75 1152 y(target)c(datat)o(yp)q(e)h
(ob)s(ject)f(w)o(as)g(de\014ned)j(at)d(the)i(target)e(pro)q(cess,)h(b)o(y)g
(the)g(same)g(sequence)h(of)f(calls)h(used)75 1208 y(to)h(de\014ne)h(it)f(at)
g(the)g(origin)h(pro)q(cess.)166 1265 y(The)f(comm)o(unication)g(op)q
(eration)g(completes)h(at)e(the)h(origin)g(after)f(the)h(data)f(ha)o(v)o(e)h
(b)q(een)h(copied)75 1321 y(out)g(of)g(the)h(origin)g(bu\013er,)g(p)q
(ossibly)h(b)q(efore)f(the)f(data)g(has)h(b)q(een)g(copied)h(in)o(to)f(the)f
(target)g(memory)l(.)75 1378 y(The)d(coun)o(ter)f(asso)q(ciated)h(with)g(the)
g(target)f(windo)o(w)h(is)g(incremen)o(ted)h(b)o(y)f(the)f(v)m(alue)i
Fl(ta)o(rget)p 1675 1378 V 17 w(increment)-1994 b Fn(>)15 b
Fm(\(June\))75 1434 y Fu(after)j(data)h(ha)o(v)o(e)f(b)q(een)j(copied)f(in)o
(to)f(the)g(target)f(memory)l(.)31 b(The)19 b(seman)o(tics)h(of)e(these)i
(coun)o(ters)e(is)75 1491 y(explained)f(in)f(Section)g(4.6.)166
1547 y(The)e Fl(MPI)p 343 1547 V 16 w(Put)h Fu(call)g(is)f(\\lo)q(cal":)20
b(the)14 b(completion)h(of)e(the)h(call)h(do)q(es)g(not)e(dep)q(end)j(on)d
(the)h(state)f(of)75 1604 y(other)i(pro)q(cesses,)g(or)g(on)g(the)g(state)f
(of)h(other)g(ongoing)g(comm)o(unications.)-1500 b Fn(?)15
b Fm(\(June\))166 1660 y Fu(The)h(target)f(pro)q(cess)i(ma)o(y)e(b)q(e)i
(iden)o(tical)h(with)f(the)f(origin)h(pro)q(cess;)g(i.e.,)f(a)g(pro)q(cess)g
(ma)o(y)g(use)g(a)75 1716 y(put)f(call)i(to)d(mo)o(v)o(e)h(data)f(in)i(its)g
(memory)l(.)j(The)d(same)e(holds)i(true)f(for)g(all)h(other)f(RMA)g(calls.)
166 1773 y(The)i(function)h(returns)e(an)h(error)f(co)q(de)i(as)e(in)i(MPI.)e
(The)h(put)g(call)h(is)f(erroneous)g(if)g(the)g(target)75 1829
y(bu\013er)h(is)g(not)f(wholly)i(con)o(tained)f(in)g(the)g(target)e(windo)o
(w.)28 b(The)18 b(target)e(datat)o(yp)q(e)h(should)i(con)o(tain)75
1886 y(only)d(relativ)o(e)f(displacemen)o(ts,)i(not)e(absolute)g(addresses.)
189 1992 y Fi(R)n(ationale.)54 b Fu(A)17 b(target)g(datat)o(yp)q(e)g(is)h
(pro)o(vided)g(in)g(the)g(put)g(call)g(so)g(as)f(to)g(allo)o(w)g(data)g(to)g
(b)q(e)189 2049 y(scattered)f(\(put\))g(or)g(gathered)g(\(get\))f(in)i(the)g
(remote)f(memory)l(,)g(without)g(in)o(v)o(olv)o(emen)o(t)h(of)f(the)189
2105 y(remote)e(pro)q(cess;)h(see)h(example)g(in)g(Section)g(4.3.3.)-1200
b Fn(>)15 b Fm(\(June\))189 2180 y Fu(The)e(use)g(of)g(the)g(windo)o(w)h
(coun)o(ter)f(allo)o(ws)g(the)g(target)f(pro)q(cess)h(to)g(b)q(e)h(informed)f
(that)g(a)g(remote)189 2237 y(access)g(o)q(ccurred)i(in)f(its)g(memory)l(.)19
b(Use)13 b(of)h(this)f(mec)o(hanism)i(is)f(facultativ)o(e:)19
b(unlik)o(e)c(comm)o(uni-)189 2293 y(cation)e(via)g(send/receiv)o(e,)h(there)
f(is)h(no)e(need)i(for)e(the)h(target)f(pro)q(cess)h(to)g(p)q(erform)f(an)h
(action)g(in)189 2349 y(order)h(that)f(the)i(comm)o(unication)g(o)q(ccur;)f
(nor)g(is)h(it)g(necessary)f(that)g(the)g(target)f(pro)q(cess)i(c)o(hec)o(k)
189 2406 y(separately)k(for)f(eac)o(h)h(comm)o(unication.)33
b(Instead,)20 b(one)f(could)h(ha)o(v)o(e)f(a)g(sequence)h(of)e(put/get)189
2462 y(calls,)j(follo)o(w)o(ed)e(b)o(y)g(a)g(barrier)g(sync)o(hronization)i
(that)d(informs)h(all)h(pro)q(cesses)g(that)e(all)j(data)189
2519 y(transfers)15 b(completed.)22 b(Ho)o(w)o(ev)o(er,)15
b(it)h(is)h(imp)q(ortan)o(t)e(to)g(ha)o(v)o(e)h(an)f(explicit)j(MPI)e(call)h
(to)e(ensure)189 2575 y(that)c(access)i(to)f(the)g(target)g(bu\013er)g(b)o(y)
g(the)h(target)e(pro)q(cess)i(o)q(ccurs)g(only)g(after)f(the)g(put)h
(transfer)189 2632 y(has)i(completed.)-601 b Fn(?)15 b Fm(\(June\))1967
46 y(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 7 9
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(7)189 49 y(RMA)16 b(calls)g(targeted)f(to)h(the)g(origin)g(o)o(wn)f
(memory)h(ma)o(y)f(b)q(e)h(b)q(ene\014cial)i(when)f(they)f(allo)o(w)g(to)189
106 y(use)g(a)f(separate)h(engine)h(for)e(async)o(hronous)h(memory)f(to)g
(memory)h(transfers.)21 b(They)16 b(are)f(v)o(ery)189 162 y(useful)j(with)f
(accum)o(ulate)h(op)q(erations,)f(to)f(allo)o(w)i(atomic)f(up)q(dates)g(of)g
(lo)q(cal)h(v)m(ariables.)27 b(\()p Fi(End)189 219 y(of)16
b(r)n(ationale.)p Fu(\))189 315 y Fi(A)n(dvic)n(e)11 b(to)i(users.)37
b Fu(The)12 b Fl(ta)o(rget)p 739 315 14 2 v 16 w(datat)o(yp)q(e)h
Fu(argumen)o(t)d(is)i(a)e(handle)j(to)d(a)h(datat)o(yp)q(e)g(ob)s(ject)f
(that)g(is)189 371 y(de\014ned)j(at)f(the)g(origin)h(pro)q(cess,)g(ev)o(en)f
(though)g(it)h(de\014nes)g(a)f(data)g(la)o(y)o(out)f(in)i(the)g(target)e(pro)
q(cess)189 428 y(memory)l(.)18 b(This)13 b(causes)g(no)f(problems)h(in)g(a)f
(homogenous)g(en)o(vironmen)o(t,)h(or)f(in)h(a)f(heterogenous)189
484 y(en)o(vironmen)o(t,)i(if)g(only)h(p)q(ortable)f(datat)o(yp)q(es)g(are)f
(used.)20 b(A)14 b(datat)o(yp)q(e)g(is)g Fi(p)n(ortable)p Fu(,)g(if)h(it)f
(is)g(basic)189 541 y(datat)o(yp)q(e,)f(or)g(it)h(is)g(deriv)o(ed)h(using)f
(only)h(the)e(t)o(yp)q(e)h(constructors)f Fl(MPI)p 1426 541
V 16 w(TYPE)p 1559 541 V 17 w(CONTIGUOUS,)j Fn(>)g Fm(\(June\))189
597 y Fl(MPI)p 274 597 V 15 w(TYPE)p 406 597 V 17 w(VECTOR)d
Fu(and)e Fl(MPI)p 781 597 V 16 w(TYPE)p 914 597 V 17 w(INDEXED)g
Fu(\()p Fl(MPI)p 1237 597 V 15 w(TYPE)p 1369 597 V 17 w(SIMPLE)p
1545 597 V 16 w(STRUCT)h Fu(should)189 653 y(b)q(e)18 b(added,)i(if)e(it)h
(mak)o(es)e(it)h(to)g(MPI2\).)28 b(Suc)o(h)19 b(datat)o(yp)q(e)e(is)i(p)q
(ortable,)g(b)q(ecause)g(all)g(displace-)189 710 y(men)o(ts)i(in)i(suc)o(h)g
(datat)o(yp)q(e)e(are)h(in)h(terms)f(of)g(exten)o(ts)f(of)h(basic)h(datat)o
(yp)q(es.)40 b(Therefore,)23 b(if)189 766 y(suc)o(h)d(datat)o(yp)q(e)g
(\014ts)f(a)h(data)f(la)o(y)o(out)h(in)h(one)f(memory)l(,)h(it)f(will)i
(\014t)d(the)i(corresp)q(onding)g(data)189 823 y(la)o(y)o(out)i(in)h(another)
g(memory)l(,)h(if)f(the)g(same)g(declarations)g(where)g(used,)i(ev)o(en)e(if)
g(the)g(t)o(w)o(o)189 879 y(systems)e(ha)o(v)o(e)g(di\013eren)o(t)h(arc)o
(hitectures.)42 b(On)24 b(the)e(other)h(hand,)h(if)f(a)g(datat)o(yp)q(e)f(w)o
(as)g(con-)189 936 y(structed)11 b(using)h Fl(MPI)p 562 936
V 16 w(TYPE)p 695 936 V 16 w(HINDEXED,)g(MPI)p 1045 936 V 15
w(TYPE)p 1177 936 V 17 w(HVECTOR)h Fu(or)e Fl(MPI)p 1552 936
V 15 w(TYPE)p 1684 936 V 17 w(STRUCT)189 992 y Fu(\(and)g Fl(MPI)p
376 992 V 16 w(CONTIGUOUS)p 680 992 V 18 w(STRUCT)p Fu(\),)g(then)h(the)g
(datat)o(yp)q(e)f(con)o(tains)g(explicit)j(b)o(yte)d(displace-)17
b Fn(?)f Fm(\(June\))189 1049 y Fu(men)o(ts.)j(These)c(displacemen)o(ts)g
(are)f(unlik)o(ely)j(to)c(b)q(e)i(c)o(hosen)g(correctly)g(if)f(they)h(\014t)f
(data)g(la)o(y)o(out)189 1105 y(on)d(one)i(memory)l(,)e(but)h(are)g(used)g
(for)g(data)f(la)o(y)o(outs)g(on)h(another)g(pro)q(cess,)g(running)h(on)f(a)f
(pro)q(ces-)189 1162 y(sor)18 b(with)h(a)f(di\013eren)o(t)h(arc)o(hitecture.)
30 b(In)19 b(the)g(later)f(case,)h(one)g(should)h(use)f(the)f(mec)o(hanisms)
189 1218 y(de\014ned)d(in)f(MPI2)g(for)f(transferring)g(datat)o(yp)q(es)h
(across)f(pro)q(cesses.)19 b(\()p Fi(End)c(of)g(advic)n(e)f(to)i(users.)p
Fu(\))189 1314 y Fi(A)n(dvic)n(e)e(to)i(implementors.)39 b
Fu(Put)14 b(can)g(b)q(e)h(implemen)o(ted)h(in)f(a)e(homogeneous)h(shared)h
(memory)189 1371 y(en)o(vironmen)o(t)h(b)o(y)g(co)q(de)h(executed)g(b)o(y)f
(the)g(calling)i(pro)q(cess.)23 b(Supp)q(ose)17 b(one)g(has,)e(in)i(this)g
(en)o(vi-)189 1427 y(ronmen)o(t,)h(and)g(implemen)o(tation)i(of)d
(send/receiv)o(e,)j(using)f(memory-to-memory)e(cop)o(y)l(.)29
b(The)189 1484 y(put)17 b(op)q(eration)g(is)g(implemen)o(ted,)h(in)g
(general,)f(as)g(a)f(\\send")h(from)f(origin)i(memory)e(to)g(an)h(in-)189
1540 y(termediate)f(bu\013er,)h(implemen)o(ted)h(as)f(memory-to-memory)e(cop)
o(y)l(,)i(follo)o(w)o(ed)g(b)o(y)f(a)h(\\receiv)o(e")189 1596
y(from)11 b(the)i(in)o(termediate)g(bu\013er)f(to)g(the)g(target)g(memory)l
(,)g(implemen)o(ted)i(again)e(using)h(memory-)189 1653 y(to-memory)g(cop)o(y)
l(.)20 b(The)15 b(in)o(termediate)h(cop)o(y)e(can)h(b)q(e)h(a)o(v)o(oided)f
(\(or)e(pip)q(elin)q(ed\))k(if)f(the)e(message)189 1709 y(passing)f(implemen)
o(tation)i(has)e(the)g(abilit)o(y)i(to)d(do)i(on-the-\015y)g(datat)o(yp)q(e)e
(in)o(terpretation)i([)p Fp(?)p Fu(].)k(It)189 1766 y(can)c(also)g(b)q(e)h
(optimized)h(a)o(w)o(a)o(y)c(in)k(the)e(prev)m(alen)o(t)h(case)f(where)h
(either)g(the)f(origin)h(bu\013er)f(or)g(the)189 1822 y(target)g(bu\013er)h
(are)g(con)o(tiguous.)189 1896 y(Consider)20 b(a)f(homogenous)g(distributed)j
(memory)d(mac)o(hine,)i(where)f(a)f(DMA)g(engine)i(allo)o(ws)189
1952 y(transfer)12 b(of)g(con)o(tiguous)h(memory)g(blo)q(c)o(ks.)20
b(If)13 b(b)q(oth)g(origin)h(and)f(target)f(bu\013er)h(are)f(con)o(tiguous,)
189 2009 y(then)i(DMA)e(can)i(b)q(e)g(directly)h(used)f(for)f(the)h(put.)19
b(If)14 b(the)g(target)e(bu\013er)i(is)g(con)o(tiguous,)f(but)h(the)189
2065 y(origin)j(bu\013er)g(is)g(not,)f(then)h(a)g(memory-to-memory)e(cop)o(y)
h(can)h(b)q(e)h(used)f(to)f(gather)g(the)h(data)189 2122 y(in)o(to)g(a)g(con)
o(tiguous)h(bu\013er,)f(and)h(the)f(DMA)g(engine)i(can)e(b)q(e)h(used)g(next)
g(for)e(the)i(transfer.)26 b(If)189 2178 y(the)14 b(origin)g(bu\013er)g(is)h
(con)o(tiguous,)f(but)g(the)g(target)f(bu\013er)h(is)g(not,)g(then)g(the)g
(DMA)g(engine)h(can)189 2235 y(b)q(e)f(used)h(to)e(transfer)g(the)h(data,)f
(and)h(an)g(agen)o(t)f(on)h(the)g(target)e(no)q(de)j(is)f(needed)i(to)d
(scatter)g(the)189 2291 y(data)i(\(the)i(target)e(datat)o(yp)q(e)h(has)g(to)g
(b)q(e)h(sen)o(t)f(to)g(that)f(agen)o(t\).)23 b(In)17 b(the)f(general)h
(case,)g(data)e(is)189 2348 y(gathered)g(on)g(the)g(origin)h(no)q(de,)f(sen)o
(t)g(to)g(the)g(target)f(no)q(de,)i(and)f(scattered)g(there.)189
2421 y(A)i(generic)h(implemen)o(tation)g(of)f(put)g(on)g(top)g(of)g(comm)o
(unication)h(handlers)g(can)f(b)q(e)h(used)g(for)189 2478 y(p)q(ortabilit)o
(y)l(,)h(in)h(net)o(w)o(ork)o(ed)d(heterogenous)h(en)o(vironmen)o(ts.)30
b(A)18 b(p)q(ersisten)o(t)h(RMA)f(handler)h(is)189 2534 y(p)q(osted)c(at)g
(eac)o(h)h(pro)q(cess.)21 b(The)15 b(origin)i(pro)q(cess)e(sends)h(the)g(put)
g(request,)f(whic)o(h)h(includes)i(the)189 2591 y(target)e(displacemen)o(t,)k
(coun)o(t)d(and)h(datat)o(yp)q(e)f(argumen)o(ts,)g(to)g(the)h(target)f(pro)q
(cess,)h(where)g(it)189 2647 y(is)g(receiv)o(ed)h(b)o(y)f(the)h(RMA)f
(handler.)29 b(The)19 b(origin)f(next)g(sends)h(the)f(data,)g(using)h(a)e
(standard)189 2704 y Fl(MPI)p 274 2704 V 15 w(SEND)g Fu(call,)h(with)e(the)h
(origin)g(address,)f(coun)o(t,)g(and)h(datat)o(yp)q(e)f(argumen)o(ts.)22
b(The)17 b(RMA)-32 46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215
y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498
y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40
780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40
1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288
y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40
1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796
y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40
2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304
y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40
2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 8 10
bop 75 -100 a Fu(8)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)189 49 y Fu(handler)23 b(at)f(the)h(target)e(pro)q(cess)i(receiv)o(es)
g(the)g(put)f(request)h(and)g(p)q(osts)f(a)g(corresp)q(onding)189
106 y(receiv)o(e,)15 b(using)h(the)g(target)e(displacemen)o(t,)i(coun)o(t)f
(and)g(datat)o(yp)q(e)g(argumen)o(ts.)189 181 y(This)e(generic)i(implemen)o
(tation)f(can)g(b)q(e)g(optimized)g(in)g(v)m(arious)g(w)o(a)o(ys:)k(the)13
b(RMA)h(handler)g(can)189 237 y(b)q(e)j(p)q(olling)h(co)q(de)f(executing)h
(on)e(a)g(separate)g(comm)o(unication)h(pro)q(cessor,)f(th)o(us)g(a)o(v)o
(oiding)h(the)189 294 y(o)o(v)o(erhead)f(of)g(in)o(terrupt)h(receiv)o(es;)h
(if)g(m)o(ultiple)g(pro)q(cesses)f(run)g(on)g(the)g(same)f(no)q(de,)i(then)f
(one)189 350 y(\(k)o(ernel?\))28 b(handler,)19 b(with)e(access)h(to)f(their)h
(memories,)g(can)g(serv)o(e)f(as)h(RMA)f(handler)i(for)e(all)189
407 y(pro)q(cesses)e(on)g(the)h(no)q(de;)f(and)h(so)e(on.)189
482 y(In)d(the)f(general)h(case,)f(a)g(put)h(call)g(requires)g(the)f
(transfer)g(of)g(a)g(descriptor)g(of)g(the)g(target)f(datat)o(yp)q(e)189
538 y(to)15 b(the)g(target)g(pro)q(cess,)h(and)f(its)h(in)o(terpretation)g(b)
o(y)g(an)f(agen)o(t)g(at)g(the)h(target)f(no)q(de.)21 b(The)16
b(de-)189 595 y(scriptor)11 b(format)f(should)j(allo)o(w)e(correct)g(in)o
(terpretation,)h(so)f(as)g(to)g(recreate)g(the)h(lo)q(cal)g(datat)o(yp)q(e)
189 651 y(that)k(w)o(ould)h(b)q(e)g(created)g(b)o(y)g(the)g(same)f(sequence)i
(of)f(calls.)26 b(One)17 b(example)h(of)e(suc)o(h)h(p)q(ortable)189
708 y(represen)o(tation)h(is)h(a)f(parse)g(tree)g(of)g(the)h(datat)o(yp)q(e)f
(de\014nition,)i(with)f(the)f(argumen)o(ts)g(of)g(the)189 764
y(t)o(yp)q(e)13 b(constructor)g(calls)h(at)f(the)h(tree)f(no)q(des.)20
b(An)14 b(example)g(of)f(a)g(non)h(p)q(ortable)g(represen)o(tation)189
821 y(is)h(a)f(t)o(yp)q(emap,)g(i.e.)20 b(a)14 b(list)i(of)e(\(basic)g(datat)
o(yp)q(e,)g(displacemen)o(t\))i(pairs.)k(In)15 b(this)g(\015attened)g(rep-)
189 877 y(resen)o(tation,)k(the)g(distinction)h(b)q(et)o(w)o(een)f(b)o(yte)g
(displacemen)o(ts)h(and)f(displacemen)o(ts)h(that)e(are)189
933 y(m)o(ultiples)f(of)d(basic)i(datat)o(yp)q(e)f(exten)o(t)g(has)g(b)q(een)
h(lost.)189 1009 y(Implemen)o(tors)j(will)i(w)o(an)o(t)c(to)i(a)o(v)o(oid)f
(an)h(access)g(to)f(the)h(target)f(windo)o(w)h(coun)o(ter)g(when)g
Fl(ta)o(r-)189 1065 y(get)p 251 1065 14 2 v 16 w(increment)14
b(=)h(0)p Fu(.)189 1140 y(A)j(high)h(qualit)o(y)g(implemen)o(tation)h(will)g
(prev)o(en)o(t)e(remote)g(accesses)g(to)g(memory)g(outside)h(the)189
1197 y(windo)o(w)14 b(that)g(w)o(as)g(exp)q(osed)h(b)o(y)f(the)h(pro)q(cess.)
20 b(This,)14 b(b)q(oth)h(for)f(debugging)h(purp)q(oses,)g(and)g(for)189
1253 y(protection)e(with)h(clien)o(t-serv)o(er)h(co)q(des)f(that)f(use)h
(RMA.)f(I.e.,)g(a)h(high-qualit)o(y)h(implemen)o(tation)189
1310 y(will)h(c)o(hec)o(k)g(windo)o(w)f(b)q(ounds)h(on)g(eac)o(h)f(RMA)g
(call.)21 b(\()p Fi(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p
Fu(\))-117 1371 y Fn(>)e Fm(\(June\))75 1510 y Fl(MPI)p 160
1510 V 16 w(IPUT\()21 b(o)o(rigin)p 425 1510 V 15 w(addr,)i(o)o(rigin)p
666 1510 V 15 w(count,)h(o)o(rigin)p 929 1510 V 16 w(datat)o(yp)q(e,)g(ta)o
(rget)p 1257 1510 V 16 w(rank,)f(ta)o(rget)p 1504 1510 V 16
w(disp,)h(ta)o(rget)p 1743 1510 V 16 w(count,)75 1567 y(ta)o(rget)p
190 1567 V 16 w(datat)o(yp)q(e,)17 b(increment,)c(comm)m(,)f(o)o(rigin)p
862 1567 V 15 w(request\))117 1644 y Fk(IN)155 b Fl(o)o(rigin)p
427 1644 V 16 w(addr)381 b Fk(address)15 b(of)f(origin)e(bu\013er)j(\(c)o
(hoice\))117 1719 y(IN)155 b Fl(o)o(rigin)p 427 1719 V 16 w(count)360
b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117
1794 y(IN)155 b Fl(o)o(rigin)p 427 1794 V 16 w(datat)o(yp)q(e)302
b Fk(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(sen)o(t)i(\(handle\))117
1869 y(IN)155 b Fl(ta)o(rget)p 433 1869 V 17 w(rank)375 b Fk(rank)14
b(of)f(target)i(\(in)o(teger\))117 1944 y(IN)155 b Fl(ta)o(rget)p
433 1944 V 17 w(disp)384 b Fk(displacemen)o(t)20 b(from)f(start)i(of)f(windo)
o(w)g(to)h(target)g(bu\013er)905 2001 y(\(in)o(teger\))117
2076 y(IN)155 b Fl(ta)o(rget)p 433 2076 V 17 w(count)353 b
Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
2151 y(IN)155 b Fl(ta)o(rget)p 433 2151 V 17 w(datat)o(yp)q(e)295
b Fk(datat)o(yp)q(e)14 b(used)h(at)f(the)g(target)h(\(handle\))117
2226 y(IN)155 b Fl(increment)399 b Fk(incremen)o(t)14 b(for)f(target)i(windo)
o(w)e(coun)o(ter)i(\(in)o(teger\))117 2301 y(IN)155 b Fl(comm)466
b Fk(comm)o(unicator)11 b(used)k(for)f(comm)o(unicati)o(on)d(\(handle\))117
2377 y(OUT)108 b Fl(o)o(rigin)p 427 2377 V 16 w(request)330
b Fk(request)16 b(at)d(origin)g(pro)q(cess)j(\(handle\))75
2501 y Fq(int)47 b(MPI)p 269 2501 15 2 v 17 w(Iput\()23 b(void)h(*origin)p
717 2501 V 16 w(addr,)f(int)g(origin)p 1115 2501 V 17 w(count,)g(MPI)p
1371 2501 V 17 w(Datatype)393 2558 y(origin)p 540 2558 V 17
w(datatype,)f(int)i(target)p 1035 2558 V 16 w(rank,)f(int)h(target)p
1434 2558 V 16 w(disp,)f(int)393 2614 y(target)p 540 2614 V
17 w(count,)g(MPI)p 796 2614 V 16 w(Datatype)g(target)p 1171
2614 V 16 w(datatype,)g(int)h(increment,)393 2670 y(MPI)p 468
2670 V 17 w(Comm)f(comm,)g(MPI)p 819 2670 V 17 w(Request)g(*origin)p
1195 2670 V 16 w(request\))1967 46 y Fm(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 9 11
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(9)75 49 y Fq(MPI)p 150 49 15 2 v 17 w(IPUT\()23 b(ORIGIN)p
454 49 V 16 w(ADDR,)g(ORIGIN)p 757 49 V 17 w(COUNT,)g(ORIGIN)p
1085 49 V 16 w(DATATYPE,)g(TARGET)p 1484 49 V 16 w(RANK,)393
106 y(TARGET)p 540 106 V 17 w(DISP,)g(TARGET)p 844 106 V 16
w(COUNT,)g(TARGET)p 1171 106 V 16 w(DATATYPE,)g(INREMENT,)g(COMM,)393
162 y(ORIGIN)p 540 162 V 17 w(REQUEST,)f(IERROR\))170 219 y(<type>)h(ORIGIN)p
484 219 V 17 w(ADDR\(*\))170 275 y(INTEGER)g(ORIGIN)p 508 275
V 17 w(COUNT,)g(ORIGIN)p 836 275 V 16 w(DATATYPE,)g(TARGET)p
1235 275 V 16 w(RANK,)g(TARGET)p 1538 275 V 17 w(DISP,)170
332 y(TARGET)p 317 332 V 17 w(COUNT,)g(TARGET)p 645 332 V 16
w(DATATYPE,)g(INCREMENT,)f(COMM,)i(ORIGIN)p 1450 332 V 16 w(REQUEST,)f
(IERROR)1890 368 y Fn(?)16 b Fm(\(June\))166 418 y Fu(This)h(is)g(the)f(non)o
(blo)q(c)o(king)i(v)o(ersion)e(of)g(put.)23 b(It)17 b(is)f(iden)o(tical)j(to)
c Fl(MPI)p 1386 418 14 2 v 16 w(PUT)p Fu(,)h(except)h(that)f(it)g(has)75
474 y(an)h Fl(o)o(rigin)p 249 474 V 15 w(request)h Fu(argumen)o(t.)24
b(The)17 b(origin)h(request)f(completes)g(after)f(data)g(ha)o(v)o(e)g(b)q
(een)i(copied)g(out)75 531 y(of)d(the)h(origin)h(memory)l(,)e(p)q(ossibly)j
(b)q(efore)e(the)g(op)q(eration)g(has)g(completed)h(in)f(target)f(memory)l(.)
22 b(The)75 587 y(origin)16 b(pro)q(cess)g(ma)o(y)f(test)g(or)h(w)o(ait)f
(for)g(the)h(comm)o(unication)g(op)q(eration)g(to)f(complete)h(at)f(the)h
(origin)75 644 y(using)g(MPI)f(test)g(and)g(w)o(ait)g(constructs.)75
766 y Ff(4.3.2)49 b(Non-blo)q(cking)19 b(and)d(Blo)q(cking)i(Get)1890
805 y Fn(>)e Fm(\(June\))75 946 y Fl(MPI)p 160 946 V 16 w(GET\()11
b(o)o(rigin)p 399 946 V 16 w(addr,)h(o)o(rigin)p 630 946 V
15 w(count,)h(o)o(rigin)p 882 946 V 15 w(datat)o(yp)q(e,)h(ta)o(rget)p
1199 946 V 16 w(rank,)e(ta)o(rget)p 1435 946 V 16 w(disp,)h(ta)o(rget)p
1663 946 V 16 w(count,)g(ta)o(r-)75 1002 y(get)p 137 1002 V
17 w(datat)o(yp)q(e,)j(increment,)d(comm)n(\))117 1079 y Fk(OUT)108
b Fl(o)o(rigin)p 427 1079 V 16 w(addr)381 b Fk(address)15 b(of)f(origin)e
(bu\013er)j(\(c)o(hoice\))117 1155 y(IN)155 b Fl(o)o(rigin)p
427 1155 V 16 w(count)360 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f
(receiv)o(ed)j(\(in)o(teger\))117 1230 y(IN)155 b Fl(o)o(rigin)p
427 1230 V 16 w(datat)o(yp)q(e)302 b Fk(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)
f(receiv)o(ed)j(\(handle\))117 1305 y(IN)155 b Fl(ta)o(rget)p
433 1305 V 17 w(rank)375 b Fk(rank)14 b(of)f(target)i(\(in)o(teger\))117
1380 y(IN)155 b Fl(ta)o(rget)p 433 1380 V 17 w(disp)384 b Fk(displacemen)o(t)
16 b(from)e(windo)o(w)i(start)g(to)h(target)f(bu\013er)h(\(in-)905
1436 y(teger\))117 1512 y(IN)155 b Fl(ta)o(rget)p 433 1512
V 17 w(count)353 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)
o(teger\))117 1587 y(IN)155 b Fl(ta)o(rget)p 433 1587 V 17
w(datat)o(yp)q(e)295 b Fk(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(sen)o(t)i
(\(handle\))117 1662 y(IN)155 b Fl(increment)399 b Fk(incremen)o(t)14
b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))117
1737 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(used)k(for)f(comm)o
(unicati)o(on)d(\(handle\))75 1861 y Fq(int)47 b(MPI)p 269
1861 15 2 v 17 w(get\()23 b(void)h(*origin)p 693 1861 V 16
w(addr,)f(int)h(origin)p 1092 1861 V 16 w(count,)f(MPI)p 1347
1861 V 17 w(Datatype)393 1918 y(origin)p 540 1918 V 17 w(datatype,)f(int)i
(target,)f(int)g(target)p 1321 1918 V 16 w(disp,)h(int)393
1974 y(target)p 540 1974 V 17 w(count,)f(MPI)p 796 1974 V 16
w(Datatype)g(target)p 1171 1974 V 16 w(datatype,)g(int)h(increment,)393
2031 y(MPI)p 468 2031 V 17 w(Comm)f(comm\))75 2117 y(MPI)p
150 2117 V 17 w(GET\()g(ORIGIN)p 430 2117 V 16 w(ADDR,)h(ORIGIN)p
734 2117 V 16 w(COUNT,)f(ORIGIN)p 1061 2117 V 16 w(DATATYPE,)g(TARGET,)g
(TARGET)p 1651 2117 V 16 w(DISP,)393 2174 y(TARGET)p 540 2174
V 17 w(COUNT,)g(TARGET)p 868 2174 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)h
(IERROR\))170 2230 y(<type>)g(ORIGIN)p 484 2230 V 17 w(ADDR\(*\))170
2287 y(INTEGER)g(ORIGIN)p 508 2287 V 17 w(COUNT,)g(ORIGIN)p
836 2287 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p 1426 2287 V
16 w(DISP,)170 2343 y(TARGET)p 317 2343 V 17 w(COUNT,)g(TARGET)p
645 2343 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(IERROR)1890
2379 y Fn(?)16 b Fm(\(June\))166 2430 y Fu(Similar)j(to)e Fl(MPI)p
468 2430 14 2 v 15 w(PUT)p Fu(,)h(except)f(that)g(the)h(direction)g(of)f
(data)g(transfer)g(is)h(rev)o(ersed.)26 b(Data)16 b(are)75
2486 y(copied)h(from)e(the)h(target)f(memory)g(to)g(the)h(origin.)23
b(The)16 b(target)f(windo)o(w)h(coun)o(ter)g(is)g(up)q(dated)h(after)d
Fn(>)i Fm(\(June\))75 2542 y Fu(the)j(data)g(ha)o(v)o(e)f(b)q(een)j(copied)f
(out)f(of)f(the)i(memory)e(of)h(the)g(target)f(pro)q(cess.)32
b(The)19 b(call)h(completes)75 2599 y(when)c(the)f(data)g(ha)o(v)o(e)f(arriv)
o(ed)i(at)e(the)h(origin,)h(and)f(after)g(the)g(target)f(coun)o(ter)h(has)g
(b)q(een)h(up)q(dated.)-32 46 y Fm(1)-32 103 y(2)-32 159 y(3)-32
215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32
498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40
780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40
1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288
y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40
1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796
y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40
2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304
y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40
2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 10 12
bop 75 -100 a Fu(10)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(IGET\()22 b(o)o(rigin)p
423 49 V 16 w(addr,)i(o)o(rigin)p 666 49 V 16 w(count,)h(o)o(rigin)p
931 49 V 15 w(datat)o(yp)q(e,)h(ta)o(rget,)f(ta)o(rget)p 1410
49 V 16 w(disp,)g(ta)o(rget)p 1650 49 V 17 w(count,)g(ta)o(r-)75
106 y(get)p 137 106 V 17 w(datat)o(yp)q(e,)16 b(increment,)d(comm)n(,)e(o)o
(rigin)p 809 106 V 16 w(request\))117 183 y Fk(OUT)108 b Fl(o)o(rigin)p
427 183 V 16 w(addr)381 b Fk(address)15 b(of)f(origin)e(bu\013er)j(\(c)o
(hoice\))117 258 y(IN)155 b Fl(o)o(rigin)p 427 258 V 16 w(count)360
b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
332 y(IN)155 b Fl(o)o(rigin)p 427 332 V 16 w(datat)o(yp)q(e)302
b Fk(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(receiv)o(ed)j(\(handle\))117
407 y(IN)155 b Fl(ta)o(rget)475 b Fk(rank)14 b(of)f(target)i(\(in)o(teger\))
117 481 y(IN)155 b Fl(ta)o(rget)p 433 481 V 17 w(disp)384 b
Fk(displacemen)o(t)16 b(from)e(windo)o(w)i(start)g(to)h(target)f(bu\013er)h
(\(in-)905 538 y(teger\))117 612 y(IN)155 b Fl(ta)o(rget)p
433 612 V 17 w(count)353 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o
(t)i(\(in)o(teger\))117 687 y(IN)155 b Fl(ta)o(rget)p 433 687
V 17 w(datat)o(yp)q(e)295 b Fk(datat)o(yp)q(e)14 b(used)h(at)f(the)g(target)h
(\(handle\))117 762 y(IN)155 b Fl(increment)399 b Fk(incremen)o(t)14
b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))117
836 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(used)k(for)f(comm)o
(unicati)o(on)d(\(handle\))117 911 y(OUT)108 b Fl(o)o(rigin)p
427 911 V 16 w(request)330 b Fk(request)16 b(at)d(origin)g(pro)q(cess)j
(\(handle\))75 1035 y Fq(int)47 b(MPI)p 269 1035 15 2 v 17
w(Iget\()23 b(void)h(*origin)p 717 1035 V 16 w(addr,)f(int)g(origin)p
1115 1035 V 17 w(count,)g(MPI)p 1371 1035 V 17 w(Datatype)393
1092 y(origin)p 540 1092 V 17 w(datatype,)f(int)i(target,)f(integer)f(target)
p 1416 1092 V 17 w(disp,)h(int)393 1148 y(target)p 540 1148
V 17 w(count,)g(MPI)p 796 1148 V 16 w(Datatype)g(target)p 1171
1148 V 16 w(datatype,)g(int)h(increment,)393 1205 y(MPI)p 468
1205 V 17 w(Comm)f(comm,)g(MPI)p 819 1205 V 17 w(Request)g(*origin)p
1195 1205 V 16 w(request\))75 1291 y(MPI)p 150 1291 V 17 w(Iget\()g(ORIGIN)p
454 1291 V 16 w(ADDR,)g(ORIGIN)p 757 1291 V 17 w(COUNT,)g(ORIGIN)p
1085 1291 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p 1675 1291 V
16 w(DISP,)393 1348 y(TARGET)p 540 1348 V 17 w(COUNT,)g(TARGET)p
868 1348 V 16 w(DATATYPE,)g(INCREMENT)f(COMM,)h(ORIGIN)p 1648
1348 V 17 w(REQUEST,)393 1404 y(IERROR\))170 1461 y(<type>)g(ORIGIN)p
484 1461 V 17 w(ADDR\(*\))170 1517 y(INTEGER)g(ORIGIN)p 508
1517 V 17 w(COUNT,)g(ORIGIN)p 836 1517 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p
1426 1517 V 16 w(DISP,)170 1574 y(TARGET)p 317 1574 V 17 w(COUNT,)g(TARGET)p
645 1574 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(ORIGIN)p 1450
1574 V 16 w(REQUEST,)f(IERROR)-117 1610 y Fn(?)15 b Fm(\(June\))166
1660 y Fu(This)h(is)f(the)h(non)o(blo)q(c)o(king)g(v)o(ersion)g(of)e
Fl(MPI)p 937 1660 14 2 v 16 w(GET)p Fu(.)75 1781 y Ff(4.3.3)49
b(Examples)75 1867 y Fp(Example)18 b(4.2)k Fu(W)l(e)f(sho)o(w)e(ho)o(w)h(to)g
(implemen)o(t)h(the)g(generic)g(indirect)h(assignmen)o(t)e
Fq(A)j(=)h(B\(map\))p Fu(,)75 1923 y(where)15 b Fq(A,)23 b(B)15
b Fu(and)f Fq(map)g Fu(ha)o(v)o(e)g(the)h(same)f(distribution,)i(and)e
Fq(map)g Fu(is)h(a)f(p)q(erm)o(utation.)20 b(T)l(o)14 b(simplify)l(,)i(w)o(e)
75 1980 y(assume)f(a)g(blo)q(c)o(k)h(distribution)h(with)e(equal)h(size)g
(blo)q(c)o(ks.)75 2083 y Fq(SUBROUTINE)22 b(MAP\()i(A,)f(B,)h(map,)f(m,)h
(comm,)f(p\))75 2139 y(INTEGER)g(m,)g(map\(m\),)g(comm,)h(p)75
2195 y(REAL)f(A\(m\),)g(B\(m\))75 2308 y(INTEGER)g(otypes\(p\),)f
(oindex\(m\),)94 b(!)24 b(used)g(to)f(construct)g(origin)g(datatypes)266
2365 y(tytpes\(p\),)f(tindex\(m\),)94 b(!)24 b(used)g(to)f(construct)g
(target)g(datatypes)266 2421 y(count\(p\),)g(blen\(m\),)f(total\(p\),)266
2478 y(sizeofreal,)g(ierr)75 2591 y(!)i(This)f(part)g(does)h(the)f(work)g
(that)h(depends)f(on)g(the)h(locations)e(of)i(B.)75 2647 y(!)g(Can)f(be)h
(reused)f(while)g(this)g(does)h(not)f(change)1967 46 y Fm(1)1967
103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967
385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959
667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893
y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119
y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345
y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570
y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796
y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022
y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248
y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474
y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699
y(48)p eop
%%Page: 11 13
bop 75 -100 a Fo(4.3.)29 b(PUT)15 b(AND)g(GET)g(CALLS)1160
b Fu(11)75 49 y Fq(CALL)23 b(MPI_EXTENT\(MPI_REAL,)e(sizeofreal,)i(ierr\))75
162 y(DO)h(i=1,m)147 219 y(blen\(i\))e(=)i(1)75 275 y(END)f(DO)75
388 y(DO)h(i=1,p)147 444 y(count\(i\))e(=)i(1)75 501 y(END)f(DO)75
614 y(CALL)g(MPI_RMA_INIT\(B,)f(m*sizeofreal,)g(sizeofreal,)h(comm,)g
(getcomm,)g(ierr\))75 727 y(!)h(This)f(part)g(does)h(the)f(work)g(that)h
(depends)f(on)g(the)h(value)f(of)g(map)h(and)75 783 y(!)g(the)f(locations)g
(of)g(the)h(arrays.)75 840 y(!)g(Can)f(be)h(reused)f(while)g(these)g(do)h
(not)f(change)75 953 y(!)h(Compute)f(number)g(of)g(elements)g(to)h(be)f
(received)g(from)g(each)h(process)75 1065 y(DO)g(i=1,m)147
1122 y(j)f(=)h(map\(i\)/m)147 1178 y(count\(j\))e(=)i(count\(j\)+1)75
1235 y(END)f(DO)75 1348 y(total\(1\))g(=)g(0)75 1404 y(DO)h(i=2,p)147
1461 y(total\(i\))e(=)i(total\(i-1\))f(+)g(count\(i-1\))75
1574 y(DO)h(i=1,p)147 1630 y(count\(i\))e(=)i(0)75 1686 y(END)f(DO)75
1799 y(!)h(compute)f(origin)g(and)g(target)g(indices)g(of)h(elements)75
1856 y(!)g(received)e(from)i(each)f(process)75 1912 y(DO)h(i=1,m)147
1969 y(j)f(=)h(map\(i\)/m)147 2025 y(k)f(=)h(MOD\(map\(i\),m\))147
2082 y(count\(j\))e(=)i(count\(j\)+1)147 2138 y(oindex\(total\(j\))d(+)j
(count\(j\)\))f(=)g(i)147 2195 y(tindex\(total\(j\))e(+)j(count\(j\)\))f(=)g
(k)75 2251 y(END)g(DO)75 2364 y(!)h(create)f(origin)g(and)g(target)g
(datatypes)g(for)g(each)h(get)f(operation)75 2420 y(DO)h(i=1,p)147
2477 y(CALL)f(MPI_TYPE_INDEXED\(count\(i\))o(,)e(blen,)i
(oindex\(total\(i\)\),)743 2533 y(MPI_REAL,)g(otype\(i\),)f(ierr\))147
2590 y(CALL)h(MPI_TYPE_COMMIT\(otype\(i\),)d(ierr\))147 2646
y(CALL)j(MPI_TYPE_INDEXED\(count\(i\))o(,)e(blen,)i(tindex\(total\(i\)\),)743
2703 y(MPI_REAL,)g(ttype\(i\),)f(ierr\))-32 46 y Fm(1)-32 103
y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385
y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667
y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 12 14
bop 75 -100 a Fu(12)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)147 49 y Fq(CALL)23 b(MPI_TYPE_COMMIT\(ttype\(i\),)d(ierr\))75
106 y(END)j(DO)75 219 y(!)h(this)f(part)g(does)h(the)f(assignment)g(itself)75
332 y(DO)h(i=1,p)147 388 y(CALL)f(MPI_GET)g(\(A,)g(1,)h(otype\(i\),)e(i,)i
(0,)g(ttype\(i\),)e(0,)i(getcomm,)f(ierr\))75 444 y(END)g(DO)75
501 y(RETURN)75 607 y Fp(Example)18 b(4.3)k Fu(A)16 b(simpler)g(v)o(ersion)g
(can)f(b)q(e)h(written,)g(that)e(do)q(es)i(not)f(require)h(to)f(built)h(a)f
(datat)o(yp)q(e)75 664 y(for)e(the)g(target)f(bu\013er.)19
b(But,)13 b(then)h(one)f(needs)h(a)f(separate)f(get)h(call)h(for)f(eac)o(h)g
(elemen)o(t,)h(as)f(illustrated)75 720 y(b)q(elo)o(w.)28 b(This)19
b(co)q(de)f(is)g(m)o(uc)o(h)g(simpler,)h(but)f(m)o(uc)o(h)g(less)g(e\016cien)
o(t,)h(for)e(large)h(arra)o(ys,)f(unless)i(h/w)e(is)75 777
y(fast)d(enough)i(to)e(transfer)h(e\016cien)o(tly)h(single)h(w)o(ords.)75
883 y Fq(SUBROUTINE)22 b(MAP\()i(A,)f(B,)h(map,)f(m,)h(comm,)f(p\))75
939 y(INTEGER)g(m,)g(map\(m\),)g(comm,)h(p)75 996 y(REAL)f(A\(m\),)g(B\(m\))
75 1052 y(INTEGER)g(sizeofreal,)f(ierr)75 1165 y(CALL)h
(MPI_TYPE_EXTENT\(MPI_REAL,)e(sizeofreal,)h(ierr\))75 1222
y(CALL)h(MPI_RMA_INIT\(B,)f(m*sizeofreal,)g(sizeofreal,)h(comm,)g(getcomm,)g
(ierr\))75 1334 y(DO)h(i=1,m)147 1391 y(j)f(=)h(map\(i\)/m)147
1447 y(k)f(=)h(MOD\(map\(i\),m\))147 1504 y(CALL)f(MPI_GET\(A\(i\),)f(1,)i
(MPI_REAL,)e(j,)i(k,)f(1,)h(MPI_REAL,)f(0,)1101 1560 y(getcomm,)g(ierr\))75
1617 y(END)g(DO)75 1673 y(RETURN)75 1816 y Fr(4.4)59 b(Accumulate)18
b(F)n(unctions)75 1918 y Fu(It)e(is)g(often)g(useful)g(in)h(a)e(put)h(op)q
(eration)g(to)f(com)o(bine)i(the)f(data)f(mo)o(v)o(ed)g(to)g(the)h(target)f
(pro)q(cess)h(with)75 1974 y(the)f(data)f(that)g(resides)h(at)f(that)g(pro)q
(cess,)h(rather)f(then)h(replacing)h(the)f(data)f(there.)20
b(This)15 b(will)h(allo)o(w,)75 2031 y(for)f(example,)i(to)e(accum)o(ulate)i
(a)e(sum,)h(b)o(y)g(ha)o(ving)g(all)h(in)o(v)o(olv)o(ed)g(pro)q(cesses)g(add)
f(their)g(con)o(tribution)75 2087 y(to)f(the)g(sum)g(v)m(ariable)i(in)f(the)f
(memory)g(of)f(one)i(pro)q(cess.)-1191 b Fn(>)15 b Fm(\(June\))1967
46 y(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 13 15
bop 75 -100 a Fo(4.4.)34 b(A)o(CCUMULA)l(TE)15 b(FUNCTIONS)1029
b Fu(13)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(A)o(CCUMULA)l(TE\(o)o(rigin)p
598 49 V 15 w(addr,)25 b(o)o(rigin)p 841 49 V 15 w(count,)g(o)o(rigin)p
1105 49 V 15 w(datat)o(yp)q(e,)h(ta)o(rget)p 1434 49 V 16 w(rank,)e(ta)o
(rget)p 1682 49 V 16 w(disp,)h(ta)o(r-)75 106 y(get)p 137 106
V 17 w(count,)16 b(ta)o(rget)p 400 106 V 16 w(datat)o(yp)q(e,)g(op,)f(comm)n
(\))117 183 y Fk(IN)155 b Fl(o)o(rigin)p 427 183 V 16 w(addr)381
b Fk(initial)12 b(address)j(of)f(bu\013er)g(\(c)o(hoice\))117
258 y(IN)155 b Fl(o)o(rigin)p 427 258 V 16 w(count)360 b Fk(n)o(um)o(b)q(er)
13 b(of)h(elemen)o(ts)f(in)h(bu\013er)h(\(non)f(negativ)o(e)f(in)o(teger\))
117 333 y(IN)155 b Fl(o)o(rigin)p 427 333 V 16 w(datat)o(yp)q(e)302
b Fk(data)14 b(t)o(yp)q(e)g(of)f(eac)o(h)i(bu\013er)g(elemen)o(t)e
(\(handle\))117 408 y(IN)155 b Fl(ta)o(rget)p 433 408 V 17
w(rank)375 b Fk(rank)14 b(of)f(destination)h(pro)q(cess)i(in)d(comm)e(\(in)o
(teger\))117 483 y(IN)155 b Fl(ta)o(rget)p 433 483 V 17 w(disp)384
b Fk(displacemen)o(t)20 b(from)f(start)i(of)f(windo)o(w)g(to)h(target)g
(bu\013er)905 540 y(\(in)o(teger\))117 615 y(IN)155 b Fl(ta)o(rget)p
433 615 V 17 w(count)353 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(target)g(bu\013er)h(\(in)o(teger\))117 690 y(IN)155 b Fl(ta)o(rget)p
433 690 V 17 w(datat)o(yp)q(e)295 b Fk(datat)o(yp)q(e)14 b(used)h(at)f
(target)g(\(handle\))117 765 y(IN)155 b Fl(increment)399 b
Fk(incremen)o(t)14 b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))
117 840 y(IN)155 b Fl(op)541 b Fk(reduce)16 b(op)q(eration)e(\(handle\))117
916 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(\(handle\))75
1040 y Fq(int)23 b(MPI)p 245 1040 15 2 v 17 w(Accumulate\(void*)f(origin)p
812 1040 V 16 w(addr,)h(int)h(origin)p 1211 1040 V 16 w(count,)393
1096 y(MPI)p 468 1096 V 17 w(Datatype)f(origin)p 844 1096 V
16 w(datatype,)g(int)g(target)p 1338 1096 V 17 w(rank,)g(int)g(target)p
1737 1096 V 17 w(disp,)393 1153 y(int)h(target)p 636 1153 V
16 w(count,)f(MPI)p 891 1153 V 17 w(Datatype)g(target)p 1267
1153 V 16 w(datatype,)g(int)g(increment,)393 1209 y(MPI)p 468
1209 V 17 w(Op)h(op,)f(MPI)p 724 1209 V 17 w(Comm)g(comm\))75
1296 y(MPI)p 150 1296 V 17 w(ACCUMULATE\(ORIGIN)p 575 1296
V 15 w(ADDR,)g(ORIGIN)p 877 1296 V 16 w(COUNT,)g(ORIGIN)p 1204
1296 V 17 w(DATATYPE,)f(TARGET)p 1603 1296 V 17 w(RANK,)393
1352 y(TARGET)p 540 1352 V 17 w(DISP,)h(TARGET)p 844 1352 V
16 w(COUNT,)g(TARGET)p 1171 1352 V 16 w(DATATYPE,)g(INCREMENT,)g(OP,)393
1409 y(COMM,)g(IERROR\))170 1465 y(<type>)g(ORIGIN)p 484 1465
V 17 w(ADDR\(*\))170 1522 y(INTEGER)g(ORIGIN)p 508 1522 V 17
w(COUNT,)g(ORIGIN)p 836 1522 V 16 w(DATATYPE,)g(TARGET)p 1235
1522 V 16 w(RANK,)g(TARGET)p 1538 1522 V 17 w(DISP,)170 1578
y(TARGET)p 317 1578 V 17 w(DATATYPE,)f(INCREMENT,)h(OP,)g(COMM,)h(IERROR)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 14 16
bop 75 -100 a Fu(14)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(IA)o(CCUMULA)l(TE\(o)o(rigin)p
611 49 V 15 w(addr,)22 b(o)o(rigin)p 851 49 V 15 w(count,)h(o)o(rigin)p
1113 49 V 15 w(datat)o(yp)q(e,)g(ta)o(rget)p 1439 49 V 16 w(rank,)f(ta)o
(rget)p 1685 49 V 16 w(disp,)g(ta)o(r-)75 106 y(get)p 137 106
V 17 w(count,)16 b(ta)o(rget)p 400 106 V 16 w(datat)o(yp)q(e,)g(op,)f(comm)n
(,)c(o)o(rigin)p 931 106 V 16 w(request\))117 183 y Fk(IN)155
b Fl(o)o(rigin)p 427 183 V 16 w(addr)381 b Fk(initial)12 b(address)j(of)f
(bu\013er)g(\(c)o(hoice\))117 258 y(IN)155 b Fl(o)o(rigin)p
427 258 V 16 w(count)360 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(bu\013er)h(\(non)f(negativ)o(e)f(in)o(teger\))117 333 y(IN)155
b Fl(o)o(rigin)p 427 333 V 16 w(datat)o(yp)q(e)302 b Fk(data)14
b(t)o(yp)q(e)g(of)f(eac)o(h)i(bu\013er)g(elemen)o(t)e(\(handle\))117
408 y(IN)155 b Fl(ta)o(rget)p 433 408 V 17 w(rank)375 b Fk(rank)14
b(of)f(destination)h(pro)q(cess)i(in)d(comm)e(\(in)o(teger\))117
483 y(IN)155 b Fl(ta)o(rget)p 433 483 V 17 w(disp)384 b Fk(displacemen)o(t)20
b(from)f(start)i(of)f(windo)o(w)g(to)h(target)g(bu\013er)905
540 y(\(in)o(teger\))117 615 y(IN)155 b Fl(ta)o(rget)p 433
615 V 17 w(count)353 b Fk(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(target)g(bu\013er)h(\(in)o(teger\))117 690 y(IN)155 b Fl(ta)o(rget)p
433 690 V 17 w(datat)o(yp)q(e)295 b Fk(datat)o(yp)q(e)14 b(used)h(at)f
(target)g(\(handle\))117 765 y(IN)155 b Fl(increment)399 b
Fk(incremen)o(t)14 b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))
117 840 y(IN)155 b Fl(op)541 b Fk(reduce)16 b(op)q(eration)e(\(handle\))117
916 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(\(handle\))117
991 y(OUT)108 b Fl(o)o(rigin)p 427 991 V 16 w(request)330 b
Fk(request)16 b(at)d(origin)g(pro)q(cess)j(\(handle\))75 1115
y Fq(int)23 b(MPI)p 245 1115 15 2 v 17 w(Iaccumulate\(void*)f(origin)p
836 1115 V 16 w(addr,)h(int)h(origin)p 1235 1115 V 16 w(count,)393
1172 y(MPI)p 468 1172 V 17 w(Datatype)f(origin)p 844 1172 V
16 w(datatype,)g(int)g(target)p 1338 1172 V 17 w(rank,)g(int)g(target)p
1737 1172 V 17 w(disp,)393 1228 y(int)h(target)p 636 1228 V
16 w(count,)f(MPI)p 891 1228 V 17 w(Datatype)g(target)p 1267
1228 V 16 w(datatype,)g(int)g(increment,)393 1285 y(MPI)p 468
1285 V 17 w(Op)h(op,)f(MPI)p 724 1285 V 17 w(Comm)g(comm,)g(MPI)p
1075 1285 V 17 w(Request)g(*origin)p 1451 1285 V 16 w(request\))75
1371 y(MPI)p 150 1371 V 17 w(IACCUMULATE\(ORIGIN)p 599 1371
V 14 w(ADDR,)h(ORIGIN)p 901 1371 V 16 w(COUNT,)f(ORIGIN)p 1228
1371 V 16 w(DATATYPE,)g(TARGET)p 1627 1371 V 16 w(RANK,)393
1427 y(TARGET)p 540 1427 V 17 w(DISP,)g(TARGET)p 844 1427 V
16 w(COUNT,)g(TARGET)p 1171 1427 V 16 w(DATATYPE,)g(INCREMENT,)g(OP,)393
1484 y(COMM,)g(ORIGIN)p 683 1484 V 17 w(REQUEST,)f(IERROR)170
1540 y(<type>)h(ORIGIN)p 484 1540 V 17 w(ADDR\(*\))170 1597
y(INTEGER)g(ORIGIN)p 508 1597 V 17 w(COUNT,)g(ORIGIN)p 836
1597 V 16 w(DATATYPE,)g(TARGET)p 1235 1597 V 16 w(RANK,)g(TARGET)p
1538 1597 V 17 w(DISP,)170 1653 y(TARGET)p 317 1653 V 17 w(COUNT,)g(TARGET)p
645 1653 V 16 w(DATATYPE,)g(INCREMENT,)f(OP,)i(COMM,)f(ORIGIN)p
1545 1653 V 16 w(REQUEST,)170 1710 y(IERROR)-117 1740 y Fn(?)15
b Fm(\(June\))166 1796 y Fu(Accum)o(ulate)21 b(the)f(con)o(ten)o(ts)g(of)g
(the)h(origin)g(bu\013er)f(\(as)g(de\014ned)h(b)o(y)g Fl(o)o(rigin)p
1517 1796 14 2 v 15 w(addr)p Fu(,)g Fl(o)o(rigin)p 1756 1796
V 16 w(count)75 1853 y Fu(and)c Fl(o)o(rigin)p 274 1853 V 15
w(datat)o(yp)q(e)p Fu(\))i(to)d(the)h(bu\013er)g(sp)q(eci\014ed)i(b)o(y)e
(argumen)o(ts)f Fl(ta)o(rget)p 1343 1853 V 16 w(count)i Fu(and)f
Fl(ta)o(rget)p 1684 1853 V 17 w(datat)o(yp)q(e)p Fu(,)75 1909
y(at)j(o\013set)g Fl(ta)o(rget)p 377 1909 V 16 w(disp)p Fu(,)j(in)f(the)f
(target)e(windo)o(w)j(sp)q(eci\014ed)g(b)o(y)f Fl(ta)o(rget)p
1326 1909 V 16 w(rank)g Fu(and)g Fl(comm)m Fu(,)e(using)j(the)75
1966 y(op)q(eration)g Fl(op)p Fu(.)38 b(This)22 b(is)g(lik)o(e)h
Fl(MPI)p 719 1966 V 15 w(PUT)f Fu(except)g(that)e(data)h(is)h(com)o(bined)h
(in)o(to)e(the)g(target)g(area)75 2022 y(instead)c(of)e(o)o(v)o(erwriting)h
(it.)22 b(The)16 b(call)h(completes)g(at)e(the)h(origin)h(after)e(the)h(data)
f(ha)o(v)o(e)h(b)q(een)h(copied)-1991 b Fn(>)15 b Fm(\(June\))75
2078 y Fu(out)j(of)g(the)h(origin)g(bu\013er;)h(the)f(target)e(coun)o(ter)i
(is)g(incremen)o(ted)h(after)d(the)i(up)q(date)g(of)g(the)f(target)75
2135 y(bu\013er)d(is)h(complete.)-557 b Fn(?)15 b Fm(\(June\))166
2191 y Fu(An)o(y)d(of)h(the)f(prede\014ned)i(op)q(erations)f(for)f
Fl(MPI)p 973 2191 V 16 w(REDUCE)h Fu(can)f(b)q(e)i(used.)19
b(User-de\014ned)14 b(functions)75 2248 y(cannot)h(b)q(e)h(used.)21
b(F)l(or)14 b(example,)i(if)g Fl(op)f Fu(is)h Fl(MPI)p 914
2248 V 16 w(SUM)p Fu(,)f(eac)o(h)g(elemen)o(t)h(of)f(the)g(origin)h(bu\013er)
g(is)f(added)75 2304 y(to)g(the)g(corresp)q(onding)h(elemen)o(t)g(in)g(the)f
(target,)f(replacing)i(the)g(former)e(v)m(alue)j(in)f(the)f(target.)166
2361 y(The)e Fl(ta)o(rget)p 372 2361 V 17 w(datat)o(yp)q(e)h
Fu(should)g(b)q(e)g(a)f(a)g(basic)h(datat)o(yp)q(e,)e(or)h(a)g(deriv)o(ed)h
(datat)o(yp)q(e)f(where)g(all)h(basic)75 2417 y(comp)q(onen)o(ts)j(are)f(of)g
(the)g(same)h(basic)g(datat)o(yp)q(e.)23 b(The)17 b(op)q(eration)g
Fl(op)f Fu(applies)j(to)c(elemen)o(ts)j(of)e(that)75 2474 y(basic)g(t)o(yp)q
(e.)166 2530 y(A)i(new)g(prede\014ned)i(op)q(eration,)e Fj(MPI)p
833 2530 13 2 v 14 w(SW)m(AP)p Fu(,)f(is)h(de\014ned.)30 b(It)18
b(corresp)q(onds)g(to)g(the)g(asso)q(ciativ)o(e)75 2587 y(function)g
Fg(f)5 b Fu(\()p Fg(a;)j(b)p Fu(\))15 b(=)h Fg(b)p Fu(;)i(i.e.,)f(the)g
(curren)o(t)h(v)m(alue)g(in)g(the)g(target)e(memory)h(is)h(replaced)g(b)o(y)f
(the)h(v)m(alue)75 2643 y(supplied)g(b)o(y)d(the)g(origin.)1967
46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 15 17
bop 75 -100 a Fo(4.4.)34 b(A)o(CCUMULA)l(TE)15 b(FUNCTIONS)1029
b Fu(15)189 49 y Fi(A)n(dvic)n(e)16 b(to)h(users.)46 b Fl(MPI)p
637 49 14 2 v 16 w(PUT)16 b Fu(is)h(a)e(sp)q(ecial)j(case)e(of)g
Fl(MPI)p 1226 49 V 16 w(A)o(CCUMULA)l(TE)p Fu(,)g(with)h(the)f(op)q(era-)189
106 y(tion)h Fj(MPI)p 361 106 13 2 v 14 w(SW)m(AP)f Fu(Note,)g(ho)o(w)o(ev)o
(er,)g(that)g Fl(MPI)p 1001 106 14 2 v 16 w(PUT)h Fu(and)f
Fl(MPI)p 1296 106 V 16 w(A)o(CCUMULA)l(TE)h Fu(ha)o(v)o(e)g(sligh)o(tly)189
162 y(di\013eren)o(t)e(atomicit)o(y)g(seman)o(tics)g({)g(see)h(Section)g
(4.7.)j(\()p Fi(End)c(of)i(advic)n(e)f(to)g(users.)p Fu(\))1890
223 y Fn(>)g Fm(\(June\))166 351 y Fh(Alternativ)o(es)o(:)42
b Fk(W)m(e)16 b(could)f(allo)o(w)g(user)h(de\014ned)h(functions.)24
b(If)16 b(w)o(e)g(do)f(so,)h(w)o(e)g(w)o(an)o(t)f(to)h(lea)o(v)o(e)g(to)f
(the)75 408 y(implemen)o(tor)c(the)j(c)o(hoice)h(of)e(ev)n(aluating)f(the)j
(function)e(either)i(at)e(the)i(origin)d(or)i(at)g(the)g(target.)k(Ho)o(w)o
(ev)o(er,)c(to)75 464 y(do)c(so,)h(w)o(e)g(m)o(ust)e(pair)h(the)i(op)q
(eration)e(de\014ned)i(at)e(the)h(origin)e(with)i(an)f(op)q(eration)g
(de\014ned)i(at)e(the)h(target.)18 b(In)10 b(the)75 520 y(case)k(of)f(a)g
(collectiv)o(e)g(reduce)i(op)q(eration)e(this)g(pairing)g(is)g(implicit)d(as)
k(eac)o(h)f(pro)q(cess)i(calls)e Fj(MPI)p 1607 520 13 2 v 15
w(REDUCE)e Fk(with)75 577 y(its)k(o)o(wn)g(op)q(eration)f(argumen)o(t.)21
b(But)15 b(accum)o(ulate)f(is)h(single-sided,)g(so)g(that)g(another)h
(pairing)d(mec)o(hanism)g(is)75 633 y(needed.)19 b(W)m(e)11
b(w)o(ould)g(need)h(an)f(additional)f(collectiv)o(e)h(call,)g(to)h(create)h
(the)f(\\equiv)n(alence)f(classes")i(of)e(op)q(erators.)1890
773 y Fn(?)16 b Fm(\(June\))75 879 y Fp(Example)i(4.4)k Fu(W)l(e)d(w)o(an)o
(t)e(to)g(compute)i Fq(B)p Fu(\()p Fq(j)p Fu(\))e(=)976 847
y Fe(P)1020 890 y Fd(map)p Fc(\()p Fd(i)p Fc(\)=)p Fd(j)1158
879 y Fq(A)p Fu(\()p Fq(i)p Fu(\).)28 b(The)18 b(arra)o(ys)f
Fq(A,)24 b(B)18 b Fu(and)g Fq(map)g Fu(are)75 935 y(distributed)f(in)f(the)f
(same)g(manner.)20 b(W)l(e)15 b(write)g(the)h(simple)g(v)o(ersion.)75
1042 y Fq(SUBROUTINE)22 b(SUM\()i(A,)f(B,)h(map,)f(m,)h(comm,)f(p\))75
1098 y(INTEGER)g(m,)g(map\(m\),)g(comm,)h(p)75 1154 y(REAL)f(A\(m\),)g
(B\(m\))75 1267 y(CALL)g(MPI_RMA_INIT\(B,)f(m,)i(MPI_REAL,)e(comm,)i
(sumcomm,)e(ierr\))75 1380 y(DO)i(i=1,m)147 1437 y(j)f(=)h(map\(i\)/m)147
1493 y(k)f(=)h(MOD\(map\(i\),m\))147 1550 y(CALL)f(MPI_ACCUMULATE\(A\(i\),)e
(1,)j(MPI_REAL,)e(j,)i(k,)f(1,)h(0,)g(MPI_SUM,)1149 1606 y(sumcomm,)f(ierr\))
75 1663 y(END)g(DO)75 1719 y(RETURN)166 1825 y Fu(This)18 b(co)q(de)h(is)f
(iden)o(tical)i(to)d(the)h(co)q(de)h(in)g(Example)f(4.3,)f(page)h(12,)g
(except)g(that)f(a)h(call)h(to)e(get)75 1882 y(has)f(b)q(een)h(replaced)h(b)o
(y)e(a)f(call)j(to)d(accum)o(ulate.)23 b(\(Note)16 b(that,)f(if)h
Fq(map)g Fu(is)h(one-to-one,)f(then)g(the)g(co)q(de)75 1938
y(computes)c Fq(B)h Fu(=)g Fq(A)p Fu(\()p Fq(map)473 1922 y
Fb(\000)p Fd(1)515 1938 y Fu(\),)f(whic)o(h)h(the)f(rev)o(erse)g(assignmen)o
(t)g(to)f(he)i(one)f(computed)h(in)f(that)g(example.\))75 1995
y(In)j(a)g(similar)g(manner,)g(w)o(e)f(can)h(replace)h(the)f(call)g(to)f(get)
h(b)o(y)f(a)h(call)g(to)f(accum)o(ulate)i(in)f(Example)g(4.2,)75
2051 y(page)g(10,)g(th)o(us)g(p)q(erforming)g(the)h(computation)f(with)h
(only)g(one)f(comm)o(unication)h(b)q(et)o(w)o(een)g(an)o(y)f(pair)75
2108 y(of)g(pro)q(cesses.)75 2229 y Ff(4.4.1)49 b(RMW)17 b(op)q(erations)75
2315 y Fu(Remote)12 b(Read)p 343 2315 14 2 v 17 w(Mo)q(dify)p
501 2315 V 16 w(W)l(rite)g(op)q(erations)g(pro)o(vide)g(atomic)f(test)g(and)h
(up)q(date)g(of)f(a)g(remote)g(v)m(ariable.)75 2371 y(They)17
b(can)h(b)q(e)g(used)f(to)g(implemen)o(t)i(sync)o(hronization)f(op)q
(erations)f(suc)o(h)h(as)e(semaphores,)i(coun)o(ting)75 2428
y(semaphores,)13 b(etc.)20 b(T)l(o)13 b(simplify)i(implemen)o(tation,)g(a)e
(Read)p 1131 2428 V 17 w(Mo)q(dify)p 1289 2428 V 17 w(W)l(rite)h(op)q
(erations)g(op)q(erates)f(on)i Fn(>)h Fm(\(June\))75 2484 y
Fu(a)f(unique)i(in)o(teger)e(v)m(ariable.)-32 46 y Fm(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 16 18
bop 75 -100 a Fu(16)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(RMW\()g(inva)o(r,)f(outva)o(r,)g
(ta)o(rget,)h(disp,)h(op,)f(comm)9 b(\))117 126 y Fk(IN)155
b Fl(inva)o(r)495 b Fk(input)14 b(bu\013er)h(to)f(RMW)f(op)q(eration)g(\(c)o
(hoice\))117 202 y(OUT)108 b Fl(outva)o(r)467 b Fk(output)14
b(bu\013er)h(of)e(RMW)h(op)q(eration)f(\(c)o(hoice\))117 277
y(IN)155 b Fl(ta)o(rget)475 b Fk(rank)14 b(of)f(target)i(pro)q(cess)g(\(in)o
(teger\))117 352 y(IN)155 b Fl(disp)513 b Fk(displacemen)o(t)13
b(to)h(target)g(bu\013er)h(\(in)o(teger\))117 427 y(IN)155
b Fl(op)541 b Fk(read-mo)q(dify-write)12 b(op)q(eration)i(\(handle\))117
502 y(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(\(handle\))75
627 y Fq(int)23 b(MPI)p 245 627 15 2 v 17 w(Rmw\()h(int)f(invar,)g(int)g
(*outvar,)g(int)h(target,)f(int)g(disp,)g(MPI)p 1551 627 V
17 w(Op)h(op,)393 683 y(MPI)p 468 683 V 17 w(Comm)f(comm)h(\))75
770 y(MPI)p 150 770 V 17 w(RMW\()f(INVAR,)g(OUTVAR,)g(TARGET,)g(DISP,)g(OP,)h
(COMM,)f(IERROR\))170 826 y(INTEGER)g(INVAR,)g(OUTVAR,)g(TARGET,)g(DISP,)g
(OP,)h(COMM,)f(IERROR)166 912 y Fu(The)15 b(call)i(has)e(the)g(same)g
(e\013ect)g(as)g(an)g(atomic)g(execution)h(of)f(the)g(co)q(de)75
1006 y Fq(outvar[i])94 b(=)24 b(targetvar[disp];)75 1063 y(targetvar[i])e(=)i
(targetvar[disp])e(op)h(invar;)75 1157 y Fu(\(assuming)15 b(a)g(windo)o(w)h
(that)e(is)i(an)f(arra)o(y)f(of)h(in)o(tegers)h(and)f Fq(disp)p
1228 1157 V 17 w(unit)23 b(=)h(sizeof\(int\))p Fu(\).)18 b(I.e.,)d(the)75
1213 y(call)e(returns)f(the)g(old)h(v)m(alue)g(of)e(the)h(target)f(v)m
(ariable)j(and)e(up)q(date)g(it,)h(according)f(the)g(to)g(the)g(op)q(eration)
75 1269 y Fl(op)p Fu(.)19 b(The)12 b(call)g(completes)h(after)d(the)i(target)
f(v)m(ariable)i(has)e(b)q(een)i(up)q(dated)f(and)g(the)g(old)g(v)m(alue)h
(returned)75 1326 y(at)i(the)g(origin.)166 1382 y(The)g(follo)o(wing)h(op)q
(erations)g(are)e(supp)q(orted:)21 b Fj(MPI)p 1057 1382 13
2 v 14 w(SW)m(AP)m(,)12 b(MPI)p 1281 1382 V 14 w(SUM)k Fu(and)f
Fj(MPI)p 1562 1382 V 14 w(BAND)p Fu(.)166 1439 y(This)h(call)h(can)e(b)q(e)i
(used)f(to)e(implemen)o(t)j(a)e(test)g(and)h(set)f(op)q(eration.)21
b(Assume)16 b(that)f Fl(ta)o(rgetva)o(r)f Fu(is)75 1495 y(a)j(Bo)q(olean)i
(lo)q(c)o(k.)27 b(A)18 b(call)h(to)e Fl(MPI)p 693 1495 14 2
v 16 w(RMW\(v,)g Fg(:)8 b(:)g(:)e Fl(,)18 b(MPI)p 1064 1495
V 15 w(SW)l(AP)l(,)h Fg(:)8 b(:)g(:)d Fl(\))17 b Fu(with)h
Fl(v=1)g Fu(sets)f(the)h(lo)q(c)o(k)g(to)f(1)75 1552 y(and)e(returns)f(the)g
(previous)i(lo)q(c)o(k)f(v)m(alue.)20 b(The)15 b(lo)q(c)o(k)g(call)h
(succeeded)g(if)f(the)f(previous)h(v)m(alue)h(w)o(as)e(0.)19
b(A)75 1608 y(call)g(to)e Fl(MPI)p 304 1608 V 15 w(RMW\(v,)h
Fg(:)8 b(:)g(:)d Fl(,)18 b(MPI)p 674 1608 V 16 w(SW)l(AP)l(,)g
Fg(:)8 b(:)g(:)e Fl(,)18 b(MPI)p 1014 1608 V 15 w(SW)l(AP)l(,)g
Fg(:)8 b(:)g(:)e Fl(\))17 b Fu(with)h Fl(v=0)g Fu(resets)f(the)h(lo)q(c)o(k)g
(to)f(0.)75 1665 y(The)g(call)i(can)e(also)g(b)q(e)g(used)h(to)f(implemen)o
(t)h(a)f(F)l(etc)o(h)p 1037 1665 V 16 w(Add)h(op)q(eration)f(\(with)g
Fj(MPI)p 1557 1665 13 2 v 14 w(SUM)p Fu(\),)g(coun)o(ting)75
1721 y(lo)q(c)o(ks,)e(etc.)189 1827 y Fi(R)n(ationale.)58 b
Fu(E\016cien)o(t)18 b(sync)o(hronization)i(requires)e(the)h(use)f(of)g(RMW)g
(op)q(erations,)h(suc)o(h)f(as)189 1884 y(de\014ned)d(here;)g(it)f(requires,)
h(in)f(a)g(preemptiv)o(e)h(execution)g(en)o(vironmen)o(t,)f(in)o(tegration)g
(with)h(the)189 1940 y(sc)o(heduler.)29 b(W)l(e)18 b(ha)o(v)o(e)g(not)g
(attempted)f(to)g(handle)j(the)e(later,)g(as)g(MPI)f(has)h(so)g(far)f
(refrained)189 1997 y(from)h(pro)o(viding)j(sc)o(heduling)g(services,)g(or)e
(from)f(making)i(assumptions)f(on)h(the)f(underlying)189 2053
y(sc)o(heduling)e(p)q(olicies.)22 b(\()p Fi(End)16 b(of)g(r)n(ationale.)p
Fu(\))166 2235 y Fh(Alternativ)o(es)o(:)47 b Fk(W)m(e)17 b(ma)o(y)e(prefer)j
(to)f(use)h Fj(long)f(int)g Fk(argumen)o(ts,)g(rather)h(than)f
Fj(int)p Fk(.)27 b(Or)18 b(should)f(it)f(b)q(e)75 2285 y Fj(MPI)p
152 2285 V 14 w(Aint)p Fk(?)166 2342 y(W)m(e)f(could)g(ha)o(v)o(e)h(a)f(blo)q
(c)o(king)f(RMW)h(call,)g(where)h(b)q(oth)g Fj(inva)o(r)g Fk(and)f
Fj(outva)o(r)h Fk(are)g(inputs:)21 b(the)16 b(call)f(blo)q(c)o(ks)75
2398 y(un)o(til)10 b(the)i(target)f(v)n(ariable)f(equals)h(to)g
Fj(outva)o(r)p Fk(.)18 b(E.g.,)10 b(a)g(call)h(to)g Fj(MPI)p
1146 2398 V 14 w(W)o(ait)p 1243 2398 V 15 w(RMW\(1,)g(b,)h
Fa(:)7 b(:)g(:)e Fj(,)11 b(MPI)p 1618 2398 V 14 w(SUM\))g Fk(returns)75
2455 y(when)i(the)h(v)n(alue)e(of)g(the)h(target)g(v)n(ariable)f(is)h(b,)f
(and)h(then)g(incremen)o(ts)g(it)f(b)o(y)h(one.)18 b(Tw)o(o)12
b(suc)o(h)h(coun)o(ters)h(can)f(b)q(e)75 2511 y(used)i(to)f(implem)o(en)o(t)e
(a)h(fair)g(semaphore,)g(using)h(the)h(co)q(de)f(sho)o(wn)g(b)q(elo)o(w.)1967
46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 17 19
bop 75 -100 a Fo(4.5.)34 b(COMPLETION)16 b(OF)f(RMA)h(OPERA)l(TIONS)807
b Fu(17)75 49 y Fq(void)23 b(lock\(\))75 106 y({)75 162 y(/*)h(get)f(ticket)g
(in)h(the)f(queue)g(*/)75 219 y(MPI_RMW\(1,)f(&myticket,)h(target,)g
(waitcount,)f(MPI_SUM,)h(comm\);)75 275 y(/*)h(Wait)f(for)g(turn)h(*/)75
332 y(MPI_Wait_RMW\(1,)e(myticket,)g(target,)h(servicedcount,)f(MPI_SUM,)h
(comm\);)75 388 y(})75 501 y(void)g(unlock\(\))75 557 y({)75
614 y(MPI_RMW\(1,)f(&dontcare,)h(target,)g(servicedcount,)f(MPI_SUM,)h
(comm\);)75 670 y(})1890 724 y Fn(?)16 b Fm(\(June\))75 863
y Fr(4.5)59 b(Completion)19 b(of)g(RMA)g(Op)r(erations)75 965
y Fu(The)d(de\014nition)i(of)e(the)g(put)h(and)f(accum)o(ulate)h(op)q
(erations)f(implies)i(that)e(the)g(comm)o(unication)h(ma)o(y)75
1021 y(con)o(tin)o(ue)d(after)e(the)i(call)g(completed)g(at)f(the)g(origin.)
20 b(Also,)14 b(MPI)f(do)q(es)g(not)g(guaran)o(tee)g(the)g(execution)75
1078 y(order,)e(at)g(the)g(target)f(memory)l(,)i(or)e(t)o(w)o(o)g(subsequen)o
(tly)j(issued)f(puts.)19 b(This)11 b(is)h(similar)h(to)d(the)i(situation)75
1134 y(in)19 b(mo)q(dern)f(shared)g(memory)g(m)o(ultipro)q(cessors,)g(where)h
(the)f(completion)h(of)e(store)g(op)q(eration)i(do)q(es)75
1191 y(not)c(imply)h(that)e(the)h(stored)g(v)m(alue)h(is)g(visible)h(to)d
(other)h(pro)q(cessors,)f(and)h(where)g(stores)g(ma)o(y)f(go)g(out)75
1247 y(of)i(order.)22 b(On)17 b(suc)o(h)g(systems,)e(a)h(separate)g(sync)g
(call)h(is)g(needed)h(to)d(ensure)i(that)e(a)h(store)g(op)q(eration)75
1303 y(is)i(\\committed",)e(and)i(will)h(precede)f(subsequen)o(t)f(stores)g
(in)h(global)g(memory)l(.)25 b(A)17 b(useful)h(construct)75
1360 y(is)g(a)g(collectiv)o(e)i(call)f(that)e(ensures)i(that)e(all)i
(previously)g(issued)g(RMA)f(calls)h(ha)o(v)o(e)f(completed)h(in)f(a)75
1416 y(group.)i(This)c(will)h(allo)o(w)f(to)f(implemen)o(t)h(algorithms)g
(that)e(consists)i(of)f(successiv)o(e)h(phases,)g(where)f(a)75
1473 y(new)e(phase)g(starts)e(when)i(all)h(comm)o(unications)f(p)q(ertaining)
h(to)f(the)f(previous)i(phase)f(ha)o(v)o(e)f(completed)75 1529
y(\(in)k(the)f(st)o(yle)g(of)g(the)g(Bulk)i(Sync)o(hronous)e(P)o(arallel)h
(mo)q(del\).)166 1586 y(The)f(MPI)h(collectiv)o(e)h(function)f
Fl(MPI)p 824 1586 14 2 v 16 w(BARRIER\(comm)n(\))p Fu(,)c(when)k(called)h
(with)e(a)g(comm)o(unicator)75 1642 y Fl(comm)10 b Fu(used)17
b(for)f(RMA)g(comm)o(unication,)h(has)f(the)g(additional)i(seman)o(tics)e(of)
g(a)g(fence)h(with)f(resp)q(ect)75 1699 y(to)f(RMA)g(calls)i(that)d(use)i
(this)g(comm)o(unicator.)k(All)d(the)e(RMA)h(calls)g(issued)h(on)e(the)g
(comm)o(unicator)75 1755 y(b)q(efore)d(the)f(call)i(to)d Fl(MPI)p
498 1755 V 16 w(BARRIER)j Fu(will)g(complete)f(at)f(the)g(target)f(b)q(efore)
i Fl(MPI)p 1455 1755 V 16 w(BARRIER)h Fu(completes.)75 1812
y(This)i(includes)i(non)o(blo)q(c)o(king)f(calls)g(started)d(b)q(efore)i(the)
g(call)h(to)e(barrier,)g(ev)o(en)h(if)g(the)g(matc)o(hing)f(w)o(ait)h
Fn(>)h Fm(\(June\))75 1868 y Fu(or)f(test)f(call)j(has)e(not)g(y)o(et)f(b)q
(een)j(issued.)1890 1870 y Fn(?)f Fm(\(June\))166 2007 y Fh(Missing:)37
b Fk(If)15 b(MPI2)f(appro)o(v)o(es)h(a)f(non)o(blo)q(c)o(king)f(barrier)i
Fj(MPI)p 1190 2007 13 2 v 15 w(IBARRIER\(com)n(m)m(,)c(request\))p
Fk(,)16 b(then,)f(this)75 2064 y(will)d(apply)i(to)f(it)h(as)g(w)o(ell.)166
2203 y Fu(It)f(is)h(also)f(useful)h(to)e(ha)o(v)o(e)h(a)g(\\fence")g(mec)o
(hanism)h(that)e(guaran)o(tees)h(completion)h(of)f(puts)g(issued)75
2259 y(b)o(y)g(one)g(pro)q(cess.)19 b(This)14 b(is)f(useful)h(if)g(a)f(pro)q
(cess)g(has)g(to)f(mo)q(dify)h(atomically)h(a)f(structure)g(stored)f(in)i
(the)75 2316 y(memory)g(of)h(other)f(pro)q(cess\(es\).)19 b(The)d(op)q
(eration)f(ma)o(y)f(consists)h(of)f(a)g(lo)q(c)o(k,)h(follo)o(w)o(ed)h(b)o(y)
e(a)h(sequence)75 2372 y(of)i(gets)g(and)h(puts,)f(follo)o(w)o(ed)h(b)o(y)f
(an)h(unlo)q(c)o(k)g(op)q(eration.)27 b(One)18 b(has)g(to)e(ensure)i(that)f
(all)h(up)q(dates)g(of)75 2429 y(remote)d(data)f(ha)o(v)o(e)h(o)q(ccurred)h
(b)q(efore)f(the)h(unlo)q(c)o(k)g(op)q(eration)f(is)h(issued.)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 18 20
bop 75 -100 a Fu(18)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(FENCE\(comm)m(,)d(rank\))117
126 y Fk(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)11 b(used)k(in)f(RMA)f
(\(handle\))117 202 y(IN)155 b Fl(rank)504 b Fk(rank)14 b(of)f(target)i(pro)q
(cess)g(\(in)o(teger\))75 326 y Fq(int)23 b(MPI)p 245 326 15
2 v 17 w(Fence\(MPI)p 478 326 V 16 w(Comm)g(comm,)h(int)f(rank\))75
413 y(MPI)p 150 413 V 17 w(FENCE\()g(COMM,)g(RANK,)g(IERROR\))170
469 y(INTEGER)g(COMM,)h(RANK,)f(IERROR)166 555 y Fu(The)18
b(call)i(returns)e(when)h(all)g(previous)g(RMA)f(calls)h(issued)h(b)o(y)e
(the)g(caller)h(on)g(this)f(comm)o(uni-)75 612 y(cator)e(to)g(the)g(target)g
(pro)q(cess)h Fl(rank)f Fu(ha)o(v)o(e)g(completed)i(in)f(the)g(memory)f(of)g
(pro)q(cess)h Fl(rank)p Fu(.)24 b(If)17 b Fl(rank)g(=)75 668
y(MPI)p 160 668 14 2 v 16 w(ANY)p 268 668 V 17 w(SOURCE)f Fu(then)g(the)f
(call)h(returns)f(when)g(all)h(RMA)g(calls)g(issued)g(b)o(y)f(the)g(caller)h
(on)f Fl(comm)75 725 y Fu(ha)o(v)o(e)g(completed.)166 864 y
Fh(Alternativ)o(es)o(:)34 b Fk(Require)14 b(put)g(and)f(accum)o(ulate)g(to)h
(b)q(e)g(complete)f(at)g(the)i(target)f(memory)m(,)c(b)q(efore)k(the)75
920 y(op)q(eration)k(completes)g(lo)q(cally)m(.)29 b(W)m(e)18
b(should)g(discuss)i(this)e(alternativ)o(e)g(seriously:)27
b(It)19 b(do)q(es)g(not)f(a\013ect)h(get,)75 977 y(whic)o(h)12
b(is)h(lik)o(ely)e(to)h(b)q(e)h(the)g(most)e(frequen)o(tly)i(used)g(RMA)g(op)
q(eration)f(\(assuming)f(an)h(o)o(wner)h(compute)f(mo)q(del\);)75
1033 y(users)19 b(can)f(still)f(o)o(v)o(erlap)h(comm)o(unicati)o(on)d(with)j
(computation,)e(using)i(non)o(blo)q(c)o(king)f(put)h(and)f(accum)o(ulate;)75
1090 y(fence)12 b(and)e(mo)q(di\014ed)f(barrier)i(are)g(not)g(needed,)h(an)o
(ymore;)e(the)h(programming)c(mo)q(del)i(is)i(easier)g(to)g(explain;)f(and)75
1146 y(shared)i(memory)d(implem)o(en)o(tations)g(\(or,)i(more)f(generally)m
(,)h(implem)o(en)o(tations)e(that)i(do)g(not)g(ha)o(v)o(e)g(in)o(termediate)
75 1203 y(bu\013ering\))j(are)f(not)g(a\013ected.)20 b(On)13
b(the)h(other)g(hand,)e(distributed)i(memory)d(implemen)o(tati)o(ons)g(with)i
(bu\013ering)75 1259 y(will)f(need)j(an)f(additional)e(end-to-end)j(ac)o(kno)
o(wledgmen)o(t.)189 1448 y Fi(R)n(ationale.)39 b Fu(W)l(e)16
b(do)e(not)h(require)h(that)e(the)h(calls)h(complete)g(also)f(at)f(the)h
(origin)h(at)e(a)h(fence)g(or)189 1505 y(barrier)i({)g(the)g(caller)h(can)f
(ensure)h(this)f(b)o(y)g(using)h(a)e(W)l(aitall)i(call.)27
b(This)17 b(c)o(hoice)h(signi\014can)o(tly)189 1561 y(simpli\014es)f
(implemen)o(tation.)k(\()p Fi(End)16 b(of)g(r)n(ationale.)p
Fu(\))189 1667 y Fi(A)n(dvic)n(e)k(to)i(users.)76 b Fu(A)21
b(fence)h(op)q(eration)f(ma)o(y)g(require)h(comm)o(unication)g(with)f(the)g
(target)189 1724 y(pro)q(cess,)15 b(or)f(with)i(all)g(pro)q(cesses)g(in)g
(the)f(comm)o(unication)h(group,)e(if)i(no)f(target)f(w)o(as)g(sp)q
(eci\014ed.)189 1780 y(\()p Fi(End)h(of)i(advic)n(e)f(to)g(users.)p
Fu(\))189 1886 y Fi(A)n(dvic)n(e)22 b(to)h(implementors.)83
b Fu(Implemen)o(tation)23 b(of)f(these)h(t)o(w)o(o)e(constructs)h(are)g(easy)
g(in)h(a)189 1943 y(shared)14 b(memory)f(implemen)o(tation:)21
b(one)14 b(merely)h(needs)g(to)e(sync)h(up)q(dates)h(to)e(memory)h(b)q(efore)
189 1999 y(the)g(barrier)h(sync)o(hronization)g(is)g(executed.)20
b(\(In)15 b(practice,)g(no)f(extra)f(co)q(de)i(will)h(b)q(e)f(needed)h(on)189
2056 y(most)e(systems.\))189 2131 y(On)g(a)f(distributed)h(memory)l(,)f
(remote)g(agen)o(t)g(implemen)o(tation,)h(on)g(can)f(use)h(a)f(coun)o(ting)h
(mec)o(h-)189 2187 y(anism.)24 b(Eac)o(h)17 b(pro)q(cess)f(k)o(eeps)h(trac)o
(k,)f(for)g(eac)o(h)h(RMA)f(comm)o(unicator,)h(of)f(the)h(total)f(n)o(um)o(b)
q(er)189 2244 y(of)f(RMA)h(op)q(erations)g(issued)h(to)f(eac)o(h)g(target,)e
(and)i(the)g(total)g(n)o(um)o(b)q(er)g(of)f(requests)h(receiv)o(ed,)189
2300 y(from)e(eac)o(h)h(origin.)189 2375 y(A)j(barrier)g(is)h(implemen)o(ted)
g(using)g(a)f(reduce-scatter)g(comm)o(unication)h(where)g(eac)o(h)f(pro)q
(cess)189 2432 y(sends)d(the)f(n)o(um)o(b)q(er)h(of)f(RMA)g(op)q(erations)h
(it)g(issued)g(for)f(eac)o(h)g(destination,)i(and)e(eac)o(h)h(pro)q(cess)189
2488 y(receiv)o(es)i(the)h(total)e(n)o(um)o(b)q(er)h(of)g(of)f(RMA)i
(requests)f(issued)h(to)e(it.)26 b(Next,)17 b(eac)o(h)g(pro)q(cess)g(initi-)
189 2545 y(ates)d(a)h(barrier)h(sync)o(hronization)g(call)h(when)e(it)h(has)f
(completed)h(handling)h(all)g(p)q(ending)g(RMA)189 2601 y(requests.)i(This)14
b(will)h(double)f(the)g(time)f(needed)i(for)e(a)g(barrier)g(sync)o
(hronization)h(proto)q(col)g(that)189 2658 y(uses)h(p)q(oin)o(t)h(to)e(p)q
(oin)o(t)i(comm)o(unication.)1967 46 y Fm(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 19 21
bop 75 -100 a Fo(4.6.)34 b(WINDO)o(W)15 b(COUNTERS)1163 b Fu(19)189
49 y(If)16 b(a)f(fence)h(call)h(sp)q(eci\014es)h(a)d(sp)q(eci\014c)j(target,)
c(then)i(the)g(calling)h(pro)q(cess)f(sends)g(to)f(that)g(target)189
106 y(the)j(coun)o(t)h(of)f(RMA)g(op)q(erations)h(it)g(issued)h(to)e(that)f
(target.)29 b(This)19 b(message)f(is)h(receiv)o(ed)h(b)o(y)189
162 y(the)14 b(RMA)g(agen)o(t)f(at)g(the)h(target)e(who)i(issues)g(a)g(reply)
h(when)f(its)g(coun)o(t)f(of)h(serviced)h(op)q(erations)189
219 y(matc)o(hes)i(the)g(coun)o(t)g(of)h(issued)g(op)q(erations.)27
b(If)18 b(the)f(fence)h(do)q(es)g(not)f(sp)q(ecify)i(a)e(target,)g(then)189
275 y(a)k(scatter)f(needs)i(to)f(b)q(e)h(used,)h(rather)e(than)g(a)g(send,)i
(follo)o(w)o(ed)f(b)o(y)f(a)g(reduce)h(sum)g(on)f(the)189 332
y(ac)o(kno)o(wledgmen)o(ts.)189 407 y(This)g(sc)o(heme)f(in)o(tro)q(duces)h
(a)f(small)h(o)o(v)o(erhead)f(on)h(eac)o(h)f(RMA)h(op)q(erations)f(\(coun)o
(ter)g(incre-)189 463 y(men)o(t\).)189 538 y(Another)d(p)q(ossible)i
(implemen)o(tation,)g(is)e(for)g(eac)o(h)g(pro)q(cess)h(to)e(send)i(to)f(eac)
o(h)g(other)g(in)o(v)o(olv)o(ed)189 595 y(pro)q(cess)k(a)g(\\\015ush")g
(message.)37 b(Since)23 b(messages)e(are)f(ordered,)j(the)e(\015ush)h
(message)e(will)j(b)q(e)189 651 y(receiv)o(ed)12 b(after)f(all)h(previous)g
(RMA)f(requests)g(where)h(receiv)o(ed.)20 b(This)11 b(will)i(require)f
Fg(n)1667 635 y Fc(2)1697 651 y Fu(messages)189 708 y(for)17
b(a)h(barrier,)h Fg(n)f Fu(messages)g(for)f(a)h(fence)h(with)g(no)f(sp)q
(eci\014c)i(target,)d(and)i(one)f(message)g(for)f(a)189 764
y(fence)f(with)f(a)g(sp)q(eci\014c)i(target.)189 839 y(This)e(sc)o(heme)h(in)
o(tro)q(duces)g(no)f(o)o(v)o(erhead)g(on)g(RMA)g(comm)o(unication.)189
914 y(Rather)f(than)f(k)o(eeping)i(trac)o(k)e(of)g(all)i(RMA)f(op)q
(erations,)g(one)g(can)g(k)o(eep)g(trac)o(k)f(only)h(of)g(put)g(and)189
971 y(accum)o(ulate)h(op)q(erations,)g(since)i(get)d(and)i(RMW)f(implicitly)j
(return)d(an)g(ac)o(kno)o(wledgmen)o(t.)189 1046 y(Alternativ)o(ely)l(,)e
(one)f(could)h(implemen)o(t)f(put)g(and)g(accum)o(ulate)g(op)q(erations)g(so)
f(that)g(they)h(return)189 1102 y(an)j(ac)o(kno)o(wledgmen)o(t,)g(once)h
(completed.)22 b(In)16 b(suc)o(h)f(case,)h(the)f(barrier)h(or)f(fence)h(co)q
(de)g(requires)189 1159 y(no)f(additional)h(comm)o(unication.)21
b(\()p Fi(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p Fu(\))75
1302 y Fr(4.6)59 b(Windo)n(w)20 b(Counters)75 1404 y Fu(This)d(section)h(sp)q
(eci\014es)g(the)f(seman)o(tics)g(of)f(the)h(windo)o(w)g(coun)o(ters)f(and)h
(the)g(op)q(erations)g(asso)q(ciated)75 1460 y(with)f(them.)166
1516 y(Eac)o(h)g(windo)o(w)h(is)h(asso)q(ciated)f(with)g(a)f(coun)o(ter)h
(and)g(a)f(threshold.)26 b(The)17 b(coun)o(ter)f(is)i(initialized)75
1573 y(to)d(zero)g(b)o(y)g(the)g(call)h(to)f Fl(MPI)p 592 1573
14 2 v 16 w(RMA)p 707 1573 V 17 w(INIT)p Fu(.)166 1629 y(Eac)o(h)c(put,)h
(get)e(and)i(accum)o(ulate)f(op)q(eration)h(incremen)o(ts)f(the)h(coun)o(ter)
f(asso)q(ciated)g(with)g(the)h(tar-)75 1686 y(get)i(windo)o(w)g(it)g
(accesses)h(b)o(y)f(the)g(v)m(alue)i(of)d(the)h Fl(increment)f
Fu(argumen)o(t.)19 b(The)14 b(coun)o(ter)g(is)h(incremen)o(ted)75
1742 y(after)j(the)i(access)f(to)f(the)h(target)f(windo)o(w)i(is)f(completed)
h(\(p)q(ossibly)l(,)i(b)q(efore)d(the)g(en)o(tire)h(op)q(eration)75
1799 y(completed\).)166 1855 y(The)14 b(target)e(pro)q(cess)i(can)g(c)o(hec)o
(k)g(the)g(v)m(alue)h(of)e(the)g(coun)o(ter)h(for)f(a)g(windo)o(w)h(in)h(its)
f(o)o(wn)f(memory)l(,)75 1912 y(b)o(y)f(calling)i Fl(MPI)p
361 1912 V 16 w(GET)p 465 1912 V 17 w(COUNTER)p Fu(.)f(In)f(addition,)i(a)e
(mec)o(hanism)g(is)h(pro)o(vided)g(for)f(the)g(target)f(windo)o(w)75
1968 y(to)17 b(w)o(ait)g(un)o(til)h(the)g(coun)o(ter)f(reac)o(hes)h(a)f
(threshold)h(v)m(alue.)28 b(The)18 b(threshold)g(is)g(set)f(or)g(reset)g(b)o
(y)h(calls)75 2025 y(to)d Fl(MPI)p 216 2025 V 16 w(SET)p 315
2025 V 17 w(THRESHOLD)p Fu(.)h(This)h(call)g(also)f(allo)q(cates)g(a)g
(request,)g(and)g(returns)g(a)f(handle)i(to)f(that)75 2081
y(request.)k(The)15 b(request)g(completes)h(when)f(the)g(coun)o(ter)g(reac)o
(hes)g(\(or)f(exceeds\))h(the)g(threshold)h(v)m(alue,)75 2137
y(at)f(whic)o(h)h(p)q(oin)o(t)g(the)f(coun)o(ter)h(is)g(decremen)o(ted)g(b)o
(y)f(the)h(threshold)g(v)m(alue.)21 b(The)16 b(threshold)g(can)g(then)f
Fn(>)h Fm(\(June\))1890 2196 y Fn(?)g Fm(\(June\))75 2194 y
Fu(b)q(e)h(reset)e(b)o(y)h(a)g(new)g(call)h(to)e Fl(MPI)p 669
2194 V 16 w(SET)p 768 2194 V 17 w(THRESHOLD)p Fu(.)h(The)g(threshold)h
(should)g(not)e(b)q(e)i(reset)e(while)75 2250 y(the)g(asso)q(ciated)h
(request)f(is)h(activ)o(e.)-32 46 y Fm(1)-32 103 y(2)-32 159
y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441
y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40
724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40
1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232
y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40
1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740
y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40
2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248
y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40
2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 20 22
bop 75 -100 a Fu(20)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fl(MPI)p 160 49 14 2 v 16 w(GET)p 264 49 V
17 w(COUNTER)h(\(comm)m(,)c(count\))117 126 y Fk(IN)155 b Fl(comm)466
b Fk(comm)o(unicator)11 b(iden)o(tifying)i(RMA)g(windo)o(w)g(\(handle\))117
199 y(OUT)108 b Fl(count)482 b Fk(curren)o(t)18 b(v)n(alue)d(of)h(asso)q
(ciated)h(coun)o(ter)g(\(non)f(negativ)o(e)g(in-)905 255 y(teger\))75
380 y Fq(int)23 b(MPI)p 245 380 15 2 v 17 w(Get)p 334 380 V
17 w(counter\(MPI)p 615 380 V 16 w(Comm)g(comm,)g(int)h(count\))75
466 y(MPI)p 150 466 V 17 w(GET)p 239 466 V 17 w(COUNTER\(COMM,)e(COUNT,)h
(IERROR\))170 523 y(INTEGER)g(COMM,)h(COUNT,)f(IERROR)166 609
y Fu(Returns)14 b(the)g(curren)o(t)g(v)m(alue)h(from)e(the)h(coun)o(ter)g
(asso)q(ciated)g(with)h(the)f(lo)q(cal)h(windo)o(w)f(of)f Fl(comm)m
Fu(.)75 760 y Fl(MPI)p 160 760 14 2 v 16 w(SET)p 259 760 V
17 w(COUNTER)p 488 760 V 17 w(THRESHOLD)j(\(comm)m(,)c(count,)k(request\))117
837 y Fk(IN)155 b Fl(comm)466 b Fk(comm)o(unicator)9 b(asso)q(ciated)k(with)e
(RMA)h(windo)o(w)f(\(handle\))117 910 y(IN)155 b Fl(count)482
b Fk(n)o(um)o(b)q(er)13 b(of)h(units)g(\(non-negativ)o(e)f(in)o(teger\))117
982 y(OUT)108 b Fl(request)452 b Fk(request)16 b(asso)q(ciated)e(with)g(coun)
o(ter)h(\(handle\))75 1107 y Fq(int)23 b(MPI)p 245 1107 15
2 v 17 w(Set)p 334 1107 V 17 w(counter)p 519 1107 V 16 w(threshold\(MPI)p
847 1107 V 16 w(Comm)g(comm,)g(int)h(count,)393 1163 y(MPI)p
468 1163 V 17 w(Request)f(*request\))75 1250 y(MPI)p 150 1250
V 17 w(SET)p 239 1250 V 17 w(COUNTER)p 424 1250 V 16 w(THRESHOLD\(COMM,)f
(COUNT,)h(REQUEST,)f(IERROR\))170 1306 y(INTEGER)h(COMM,)h(COUNT,)f(REQUEST,)
f(IERROR)166 1393 y Fu(Set)17 b(the)g(threshold)h(for)f(the)g(coun)o(ter)g
(of)f(the)i(lo)q(cal)g(windo)o(w)f(asso)q(ciated)h(with)f Fl(comm)11
b Fu(to)17 b Fl(count)75 1449 y Fu(and)e(asso)q(ciates)g(a)g(request)h(with)f
(the)g(ev)o(en)o(t)g(\\coun)o(ter)g(reac)o(hes)g(threshold".)189
1537 y Fi(R)n(ationale.)46 b Fu(The)16 b(use)h(of)f(windo)o(w)g(coun)o(ters)g
(could)h(b)q(e)g(a)o(v)o(oided)f(and)h(replaced)g(b)o(y)f(messages)189
1593 y(sen)o(t)21 b(b)o(y)h(the)g(origin)h(pro)q(cess)f(to)f(the)i(target)d
(pro)q(cess.)41 b(But)22 b(this)g(will)i(require)f(additional)189
1649 y(comm)o(unication,)f(in)f(a)f(distributed)h(memory)f(system.)35
b(In)21 b(the)f(case)h(of)e(a)h(get,)h(one)g(w)o(ould)189 1706
y(need)c(an)f(additional)i(ac)o(kno)o(wledgmen)o(t)e(message;)g(in)i(a)e
(case)g(of)g(a)g(put,)h(one)f(w)o(ould)h(ha)o(v)o(e)f(the)189
1762 y(o)o(v)o(erhead)e(of)h(a)g(fence)h(and)f(an)g(additional)i(ac)o(kno)o
(wledgmen)o(t)e(message.)189 1835 y(Note)k(that)f(coun)o(ter)h(seman)o(tics)h
(are)f(carefully)h(designed)h(to)e(a)o(v)o(oid)g(race)g(conditions.)33
b(Since)189 1891 y(coun)o(ters)18 b(are)g(only)h(incremen)o(ted)h(and)f
(decremen)o(ted)g(and)g(nev)o(er)g(set,)g(incremen)o(ts)g(from)f(in-)189
1948 y(coming)13 b(puts)g(and)h(gets)e(are)h(nev)o(er)g(lost.)20
b(The)13 b(only)h(thing)f(to)g(w)o(atc)o(h)f(out)h(for)f(is)i(that)e(a)h
(coun)o(ter)189 2004 y(ma)o(y)h(exceed)i(its)g(threshold)g(v)m(alue.)21
b(\()p Fi(End)15 b(of)i(r)n(ationale.)p Fu(\))189 2092 y Fi(A)n(dvic)n(e)d
(to)h(implementors.)39 b Fu(The)14 b(coun)o(ter)f(need)i(not)e(b)q(e)i(up)q
(dated)f(\(th)o(us)f(reducing)i(comm)o(uni-)189 2148 y(cation)f(o)o(v)o
(erhead\))f(when)h(the)g(incremen)o(t)h(is)f(zero.)20 b(On)14
b(a)g(distributed)h(memory)e(system)h(where)189 2205 y(RMA)19
b(op)q(erations)g(are)g(implemen)o(ted)i(b)o(y)e(a)f(remote)h(agen)o(t,)g
(the)g(agen)o(t)f(will)j(incremen)o(t)f(the)189 2261 y(target)14
b(coun)o(ter.)21 b(In)16 b(a)f(shared)h(memory)f(implemen)o(tation,)h(the)g
(coun)o(ter)f(will)i(b)q(e)g(up)q(dated)f(b)o(y)189 2318 y(the)f(origin)h
(pro)q(cess;)f(lo)q(c)o(king)h(is)g(required.)21 b(\()p Fi(End)15
b(of)i(advic)n(e)f(to)h(implementors.)p Fu(\))75 2458 y Fr(4.7)59
b(Semantics)18 b(of)i(RMA)e(Op)r(erations)75 2561 y Ff(4.7.1)49
b(Order)75 2647 y Fu(When)13 b(a)g(put)g(or)g(get)f(completes)i(then)f(the)h
(op)q(eration)f(has)g(completed)h(at)e(the)h(origin,)h(and)f(the)g(origin)75
2704 y(bu\013er)i(can)h(b)q(e)g(accessed.)21 b(In)16 b(the)f(case)g(of)g(a)g
(put,)g(the)h(data)f(has)g(b)q(een)h(copied)h(out;)d(in)j(the)e(case)g(of)g
(a)1967 46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967
272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959
554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780
y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959
1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959
1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959
1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959
1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959
1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959
2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959
2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959
2643 y(47)1959 2699 y(48)p eop
%%Page: 21 23
bop 75 -100 a Fo(4.7.)34 b(SEMANTICS)16 b(OF)f(RMA)g(OPERA)l(TIONS)853
b Fu(21)75 49 y(get,)14 b(the)i(data)e(has)h(b)q(een)i(copied)f(in.)1165
b Fn(>)16 b Fm(\(June\))166 106 y Fu(When)j(a)f(put)g(or)g(get)g(up)q(dates)h
(the)g(target)e(request,)i(then)g(the)f(op)q(eration)h(has)f(completed)i(at)
75 162 y(the)d(target,)f(and)h(the)g(target)f(bu\013er)h(can)g(b)q(e)h
(accessed.)26 b(In)17 b(the)h(case)f(of)f(a)h(get,)f(the)i(data)e(has)h(b)q
(een)75 219 y(copied)f(out;)f(in)h(the)f(case)g(of)g(a)g(put,)g(the)g(data)g
(has)g(b)q(een)h(copied)h(in.)605 b Fn(?)16 b Fm(\(June\))166
275 y Fu(Calling)21 b Fl(MPI)p 412 275 14 2 v 16 w(BARRIER)h
Fu(with)e(a)g(comm)o(unicator)g(returned)h(b)o(y)f(one)g(of)g(the)g
Fl(MPI)p 1658 275 V 16 w(RMA)p 1773 275 V 17 w(INIT)75 332
y Fu(calls)j(has)f(the)g(additional)h(seman)o(tics)f(of)g(a)f(fence)i(with)f
(resp)q(ect)h(to)e(puts)h(and)g(gets.)40 b(All)23 b(RMA)75
388 y(comm)o(unications)12 b(started)e(on)h(the)g(comm)o(unicator)g(b)q
(efore)g(the)h(call)g(to)e Fl(MPI)p 1404 388 V 16 w(BARRIER)i
Fu(will)h(complete)75 444 y(at)f(the)h(target)e(b)q(efore)i
Fl(MPI)p 554 444 V 16 w(BARRIER)h Fu(completes.)20 b(Similarly)l(,)15
b(a)d(call)i(to)e Fl(MPI)p 1449 444 V 16 w(FENCE)h Fu(has)f(the)h(e\013ect)75
501 y(that)i(all)h(RMA)g(comm)o(unications)g(started)f(b)o(y)g(the)h(caller)g
(on)g(the)g(comm)o(unicator)f(\(to)f(the)i(sp)q(eci\014ed)75
557 y(target\))e(b)q(efore)h(the)g(call)i(to)d(fence)i(will)h(complete)f(at)f
(the)g(target\(s\))e(b)q(efore)j(the)f(fence)h(call)g(returns.)166
614 y(No)g(other)g(order)g(requiremen)o(ts)h(are)f(imp)q(osed)h(on)g(RMA)f
(op)q(erations.)24 b(Tw)o(o)15 b(RMA)i(op)q(erations)75 670
y(issued)d(consecutiv)o(ely)h(b)o(y)e(the)g(same)g(pro)q(cess)g(for)g(the)g
(same)f(target)g(can)i(execute)f(in)h(arbitrary)f(order,)75
727 y(unless)19 b(the)f(user)h(ensures)f(that)g(the)g(\014rst)f(op)q(eration)
i(has)f(completed)h(b)q(efore)f(it)g(issues)h(the)f(second)75
783 y(one.)h(Similarly)l(,)d(there)e(is)g(no)f(guaran)o(teed)g(order)h(of)f
(execution)h(b)q(et)o(w)o(een)g(message)f(passing)h(calls)h(and)75
840 y(RMA)k(calls,)h(ev)o(en)f(when)g(issued)g(b)o(y)g(the)f(same)h(pro)q
(cess)f(with)h(the)g(same)f(comm)o(unicator)g(and)h(the)75
896 y(same)c(destination.)1469 b Fn(>)16 b Fm(\(June\))166
953 y Fu(Note)d(that)f(\\b)q(efore")h(or)g(\\after",)f(as)g(used)i(in)g(the)f
(previous)h(paragraphs,)e(mean)i(b)q(efore)f(or)g(after)75
1009 y(in)e(program)e(order)h(at)f(eac)o(h)i(pro)q(cess.)18
b(If)11 b(a)e(pro)q(cess)i(is)f(m)o(ultithreaded,)i(then)f(t)o(w)o(o)e(op)q
(erations)h(executed)75 1065 y(b)o(y)g(the)h(same)f(pro)q(cess)h(ma)o(y)e
(not)h(b)q(e)h(ordered,)h(in)f(whic)o(h)g(case)g(there)f(is)h(no)f(order)g
(requiremen)o(t.)19 b(Th)o(us,)11 b(a)75 1122 y(barrier)h(executed)i(b)o(y)e
(a)g(thread)g(only)h(enforces)f(completion)i(of)e(RMA)g(calls)h(that)f
(preceed)h(the)g(barrier)75 1178 y(call)19 b(in)g(program)e(order;)j(an)e
(RMA)g(call)h(executed)h(b)o(y)e(another)g(concurren)o(t)g(thread)g(ma)o(y)l
(,)g(or)g(ma)o(y)75 1235 y(not)d(not)g(b)q(e)g(a\013ected.)1426
b Fn(?)16 b Fm(\(June\))189 1312 y Fi(R)n(ationale.)39 b Fu(The)14
b(order)f(b)q(eha)o(vior)i(describ)q(ed)g(corresp)q(onds)g(to)e(an)h(op)q
(erational)g(mo)q(del)h(where)189 1369 y(RMA)k(requests)g(are)f(handled)j(b)o
(y)e(an)g(RMA)g(agen)o(t,)g(suc)o(h)g(as)g(a)g(p)q(ersisten)o(t)g(comm)o
(unication)189 1425 y(handler;)h(the)e(agen)o(t)f(p)q(osts)h(a)g(non)o(blo)q
(c)o(king)h(receiv)o(e)g(to)e(handle)i(a)f(put)g(request)g(and)h(p)q(osts)e
(a)189 1482 y(non)o(blo)q(c)o(king)d(send)f(to)f(handle)i(a)f(get)f(request.)
19 b(Th)o(us,)13 b(ev)o(en)g(though)g(incoming)h(RMA)f(requests)189
1538 y(are)21 b(handled)i(in)f(order,)h(the)e(accesses)h(to)f(the)g(target)g
(memory)g(ma)o(y)f(o)q(ccur)i(out)g(of)f(order.)189 1595 y(Consider)14
b(the)h(case)f(of)g(a)g(blo)q(c)o(king)h(put,)f(immediately)i(follo)o(w)o(ed)
f(b)o(y)f(another)g(blo)q(c)o(king)h(put)g(to)189 1651 y(the)j(same)h
(destination.)31 b(Eac)o(h)18 b(\\send")h(matc)o(hes)f(its)h(o)o(wn)f
(\\receiv)o(e",)h(but)g(the)g(\014rst)f(\\send")189 1708 y(\(put\))10
b(ma)o(y)g(return)h(b)q(efore)g(data)g(has)f(b)q(een)j(copied)f(in)f(the)g
(remote)g(memory)l(.)18 b(Th)o(us,)11 b(the)g(up)q(date)189
1764 y(of)k(the)h(target)f(memory)h(b)q(e)g(the)h(t)o(w)o(o)d(put)j(calls)g
(ma)o(y)e(o)o(v)o(erlap.)22 b(Consider)17 b(a)f(put,)g(follo)o(w)o(ed)g(b)o
(y)189 1821 y(a)h(get,)g(with)g(the)g(same)g(comm)o(unicator)g(and)h(same)f
(target)f(bu\013er.)26 b(The)17 b(put)h(ma)o(y)e(complete)189
1877 y(and)f(the)f(get)h(ma)o(y)f(start)f(b)q(efore)i(data)f(w)o(as)g
(written)h(in)h(the)f(target)e(bu\013er;)i(the)f(t)o(w)o(o)g(calls)i(ma)o(y)
189 1933 y(o)o(v)o(erlap,)e(and)i(the)f(get)g(need)h(not)f(return)g(the)g
(data)f(written)i(b)o(y)f(the)g(put.)189 2004 y(An)k(alternativ)o(e)h(b)q
(eha)o(vior)g(is)g(that)f(ac)o(hiev)o(ed)i(b)o(y)e(an)h(op)q(erational)g(mo)q
(del)g(where)g(the)g(RMA)189 2061 y(agen)o(t)13 b(p)q(osts)g(blo)q(c)o(king)i
(sends)f(to)f(satisfy)g(get)h(requests,)f(and)h(blo)q(c)o(king)h(receiv)o(es)
f(to)f(satisfy)h(put)189 2117 y(requests.)19 b(Th)o(us,)c(the)h(handling)g
(of)f(RMA)h(requests)f(is)g(serialized.)22 b(In)16 b(this)g(mo)q(del)231
2203 y(1.)22 b(Eac)o(h)15 b(RMA)g(op)q(eration)g(is)h(executed)g(atomically)
231 2268 y(2.)22 b(RMA)13 b(op)q(erations)h(issued)g(successiv)o(ely)h(b)q(e)
f(the)g(same)f(origin)h(to)f(the)g(same)g(target,)f(using)289
2324 y(the)j(same)g(comm)o(unicator,)f(are)h(executed)h(in)g(the)g(order)f
(they)g(w)o(ere)g(issued.)189 2410 y(This)e(alternativ)o(e)f(mo)q(del)h(pro)o
(vides)g(stronger)f(order)g(and)g(atomicit)o(y)g(guaran)o(tees,)g(and)h(is)g
(easier)189 2466 y(to)h(program)g(with;)h(ho)o(w)o(ev)o(er,)f(it)i(will)g
(lead)g(to)f(lo)o(w)o(er)g(p)q(erformance,)g(as)f(it)i(reduces)g(the)f(opp)q
(or-)189 2523 y(tunities)h(for)e(concurrency)l(.)21 b(In)15
b(a)g(shared)g(memory)g(implemen)o(tation,)g(one)h(will)g(lik)o(ely)h(need)f
(to)189 2579 y(lo)q(c)o(k)i(the)f(windo)o(w)h(accessed)g(and)g(serialize)h
(all)g(accesses.)27 b(In)18 b(a)f(distributed)i(memory)l(,)f(RMA)189
2636 y(agen)o(t)g(implemen)o(tation,)j(the)d(agen)o(t)h(will)h(ha)o(v)o(e)f
(to)f(use)h(blo)q(c)o(king)h(sends)g(and)f(receiv)o(es,)h(th)o(us)189
2692 y(reducing)c(concurrency)l(.)21 b(\()p Fi(End)16 b(of)g(r)n(ationale.)p
Fu(\))-32 46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32
272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40
554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40
836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40
1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345
y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40
1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853
y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40
2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361
y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40
2643 y(47)-40 2699 y(48)p eop
%%Page: 22 24
bop 75 -100 a Fu(22)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Ff(4.7.2)49 b(Progress)75 135 y Fu(Once)18
b(a)f(RMA)g(op)q(eration)h(is)g(p)q(osted,)f(it)h(should)g(complete,)g
(irresp)q(ectiv)o(e)h(of)d(other)h(op)q(erations)h(o)q(c-)75
192 y(curring)f(in)h(the)f(system.)24 b(The)17 b(completion)h(of)e(a)h(put,)g
(accum)o(ulate)g(or)f(RMW)h(op)q(eration)g(in)g(target)75 248
y(memory)e(means)h(only)h(that)e(data)g(is)i(stored)e(logically)j(in)f(the)f
(target)e(windo)o(w)i(at)g(is)g(visible)i(to)d(sub-)75 304
y(sequen)o(t)j(RMA)g(op)q(erations)f(originated)h(at)f(an)o(y)h(pro)q(cess.)
27 b(Before)18 b(using)g(loads)g(and)g(stores)e(on)i(the)-1992
b Fn(>)15 b Fm(\(June\))75 361 y Fu(target)g(pro)q(cess)i(to)e(access)i(data)
e(up)q(dated)i(b)o(y)f(a)g(prior)h(RMA)f(op)q(eration,)h(the)f(target)f(pro)q
(cess)i(m)o(ust)75 417 y(\014rst)h(issue)h(a)e(call)j(to)d
Fl(MPI)p 554 417 14 2 v 16 w(WINDO)o(W)p 766 417 V 16 w(IN)p
Fu(;)g(similarly)j(a)e(call)h(to)f Fl(MPI)p 1314 417 V 15 w(WINDO)o(W)p
1525 417 V 16 w(OUT)h Fu(m)o(ust)f(o)q(ccur)75 474 y(after)c(data)f(w)o(as)h
(stored)g(b)o(y)g(a)g(pro)q(cess)h(in)g(its)g(memory)f(and)g(b)q(efore)h
(this)g(data)e(can)i(b)q(e)g(accessed)g(b)o(y)f(a)75 530 y(get)h(\(see)g(b)q
(elo)o(w\).)-503 b Fn(?)15 b Fm(\(June\))75 673 y Fr(4.8)59
b(A)n(tomicit)n(y)18 b(and)i(Memo)n(ry)e(Coherence)i(Op)r(erations)-117
730 y Fn(>)15 b Fm(\(June\))75 775 y Fu(First,)f(some)h(terminology)l(.)166
831 y(The)i(execution)i(of)d(a)h(message-passing)g(program)f(imp)q(oses)i(a)f
(partial)h(order)f(on)g(the)g(program)75 888 y(op)q(erations:)i(this)c(is)f
(the)g(transitiv)o(e)g(closure)h(of)e(the)h(order)g(relation)g(at)g(eac)o(h)g
(pro)q(cess)g(\(if)g(the)g(pro)q(cess)75 944 y(is)h(single)g(threaded,)f
(then)h(all)g(the)f(op)q(erations)g(executed)h(b)o(y)f(this)h(pro)q(cess)f
(are)g(totally)g(ordered\))g(and)75 1001 y(the)g(order)g(implied)j(b)o(y)d
(comm)o(unications)h(\(a)f(receiv)o(e)h(completes)g(after)f(the)g(matc)o
(hing)g(send)h(started,)75 1057 y(a)g(windo)o(w)g(coun)o(ter)f(is)i(up)q
(dated)f(after)g(the)g(corresp)q(onding)g(RMA)h(call)g(started,)d(etc.\).)19
b(W)l(e)c(sa)o(y)g(that)75 1114 y(Op)q(eration)h(B)f Fi(fol)r(lows)g
Fu(op)q(eration)g(A)g(\(and)g(A)g Fi(pr)n(e)n(c)n(e)n(des)f
Fu(B\))g(if)i(AB)f(b)q(elongs)h(to)f(this)g(partial)g(order;)g(A)75
1170 y(and)h(B)g(are)f Fi(c)n(oncurr)n(ent)g Fu(if)h(neither)h(A)f(precedes)g
(B)g(nor)g(B)g(precedes)g(A.)g(Note)f(that)g(this)h(is)g(a)g(logical)75
1227 y(order)f(of)g(execution,)h(not)e(a)h(ph)o(ysical,)h(w)o(all-clo)q(c)o
(k)h(ordering.)166 1283 y(Tw)o(o)12 b(accesses)g(to)g(the)h(same)f(memory)g
(lo)q(cation)i(are)e Fi(c)n(on\015icting)f Fu(if)i(there)g(is)g(a)f(lo)q
(cation)h(accessed)75 1339 y(b)o(y)i(b)q(oth)g(and)h(up)q(dated)g(b)o(y)f(at)
g(least)g(one)g(of)g(the)g(t)o(w)o(o)f(op)q(erations.)166 1396
y(W)l(e)g(distinguish)j(b)q(et)o(w)o(een)d Fi(lo)n(c)n(al)f
Fu(memory)h(accesses,)g(and)h Fi(r)n(emote)f Fu(memory)f(accesses.)20
b(Accesses)75 1452 y(executed)h(b)o(y)g(a)f(pro)q(cess)h(to)f(its)g(o)o(wn)g
(memory)g(using)i(loads)e(or)g(stores)g(are)g(lo)q(cal;)k(also)d(are)f(lo)q
(cal)75 1509 y(accesses)d(to)g(a)g(comm)o(unication)g(bu\013er)h(p)q
(erformed)f(b)o(y)g(a)g(send)g(or)g(receiv)o(e)h(MPI)f(call)h(and)g(accesses)
75 1565 y(to)13 b(the)i(origin)f(bu\013er)g(of)g(an)g(RMA)g(call.)21
b(Accesses)15 b(to)e(the)h(target)f(bu\013er)h(of)g(an)g(RMA)g(op)q(eration)g
(are)75 1622 y(remote)e(\(including)k(the)d(case)f(of)h(an)g(RMA)g(op)q
(eration)g(where)g(a)g(pro)q(cess)g(accesses)g(its)g(o)o(wn)f(memory\).)166
1678 y(MPI)k(has)g(to)f(sp)q(ecify)i(the)f(outcome)g(of)f(concurren)o(t,)h
(con\015icting)h(accesses)g(to)e(a)h(windo)o(w.)22 b(Tw)o(o)75
1735 y(issues)16 b(arise:)131 1828 y(1.)22 b(Con\015icting)16
b(remote)f(RMA)g(accesses)g(to)g(the)g(same)g(windo)o(w.)131
1922 y(2.)22 b(Con\015icts)15 b(b)q(et)o(w)o(een)h(lo)q(cal)g(and)f(remote)g
(accesses)h(to)e(a)h(windo)o(w.)75 2042 y Fl(Con\015icting)h(Remote)d
(Accesses)75 2128 y Fu(The)i(execution)h(of)f(a)g(Read)p 565
2128 V 17 w(Mo)q(dify)p 723 2128 V 17 w(W)l(rite)g(or)g(accum)o(ulate)g(up)q
(date)h(of)e(an)h(elemen)o(t)h(is)g(atomic)f(with)75 2185 y(resp)q(ect)e(to)e
(other)h(Read)p 500 2185 V 17 w(Mo)q(dify)p 658 2185 V 17 w(W)l(rite)h(and)f
(accum)o(ulate)h(up)q(dates)g(of)e(that)h(same)g(elemen)o(t.)20
b(I.e.,)12 b(the)75 2241 y(outcome)f(is)g(as)g(if)g(the)g(RMW)g(and)g(accum)o
(ulate)g(up)q(dates)h(of)e(that)g(elemen)o(t)i(w)o(ere)f(serialized.)20
b(Ho)o(w)o(ev)o(er,)75 2298 y(if)c(t)o(w)o(o)f(accum)o(ulate)h(op)q(erations)
g(up)q(date)g(the)g(same)g(t)o(w)o(o)e(elemen)o(ts,)j(then)f(these)g(t)o(w)o
(o)e(elemen)o(ts)j(ma)o(y)75 2354 y(b)q(e)22 b(up)q(dated)f(in)h(a)f
(di\013eren)o(t)g(order)g(b)o(y)g(the)g(t)o(w)o(o)f(op)q(erations:)31
b(while)23 b(eac)o(h)e(individual)j(up)q(date)d(is)75 2410
y(atomic,)15 b(the)g(en)o(tire)h(accum)o(ulate)f(op)q(eration)h(is)f(not)g
(atomic.)166 2467 y(It)g(is)h(erroneous)f(to)g(ha)o(v)o(e)g(concurren)o(t)g
(accum)o(ulate)h(or)e(RMW)h(up)q(dates)h(of)f(v)m(ariables)i(that)d(par-)75
2523 y(tially)19 b(o)o(v)o(erlap;)g(e.g.,)e(accum)o(ulating)i(in)o(to)f
(complex)h(n)o(um)o(b)q(ers,)f(of)g(t)o(yp)q(e)g Fj(MPI)p 1463
2523 13 2 v 14 w(COMPLEX)p Fu(,)f(and)h(con-)75 2580 y(curren)o(tly)c(accum)o
(ulating)g(in)o(to)g(the)f(real)h(or)f(imaginary)h(part)e(of)h(these)h(n)o
(um)o(b)q(ers,)g(of)f(t)o(yp)q(e)g Fj(MPI)p 1748 2580 V 15
w(REAL)p Fu(.)1967 46 y Fm(1)1967 103 y(2)1967 159 y(3)1967
215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967
498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724
y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959
1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959
1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959
1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959
1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959
1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959
2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959
2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959
2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 23 25
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(23)166 49 y(The)14 b(outcome)f(is)h(unde\014ned)h(if)
f(put)g(or)f(get)g(op)q(erations)h(con\015ict)g(with)g(other)f(RMA)h(op)q
(erations.)75 106 y(Also,)i(the)g(outcome)f(is)h(unde\014ned)i(if)e(t)o(w)o
(o)e(RMA)i(op)q(erations)g(using)g(di\013eren)o(t)g(comm)o(unicators)f(p)q
(er-)75 162 y(form)f(con\015icting)j(accesses)f(to)e(the)h(same)g(lo)q
(cation.)189 267 y Fi(R)n(ationale.)39 b Fu(The)14 b(mo)q(del)h(pro)o(vide)f
(w)o(eak)o(er)f(seman)o(tics)h(for)f(con\015icts)i(in)o(v)o(olving)g(put/get)
f(op)q(er-)189 323 y(ations)h(and)h(stronger)e(atomicit)o(y)h(condidions)j
(for)c(con\015icts)j(in)o(v)o(olving)f(accum)o(ulate)g(or)f(RMW)189
380 y(op)q(erations.)36 b(The)21 b(assumption)f(is)i(that)d(con\015icting)j
(put/get)f(op)q(erations)f(will)j(b)q(e)e(ordered)189 436 y(using)16
b(some)e(sync)o(hronization)i(mec)o(hanism,)g(as)f(otherwise)g(the)g
(execution)i(is)e(nondeterminis-)189 493 y(tic.)20 b(On)15
b(the)f(other)g(hand,)h(accum)o(ulate)g(and)f(RMW)h(op)q(erations)f(ma)o(y)g
(b)q(e)h(used)g(to)f(implemen)o(t)189 549 y(sync)o(hronization)f(primitiv)o
(es,)g(suc)o(h)g(as)f(semaphores;)g(therefore)g(it)g(is)h(imp)q(ortan)o(t)f
(to)f(allo)o(w)i(con-)189 606 y(curren)o(t,)e(con\015icting)i(accesses)f(to)f
(the)h(same)f(v)m(ariable)i(b)o(y)e(suc)o(h)h(op)q(erations,)g(with)g(the)f
(outcome)189 662 y(corresp)q(onding)16 b(to)e(some)h(serialization)i(of)e
(these)g(accesses.)189 737 y(With)e(the)h(curren)o(t)f(de\014nition,)i(put)f
(or)f(get)f(op)q(erations)i(requires)g(no)f(lo)q(c)o(king)i(\(except)e(for)g
(b)o(yte)189 793 y(aligned)j(put/get)e(op)q(erations)g(on)h(systems)f(that)f
(supp)q(ort)i(only)g(w)o(ord)f(aligned)i(transfers)d({)i(see)189
850 y(the)g(advise)h(to)e(implemen)o(tors,)i(b)q(elo)o(w\).)189
925 y(Lo)q(c)o(ks)22 b(are)g(needed)i(to)d(guaran)o(tee)h(atomicit)o(y)g(of)f
(accum)o(ulate)i(or)f(RMW)g(op)q(erations)g(in)h(a)189 981
y(shared)17 b(memory)g(implemen)o(tation,)h(or)f(an)o(y)g(implemen)o(tation)i
(where)e(more)g(than)g(one)h(agen)o(t)189 1038 y(ma)o(y)c(executed)j(RMW)e
(op)q(erations.)21 b(W)l(e)16 b(left)f(the)h(outcome)f(of)g(con\015icting)i
(RMA)f(op)q(erations)189 1094 y(that)11 b(use)h(di\013eren)o(t)g(windo)o(ws)h
(unde\014ned,)h(so)d(as)h(to)f(allo)o(w)i(an)e(implemen)o(tation)j(that)d
(asso)q(ciates)189 1151 y(the)i(lo)q(c)o(k)h(with)g(the)g(windo)o(w,)g(or)f
(parts)g(of)g(it.)20 b(Otherwise,)14 b(it)g(b)q(ecomes)g(necessary)g(to)f
(asso)q(ciate)189 1207 y(lo)q(c)o(ks)18 b(with)f(segmen)o(ts)g(of)g(memory)l
(,)h(or)e(to)h(understand)h(the)g(o)o(v)o(erlap)f(structure)g(of)g(windo)o
(ws.)189 1264 y(\()p Fi(End)e(of)i(r)n(ationale.)p Fu(\))189
1368 y Fi(A)n(dvic)n(e)22 b(to)h(implementors.)82 b Fu(The)23
b(atomicit)o(y)f(requiremen)o(t)h(for)f(accum)o(ulate)h(and)f(RMW)189
1425 y(op)q(erations)c(can)h(b)q(e)g(implemen)o(ted)h(with)f(no)f(lo)q(c)o
(king)i(when)e(the)h(up)q(dates)g(to)f(a)g(windo)o(w)g(are)189
1481 y(done)h(b)o(y)f(a)h(unique)h(RMA)f(agen)o(t:)26 b(the)19
b(up)q(dates)g(are)f(serialized)j(b)o(y)e(the)f(agen)o(t.)30
b(Lo)q(c)o(king)19 b(is)189 1538 y(required)e(in)h(an)f(implemen)o(tation)g
(where)g(RMA)g(accesses)g(to)f(a)h(windo)o(w)f(ma)o(y)g(b)q(e)i(p)q(erformed)
189 1594 y(b)o(y)g(m)o(ultiple)i(threads.)29 b(The)19 b(lo)q(c)o(k)g(gran)o
(ularit)o(y)f(can)g(b)q(e)h(as)f(large)g(as)g(the)h(en)o(tire)g(windo)o(w,)f
(or)189 1650 y(as)d(small)h(as)g(the)f(largest)h(en)o(tit)o(y)f(that)g(can)h
(b)q(e)g(up)q(dated)h(atomically)f({)g(one)f(can)h(trade-o\013)f(the)189
1707 y(added)g(o)o(v)o(erhead)g(of)g(\014ner)h(lo)q(c)o(ks)f(against)g(the)g
(additional)i(concurrency)f(they)f(pro)o(vide.)189 1782 y(Additional)20
b(care)d(is)i(needed)g(to)f(supp)q(ort)g(partial)h(w)o(ord)e(up)q(dates,)i
(on)f(sytems)f(that)h(supp)q(ort)189 1838 y(only)g(w)o(ord)e(aligned)j(DMA,)d
(or)h(w)o(ord)g(aligned)h(memory)f(access.)27 b(Consider)18
b(\014rst)f(the)g(case)g(of)189 1895 y(a)g(partial)i(up)q(date)f(to)g(a)g(w)o
(ord)f(that)g(is)i(fully)g(con)o(tained)g(within)g(the)f(windo)o(w.)29
b(This)18 b(up)q(date)189 1951 y(will)g(b)q(e)g(implemen)o(ted)g(as)f(a)g
(load-mo)q(dify-store)g(cycle,)h(protected)f(b)o(y)g(a)f(lo)q(c)o(k.)26
b(P)o(artial)17 b(w)o(ord)189 2008 y(accesses)i(to)f(suc)o(h)h(w)o(ords)f
(will)i(also)f(b)q(e)g(protected)g(b)o(y)g(the)g(same)f(lo)q(c)o(k.)31
b(F)l(ull)20 b(w)o(ord)e(up)q(dates)189 2064 y(need)e(not)f(b)q(e)h
(protected)f(b)o(y)h(a)f(lo)q(c)o(k.)21 b(This,)15 b(b)q(ecause)i(the)e
(outcome)g(of)g(con\015icting)i(accesses)e(is)189 2121 y(unde\014ned.)21
b(Th)o(us,)14 b(w)o(e)g(only)g(need)h(to)f(w)o(orry)f(ab)q(out)h(\\false)g
(con\015icts")h(b)q(et)o(w)o(een)f(suc)o(h)g(up)q(dates)189
2177 y(that)j(are)h(due)h(to)f(false)g(sharing,)h(i.e.,)g(to)f(concurren)o(t)
g(up)q(dates)h(to)f(non-o)o(v)o(erlaping)h(parts)e(of)189 2233
y(the)e(same)g(w)o(ord.)k(But)c(suc)o(h)h(con\015icts)g(m)o(ust)e(in)o(v)o
(olv)o(e)i(t)o(w)o(o)e(non)h(w)o(ord-aligned)i(accesses.)189
2308 y(The)i(other)g(case)g(to)g(consider)h(is)g(a)f(partial)h(w)o(ord)e(up)q
(date)i(for)f(a)g(w)o(ord)g(that)f(is)i(only)g(partly)189 2365
y(con)o(tained)e(in)g(the)f(windo)o(w,)h(when)f(the)h(windo)o(w)f(b)q
(oundary)h(is)g(not)e(w)o(ord-aligned.)27 b(The)18 b(put)189
2421 y(op)q(eration)11 b(cannot)g(write)g(the)g(part)g(of)f(the)i(w)o(orld)f
(outside)g(the)h(windo)o(w)f(since)h(this)g(ma)o(y)e(con\015ict)189
2478 y(with)16 b(concurren)o(t)g(stores)g(to)f(the)i(part)e(of)h(the)g(w)o
(ord)g(that)f(is)i(outside)g(the)f(windo)o(w.)23 b(\(Lo)q(c)o(king)189
2534 y(will)16 b(not)e(help.\))21 b(A)15 b(solution)g(to)f(this)h(problem)g
(is)h(that)e(suc)o(h)h(w)o(ord)f(is)h(not)f(remotely)h(accessed.)189
2591 y(A)g(safet)o(y)g(area)g(is)h(preserv)o(ed)h(at)e(the)g(windo)o(w)h(b)q
(oundaries.)23 b(This)16 b(area)f(consists)h(of)f(the)h(b)o(ytes)189
2647 y(from)h(the)i(windo)o(w)f(b)q(oundary)h(to)f(the)h(nearest)f(w)o(ord)g
(b)q(oundary)l(.)30 b(A)18 b(separate)g(cop)o(y)h(of)f(this)189
2704 y(safet)o(y)12 b(area)g(is)i(main)o(tained,)g(and)f(an)o(y)g(put/get)f
(access)i(to)e(that)g(area)h(uses)g(this)h(separate)e(cop)o(y)l(.)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 24 26
bop 75 -100 a Fu(24)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)189 49 y Fu(Note)20 b(that)g(there)g(is)h(no)g(added)g(complexit)o(y)
h(for)d(w)o(ord-aligned)j(RMA)f(accesses.)36 b(Users)20 b(of)189
106 y(mac)o(hines)j(that)e(do)h(not)g(supp)q(ort)g(b)o(yte)g(or)g(half-w)o
(ord)g(accesses)h(presumably)g(exp)q(ect)g(that)189 162 y(p)q(erformance)h
(su\013ers)f(for)g(non-aligned)i(accesses.)46 b(The)24 b(same)g(ideas)g
(extend)g(to)f(systems)189 219 y(with)15 b(stronger)f(alignemen)o(t)j
(restrictions,)e(e.g.,)f(a)h(system)f(where)i(DMA)f(is)g(alw)o(a)o(ys)g(cac)o
(he-line)189 275 y(aligned.)21 b(\()p Fi(End)15 b(of)i(advic)n(e)f(to)h
(implementors.)p Fu(\))75 395 y Fl(Con\015icting)f(Lo)q(cal)f(and)h(Remote)d
(Accesses)75 481 y Fu(W)l(e)19 b(prop)q(ose)f(to)g(use)h(a)f(dela)o(y)o(ed)h
(consistency)g(mo)q(del)g(to)f(handle)i(suc)o(h)f(con\015icts.)30
b(This)19 b(allo)o(ws)g(for)75 537 y(more)c(e\016cien)o(t)h(implemen)o
(tations,)g(and)f(will)i(not)e(burden)h(unduly)h(users)e(\(see)g(rationale)h
(b)q(elo)o(w\).)166 594 y(T)l(o)g(understand)h(the)g(coherence)h(mo)q(del,)f
(one)g(ma)o(y)f(think)h(that)f(there)h(are)f(t)o(w)o(o)f(copies)j(of)e(eac)o
(h)75 650 y(RMA)h(windo)o(w:)23 b(a)16 b Fi(private)h Fu(cop)o(y)g(that)f(is)
h(accessed)h(b)o(y)e(lo)q(cal)i(accesses,)f(and)g(a)f Fi(public)h
Fu(cop)o(y)f(that)g(is)75 707 y(accessed)j(b)o(y)g(remote)f(accesses.)30
b(When)20 b(a)e(lo)q(cal)i(store)e(op)q(eration)g(is)i(complete,)g(then)f
(the)f(priv)m(ate)75 763 y(windo)o(w)e(cop)o(y)f(has)h(b)q(een)g(up)q(dated,)
h(but)e(the)h(up)q(date)g(ma)o(y)f(not)g(b)q(e)i(y)o(et)e(visible)j(in)e(the)
g(public)h(cop)o(y;)75 820 y(the)h(new)g(v)m(alue)h(will)h(b)q(e)f(copied)g
(\(b)o(yte)e(b)o(y)h(b)o(yte\))g(in)o(to)g(the)g(public)i(cop)o(y)e(some)f
(time)h(later.)29 b(When)75 876 y(an)15 b(RMA)g(put)h(or)e(accum)o(ulate)i
(or)f(RMW)g(op)q(eration)g(has)g(completed)h(at)f(the)g(target)f(memory)l(,)h
(then)75 933 y(the)22 b(public)i(windo)o(w)e(cop)o(y)g(at)f(the)h(target)f
(has)h(b)q(een)h(up)q(dated;)i(the)d(new)g(v)m(alue)i(will)f(b)q(e)g(copied)
75 989 y(\(b)o(yte)d(b)o(y)h(b)o(yte\))f(to)g(the)h(priv)m(ate)g(cop)o(y)f
(at)g(that)g(pro)q(cess)h(some)g(time)g(later.)36 b(A)20 b(strict)h
(coherence)75 1045 y(mo)q(del)15 b(w)o(ould)g(require)f(that)g(an)g(up)q
(date)h(is)f(propagated)g(from)f(one)h(cop)o(y)g(to)g(another)f(as)h(so)q(on)
g(as)g(the)75 1102 y(up)q(date)h(to)q(ok)f(place.)21 b(A)14
b(dela)o(y)o(ed)i(coherence)f(proto)q(col)g(allo)o(ws)f(one)h(to)f(dela)o(y)h
(the)g(cop)o(ying.)20 b(Windo)o(w)75 1158 y(sync)o(hronization)c(p)q(oin)o
(ts)f(are)g(de\014ned)h(where)f(the)g(coherence)h(b)q(et)o(w)o(een)f(the)g(t)
o(w)o(o)f(copies)h(is)h(restored.)75 1215 y(The)e(cop)o(ying)f(from)g(one)g
(windo)o(w)h(to)f(another)g(o)q(ccurs)g(some)g(time)h(after)e(the)i(up)q
(date)g(to)q(ok)e(place)j(and)75 1271 y(b)q(efore)g(the)h(cop)o(y)f(sync)o
(hronization)h(completes.)166 1328 y(Accesses)22 b(to)f(a)g(windo)o(w)h(are)f
(divided)j(in)f(successiv)o(e)f(ep)q(o)q(c)o(hs.)40 b(Ep)q(o)q(c)o(hs)22
b(are)g(either)g Fi(lo)n(c)n(al)e Fu(or)75 1384 y Fi(r)n(emote)p
Fu(.)31 b(During)20 b(a)e(lo)q(cal)i(ep)q(o)q(c)o(h)g(the)f(\(priv)m(ate)g
(cop)o(y)g(of)g(the\))g(windo)o(w)g(ma)o(y)f(b)q(e)i(up)q(dated)f(lo)q(cally)
l(,)75 1441 y(but)f(the)h(\(public)h(cop)o(y)e(of)g(the\))g(windo)o(w)h
(should)g(not)f(b)q(e)h(up)q(dated)g(remotely)l(.)30 b(Lo)q(cal)19
b(and)g(remote)75 1497 y(read)f(accesses)h(are)f(allo)o(w)o(ed.)29
b(During)19 b(a)f(remote)g(ep)q(o)q(c)o(h,)h(the)g(\(public)h(cop)o(y)e(of)g
(the\))g(windo)o(w)h(ma)o(y)75 1554 y(b)q(e)h(up)q(dated)g(remotely)l(,)f
(but)h(the)f(\(lo)q(cal)h(cop)o(y)e(of)h(the\))g(windo)o(w)g(should)h(not)f
(b)q(e)h(up)q(dated)f(lo)q(cally)l(.)75 1610 y(Lo)q(cal)h(and)f(remote)f
(read)h(accesses)g(are)f(allo)o(w)o(ed.)31 b(A)19 b(lo)q(cal)h(ep)q(o)q(c)o
(h)f(m)o(ust)g(b)q(e)g(completed)h(b)o(y)e(a)h(call)75 1666
y(to)c Fl(MPI)p 216 1666 14 2 v 16 w(WINDO)o(W)p 428 1666 V
16 w(OUT\(comm)n(\))p Fu(,)d(whic)o(h)17 b(completes)g(the)f(transfer)g(of)g
(all)h(lo)q(cal)g(up)q(dates)f(from)g(the)75 1723 y(priv)m(ate)h(windo)o(w)f
(cop)o(y)f(to)h(the)g(public)i(windo)o(w)e(cop)o(y)l(.)22 b(A)16
b(remote)f(ep)q(o)q(c)o(h)i(m)o(ust)e(b)q(e)i(completed)f(b)o(y)g(a)75
1779 y(call)h(to)f Fl(MPI)p 301 1779 V 16 w(WINDO)o(W)p 513
1779 V 16 w(IN\(com)o(m)n(\))o Fu(,)d(whic)o(h)k(completes)g(the)g(transfer)e
(of)h(all)h(remote)f(up)q(dates)h(from)75 1836 y(the)e(public)h(cop)o(y)f(to)
e(the)i(priv)m(ate)g(cop)o(y)l(.)20 b(Th)o(us,)14 b(the)h(coherence)g(b)q(et)
o(w)o(een)g(the)f(t)o(w)o(o)f(windo)o(w)i(copies)g(is)75 1892
y(restored)g(when)g(these)h(t)o(w)o(o)e(functions)i(are)f(called.)75
2043 y Fl(MPI)p 160 2043 V 16 w(WINDO)o(W)p 372 2043 V 16 w(IN\(com)o(m)n(\))
117 2120 y Fk(OUT)108 b Fl(comm)466 b Fk(comm)o(unicator)9
b(asso)q(ciated)k(with)e(RMA)h(windo)o(w)f(\(handle\))75 2245
y Fq(int)23 b(MPI)p 245 2245 15 2 v 17 w(Window)p 406 2245
V 17 w(in\(MPI)p 567 2245 V 16 w(Comm)g(comm\))75 2331 y(MPI)p
150 2331 V 17 w(WINDOW)p 311 2331 V 16 w(IN\()h(COMM,)f(IERROR\))170
2388 y(INTEGER)g(COMM,)h(IERROR)166 2474 y Fu(Completes)13
b(the)h(transfers)e(of)h(all)h(up)q(dates)f(to)g(the)g(public)j(windo)o(w)d
(cop)o(y)g(in)o(to)g(the)g(priv)m(ate)h(cop)o(y)l(.)1967 46
y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 25 27
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(25)75 49 y Fl(MPI)p 160 49 14 2 v 16
w(WINDO)o(W)p 372 49 V 16 w(OUT\(comm)m(\))117 126 y Fk(OUT)108
b Fl(comm)466 b Fk(comm)o(unicator)9 b(asso)q(ciated)k(with)e(RMA)h(windo)o
(w)f(\(handle\))75 251 y Fq(int)23 b(MPI)p 245 251 15 2 v 17
w(Window)p 406 251 V 17 w(out\(MPI)p 591 251 V 16 w(Comm)g(comm\))75
337 y(MPI)p 150 337 V 17 w(WINDOW)p 311 337 V 16 w(OUT\()g(COMM,)h(IERROR\))
170 394 y(INTEGER)f(COMM,)h(IERROR)166 480 y Fu(Completes)15
b(the)f(transfer)g(of)g(all)h(up)q(dates)g(to)f(the)h(priv)m(ate)g(windo)o(w)
f(cop)o(y)h(in)o(to)f(the)h(public)h(cop)o(y)l(.)166 537 y(Supp)q(ose)22
b(that)e(a)g(pro)q(cess)h(lo)q(cally)h(writes)f(a)f(windo)o(w)h(lo)q(cation,)
i(and)d(that)g(lo)q(cation)i(is)f(later)75 593 y(accessed)15
b(remotely)f(b)o(y)g(a)g(get.)19 b(Then,)c(the)f(get)g(will)i(surely)f
(return)f(the)g(v)m(alue)i(of)d(the)i(store)e(pro)o(vided)75
650 y(that)131 743 y(1.)22 b(a)15 b(call)h(to)e Fl(MPI)p 450
743 14 2 v 16 w(WINDO)o(W)p 662 743 V 16 w(OUT)i Fu(o)q(ccurred)g(after)e
(the)i(store)e(and)i(b)q(efore)f(the)g(get.)131 837 y(2.)22
b(An)o(y)16 b(other)h(lo)q(cal)g(or)g(remote)f(up)q(date)h(to)f(that)g(same)g
(lo)q(cation)i(either)f(precedes)h(the)f(store)e(or)189 894
y(follo)o(ws)g(the)g(get.)75 988 y(If)j(the)g(call)h(to)e Fl(MPI)p
433 988 V 15 w(WINDO)o(W)p 644 988 V 16 w(OUT)i Fu(did)g(not)e(tak)o(e)g
(place,)i(the)f(get)f(ma)o(y)g(or)g(ma)o(y)g(not)g(return)h(the)75
1044 y(stored)j(v)m(alue.)40 b(Note)22 b(that)e(in)j(order)e(that)g(the)h
(call)g(to)f Fl(MPI)p 1204 1044 V 16 w(WINDO)o(W)p 1416 1044
V 16 w(OUT)h Fu(precede)h(the)f(get,)75 1100 y(it)d(is)g(necessary)g(that)f
(the)h(t)o(w)o(o)f(pro)q(cesses)h(sync)o(hronize.)31 b(E.g.,)19
b(a)f(message)g(could)i(b)q(e)g(sen)o(t)e(b)o(y)h(the)75 1157
y(\014rst)j(pro)q(cess)g(after)f(the)h(call)h(to)e(windo)o(w)p
845 1157 V 17 w(out)h(and)g(receiv)o(ed)h(b)o(y)f(the)g(other)f(pro)q(cess)i
(b)q(efore)f(the)75 1213 y(call)e(to)e(get.)30 b(Suc)o(h)19
b(sync)o(hronization)g(ma)o(y)f(also)h(o)q(ccur)g(indirectly)l(,)j(via)c(a)h
(c)o(hain)g(of)f(sync)o(hronizing)75 1270 y(comm)o(unications,)d(or)g(ma)o(y)
g(b)q(e)h(done)f(via)h(a)e(collectiv)o(e)j(call,)f(suc)o(h)g(as)f(barrier.)
166 1326 y(Supp)q(ose)k(that)e(an)g(RMA)h(op)q(eration)g(\(suc)o(h)g(as)f
(put\))h(up)q(dates)g(a)f(windo)o(w)h(lo)q(cation,)h(and)f(that)75
1383 y(lo)q(cation)d(is)g(later)f(read)h(lo)q(cally)l(.)21
b(The)15 b(load)f(will)i(surely)f(return)g(the)f(v)m(alue)i(of)e(the)g(put)h
(pro)o(vided)g(that)131 1477 y(1.)22 b(a)15 b(call)h(to)e Fl(MPI)p
450 1477 V 16 w(WINDO)o(W)p 662 1477 V 16 w(IN)h Fu(o)q(ccurred)h(after)f
(the)g(put)g(and)h(b)q(efore)f(the)g(load.)131 1570 y(2.)22
b(An)o(y)14 b(other)g(lo)q(cal)i(\(store\))d(or)h(global)h(\(put,)f(accum)o
(ulate,)h(RMW\))f(up)q(date)h(to)f(that)g(same)g(lo)q(ca-)189
1627 y(tion)h(either)h(precedes)g(the)f(put)h(or)e(follo)o(ws)i(the)f(load.)
75 1721 y(If)j(the)g(call)h(to)e Fl(MPI)p 433 1721 V 16 w(WINDO)o(W)p
645 1721 V 16 w(IN)h Fu(did)h(not)e(tak)o(e)g(place,)j(then)e(the)g(load)g
(ma)o(y)f(or)g(ma)o(y)h(not)f(return)75 1777 y(the)f(v)m(alue)i(of)d(the)i
(put.)22 b(Note)16 b(that)g(in)h(order)e(that)h(the)g(call)h(to)f
Fl(MPI)p 1297 1777 V 16 w(WINDO)o(W)p 1509 1777 V 16 w(IN)g
Fu(follo)o(w)g(the)g(put,)75 1834 y(it)j(is)g(necessary)g(that)f(the)h(t)o(w)
o(o)f(pro)q(cesses)h(sync)o(hronize.)31 b(E.g.,)19 b(a)f(message)g(could)i(b)
q(e)g(sen)o(t)e(b)o(y)h(the)75 1890 y(\014rst)c(pro)q(cess)g(after)f(the)i
(put)f(and)g(receiv)o(ed)h(b)o(y)g(the)f(other)f(pro)q(cess)i(b)q(efore)f
(the)g(call)i(to)d(windo)o(w)p 1783 1890 V 17 w(out;)75 1946
y(or)19 b(a)g(call)h(to)f Fl(MPI)p 409 1946 V 16 w(GET)p 513
1946 V 17 w(COUNTER)h Fu(could)h(ha)o(v)o(e)e(indicated)i(completion)g(of)e
(the)g(put,)h(b)q(efore)g(the)75 2003 y(call)15 b(to)d Fl(MPI)p
295 2003 V 16 w(WINDO)o(W)p 507 2003 V 16 w(IN)p Fu(;)h(or)g(pro)q(cesses)h
(ma)o(y)e(ha)o(v)o(e)h(sync)o(hronized)i(though)e(a)g(collectiv)o(e)j(call)e
(or)f(an)75 2059 y(indirect)k(c)o(hain)f(of)f(sync)o(hronizing)h(ev)o(en)o
(ts.)166 2116 y(Note)g(that)h(races)f(will)j(o)q(ccur)e(if)h(a)e(windo)o(w)h
(lo)q(cation)h(is)f(b)q(oth)g(remotely)g(up)q(dated)h(and)f(lo)q(cally)75
2172 y(read)f(during)g(an)g(external)g(ep)q(o)q(c)o(h)h(for)e(that)g(windo)o
(w.)21 b(The)16 b(lo)q(cal)h(read)f(ma)o(y)f(return)g(an)o(y)h(v)m(alue)h
(that)75 2229 y(is)h(consisten)o(t)g(with)g(the)g(t)o(w)o(o-cop)o(y)f(mo)q
(del;)i(i.e.,)f(the)g(v)m(alue)h(of)e(eac)o(h)h(b)o(yte)g(read)f(ma)o(y)g(b)q
(e)i(the)f(v)m(alue)75 2285 y(of)d(the)h(b)o(yte)g(at)f(the)h(start)f(of)h
(the)g(ep)q(o)q(c)o(h,)g(or)f(an)o(y)h(of)f(the)h(v)m(alues)h(successiv)o
(ely)h(written)e(in)h(this)f(b)o(yte)75 2342 y(b)o(y)h(remote)f(up)q(dates)h
(that)f(precede)i(or)e(are)g(concurren)o(t)h(with)g(the)g(lo)q(cal)g(read.)25
b(And,)17 b(similarly)l(,)i(for)75 2398 y(remote)c(reads)g(during)h(an)f(in)o
(ternal)h(ep)q(o)q(c)o(h.)166 2455 y(The)21 b(sc)o(hematic)g(co)q(des)g(b)q
(elo)o(w)g(illustrate)g(correct)f(use)h(of)f(the)h(calls)h
Fl(MPI)p 1512 2455 V 15 w(WINDO)o(W)p 1723 2455 V 16 w(IN)f
Fu(and)75 2511 y Fl(MPI)p 160 2511 V 16 w(WINDO)o(W)p 372 2511
V 16 w(OUT)16 b Fu(for)e(get)h(after)f(store,)h(put)g(after)f(store)h(and)g
(load)h(after)e(put)h(dep)q(endencie)q(s.)-32 46 y Fm(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 26 28
bop 75 -100 a Fu(26)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)p 165 79 2 69 v 191 49 a Fu(Origin)p 191 65 127 2 v
156 w(T)l(arget)p 472 65 131 2 v 165 148 2 69 v 472 118 a(lo)q(cal)i(stores)p
165 216 V 472 187 a Fl(OUT)p 165 273 2 57 v 472 256 a Fu(some)e(sync.)8
b(to)p 165 325 V 472 308 a(guaran)o(tee...)p 165 382 V 191
365 a(...)191 b Fn(.)p 165 438 V 191 421 a Fu(b)q(eing)16 b(after)p
165 503 2 69 v 191 473 a(the)f(out)p 165 559 2 57 v 191 542
a Fl(GET)193 b Fu(lo)q(cal)17 b(loads)p 165 612 V 472 595 a(allo)o(w)o(ed)p
766 79 2 69 v 791 49 a(Origin)p 791 65 127 2 v 157 w(T)l(arget)p
1073 65 131 2 v 766 148 2 69 v 1073 118 a(lo)q(cal)f(stores)p
766 216 V 1073 187 a Fl(OUT)p 766 273 2 57 v 1073 256 a Fu(some)f(sync.)8
b(to)p 766 325 V 1073 308 a(guaran)o(tee...)p 766 382 V 791
365 a(...)192 b Fn(.)p 766 438 V 791 421 a Fu(b)q(eing)17 b(after)p
766 503 2 69 v 791 473 a(the)f(out)p 766 559 2 57 v 791 542
a Fl(PUT)191 b Fu(lo)q(cal)16 b(stores)p 766 612 V 1073 595
a(forbidden)p 1366 79 2 69 v 1392 49 a(Origin)p 1392 65 127
2 v 192 w(T)l(arget)p 1709 65 131 2 v 1366 148 2 69 v 1392
118 a Fl(PUT)p 1366 204 2 57 v 1392 187 a Fu(RMA-sync.)p 1366
256 V 1392 239 a(to)f(guaran)o(tee)p 1366 313 V 1658 296 a
Fn(&)6 b Fu(that)15 b(the)p 1366 365 V 1709 348 a(data)g(is)p
1366 417 V 1709 400 a(stored)g(in)o(to)p 1366 470 V 1709 453
a(the)h(target)p 1366 534 2 69 v 1709 505 a(windo)o(w)p 1366
603 V 1709 574 a Fl(IN)p 1366 660 2 57 v 1709 643 a Fu(lo)q(cal)h(access)189
724 y Fi(R)n(ationale.)51 b Fu(The)18 b(underlying)h(assumption)e(to)g(the)g
(prop)q(osed)g(design)h(is)g(that,)f(in)h(a)e(correct)189 780
y(program,)e(sync)o(hronization)i(b)q(et)o(w)o(een)g(pro)q(cesses)g(is)g
(only)g(done)g(with)g(message-passing)f(calls,)189 837 y(suc)o(h)20
b(as)f(send-receiv)o(e)j(or)d(barrier,)i(or)e(with)i(accum)o(ulate)f(or)f
(RMW)h(accesses)g(to)f(the)h(same)189 893 y(v)m(ariable,)25
b(for)d(semaphores)g(or)g(similar)i(sync)o(hronizing)g(data)e(structures.)41
b(Tw)o(o)22 b(pro)q(cesses)189 950 y(cannot)11 b(sync)o(hronize)i(b)o(y)e(co)
q(de)h(where)g(one)g(pro)q(cess)g(accesses)g(a)f(v)m(ariable)i(with)f(lo)q
(cal)h(load/store)189 1006 y(op)q(erations,)i(and)g(the)g(other)g(pro)q(cess)
h(accesses)f(that)g(v)m(ariable)h(using)g(RMA)f(op)q(erations.)189
1078 y(The)c(execution)h(order)e(for)g(the)h(parallel)i(co)q(de)e(is)g
(determined)i(b)o(y)d(the)h(ordering)g(of)g(these)g(explicit)189
1135 y(syc)o(hronization)h(ev)o(en)o(ts.)18 b(Th)o(us,)12 b(if)g(a)g(lo)q
(cal)g(load)g(accesses)g(the)g(same)f(lo)q(cation)i(as)e(a)g(remote)g(put,)
189 1191 y(then)g(these)h(t)o(w)o(o)e(accesses)i(m)o(ust)e(b)q(e)i(ordered)g
(b)o(y)f(explicit)j(sync)o(hronization)e(ev)o(en)o(ts.)18 b(Therefore,)189
1248 y(a)c(dela)o(y)o(ed)h(sync)o(hronization)g(mo)q(del,)h(where)e
(coherence)i(b)q(et)o(w)o(een)e(lo)q(cal)i(and)f(remote)f(accesses)189
1304 y(is)22 b(restored)f(only)i(at)e(sync)o(hronization)h(p)q(oin)o(ts)h(is)
f(adequate,)h(since)g(it)f(will)h(not)f(a\013ect)f(the)189
1360 y(seman)o(tics)c(of)f(correct)g(programs.)24 b(Dela)o(y)o(ed)17
b(coherence)h(allo)o(ws)f(for)f(impro)o(v)o(ed)h(p)q(erformance,)189
1417 y(esp)q(ecially)g(on)f(systems)e(where)i(explicit)h(action)e(is)h
(needed)h(for)d(coherence.)189 1489 y(Dela)o(y)o(ed)h(coherence)h(could)h(b)q
(e)f(implemen)o(ted)h(implicitly)l(,)h(b)o(y)d(p)q(erforming)h(coherence)g
(actions)189 1545 y(on)j(eac)o(h)h(windo)o(w,)g(for)f(eac)o(h)g(MPI)h(call.)
34 b(But)19 b(this)h(w)o(ould)g(b)q(e)g(an)g(exp)q(ensiv)o(e)h(solution.)33
b(In-)189 1602 y(stead,)21 b(w)o(e)e(pro)o(vide)i(explicit)i(coherence)e
(enforcing)g(calls,)h(namely)f Fl(MPI)p 1512 1602 14 2 v 16
w(WINDO)o(W)p 1724 1602 V 16 w(IN)f Fu(and)189 1658 y Fl(MPI)p
274 1658 V 15 w(WINDO)o(W)p 485 1658 V 17 w(OUT)p Fu(.)15 b(It)g(is)g(the)g
(user)h(resp)q(onsibilit)o(y)h(to)d(insert)i(those)f(calls)h(at)e(sync)o
(hroniza-)189 1715 y(tion)h(p)q(oin)o(ts)h(\()p Fi(End)f(of)i(r)n(ationale.)p
Fu(\))189 1799 y Fi(A)n(dvic)n(e)i(to)i(implementors.)66 b
Fu(In)21 b(an)e(implemen)o(tation)i(where)f(lo)q(cal)h(accesses)f(are)f
(coheren)o(t)189 1856 y(with)h(remote)f(accesses,)i(then)g(the)f(calls)h(to)e
Fl(MPI)p 1087 1856 V 16 w(WINDO)o(W)p 1299 1856 V 16 w(IN)g
Fu(and)h Fl(MPI)p 1554 1856 V 16 w(WINDO)o(W)p 1766 1856 V
16 w(OUT)189 1912 y Fu(are)e(no)q(ops:)26 b(in)19 b(e\013ect,)f(the)g(t)o(w)o
(o)f(windo)o(w)i(copies)g(are)f(iden)o(tical.)31 b(Suc)o(h)19
b(is)f(the)h(case)f(if)h(RMA)189 1969 y(remote)e(accesses)h(are)f(executed)i
(b)o(y)e(a)h(soft)o(w)o(are)d(agen)o(t)i(running)i(on)f(the)g(same)f(pro)q
(cessor)g(as)189 2025 y(the)e(lo)q(cal)h(pro)q(cess,)f(or)f(on)h(a)g(comm)o
(unication)h(copro)q(cessor)e(that)h(is)g(cac)o(he)h(coheren)o(t)f(with)g
(the)189 2082 y(computation)f(pro)q(cessor.)19 b(The)14 b(same)g(holds)g
(true)g(in)h(an)f(implemen)o(tation)h(of)f(RMA)g(on)g(top)f(of)189
2138 y(coheren)o(t)i(shared)h(memory)l(,)f(where)h(remote)f(accesses)h(are)f
(p)q(erformed)h(directly)h(b)o(y)f(the)f(origin)189 2195 y(pro)q(cess.)189
2267 y(The)h(non)o(trivial)g(case)g(is)g(where)g(RMA)g(remote)f(accesses)h
(are)f(not)g(cac)o(he)h(coheren)o(t)g(with)g(lo)q(cal)189 2323
y(accesses.)k(There)15 b(are)g(sev)o(eral)g(cases)h(to)e(consider)231
2411 y(1.)22 b(RMA)f(is)h(implemen)o(ted)h(using)g(non-coheren)o(t)e(shared)h
(memory)l(,)g(with)g(write-through)289 2467 y(incoheren)o(t)g(cac)o(hes.)37
b(Both)21 b(lo)q(cal)i(and)e(remote)f(writes)i(go)e(directly)i(to)f(memory)l
(,)h(but)289 2524 y(b)q(oth)15 b(lo)q(cal)h(and)g(remote)e(reads)h(ma)o(y)g
(go)f(to)h(cac)o(he)g(\(Cra)o(y)f(T90,)g(Cra)o(y)h(T3D?\).)231
2591 y(2.)22 b(RMA)e(is)h(implemen)o(ted)i(using)e(non-coheren)o(t)g(shared)f
(memory)l(,)h(with)g(write-bac)o(k)g(in-)289 2647 y(coheren)o(t)c(cac)o(hes.)
28 b(Both)17 b(lo)q(cal)i(and)f(remote)f(accesses)h(ma)o(y)f(go)g(to)g(cac)o
(he,)h(rather)f(than)289 2704 y(memory)d(\(NEC)h(Cenju-3,)g(Mercury)g(RA)o
(CE)g(i860?\).)1967 46 y Fm(1)1967 103 y(2)1967 159 y(3)1967
215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967
498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724
y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959
1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959
1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959
1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959
1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959
1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959
2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959
2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959
2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 27 29
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(27)231 49 y(3.)22 b(RMA)d(is)h(implemen)o(ted)h
(using)f(a)f(non-coheren)o(t)h(DMA)e(engine.)34 b(Remote)19
b(accesses)h(go)289 106 y(directly)g(to)e(memory)l(,)h(whereas)g(lo)q(cal)h
(accesses)f(\(b)q(oth)g(reads)g(and)g(writes\))f(ma)o(y)g(go)h(to)289
162 y(cac)o(he)c(\(Cra)o(y)f(T3D?\).)189 256 y(Assume)h(that)g(the)h(windo)o
(w)f(is)h(cac)o(he)g(line)h(aligned.)22 b(A)16 b(call)g(to)f
Fl(MPI)p 1404 256 14 2 v 16 w(WINDO)o(W)p 1616 256 V 16 w(OUT\(comm)m(\))189
312 y Fu(is)k(a)f(no)q(op)g(in)h(the)g(\014rst)f(case)g(\(except)h(that)e(a)h
(memory)g(sync)h(ma)o(y)e(b)q(e)i(needed)h(to)e(force)g(the)189
369 y(stores)12 b(out)h(of)g(the)g(store)g(queue,)h(on)f(some)g(systems\).)19
b(In)14 b(the)f(second)h(and)f(third)h(case,)f(the)h(call)189
425 y(will)k(\015ush)g(the)f(cac)o(he)g(lines)h(in)g(the)f(windo)o(w)g(from)f
(the)h(calling)i(pro)q(cessor)d(cac)o(he)h(to)g(memory)189
482 y(\(it)e(is)g(su\016cien)o(t)h(to)f(\015ush)h(dirt)o(y)f(lines\).)189
557 y(A)20 b(call)h(to)e Fl(MPI)p 476 557 V 16 w(WINDO)o(W)p
688 557 V 16 w(IN)h Fu(will)i(in)o(v)m(alidate)g(the)e(cac)o(he)g(lines)i(in)
f(the)f(windo)o(w)h(from)e(the)189 613 y(calling)e(pro)q(cessor)e(cac)o(he)g
(\(it)g(is)h(su\016cien)o(t)g(to)e(in)o(v)m(alidate)k(lines)e(that)f(w)o(ere)
g(remotely)g(up)q(dated)189 670 y(in)j(the)f(last)g(ep)q(o)q(c)o(h\).)26
b(Since)19 b(no)e(lo)q(cal)h(accesses)f(w)o(ere)g(p)q(erformed)h(in)f(the)h
(last)f(ep)q(o)q(c)o(h,)h(all)g(the)189 726 y(windo)o(w)e(cac)o(he)h(lines)h
(are)e(clean,)h(and)g(a)f(\015ush)h(call)g(ma)o(y)f(b)q(e)h(used,)g(instead)g
(of)f(an)g(in)o(v)m(alidate.)189 783 y(\(A)f(\\\015ush")h(copies)h(bac)o(k)e
(the)h(dirt)o(y)g(lines)h(to)f(memory)f(and)h(in)o(v)m(alidate)h(the)f(en)o
(tries)g(in)h(cac)o(he;)189 839 y(an)g(\\in)o(v)m(alidate")i(do)q(es)f(not)g
(cop)o(y)f(bac)o(k)h(the)f(lines)j(in)e(memory)l(.)27 b(On)18
b(some)g(systems,)f(\015ush)h(is)189 896 y(allo)o(w)o(ed)c(in)g(user)g(mo)q
(de,)g(whereas)g(in)o(v)m(alidate)h(requires)g(sup)q(ervisory)f(mo)q(de,)g
(so)g(that)f(a)g(\015ush)h(is)189 952 y(c)o(heap)q(er.)20 b(W)l(e)14
b(assume)h(that)f(the)g(\015ush)h(implemen)o(tation)h(is)f(smart)f(enough)h
(not)f(to)g(cop)o(y)g(bac)o(k)189 1009 y(to)g(memory)h(clean)h(lines.\))189
1084 y(In)h(addition,)g(in)g(the)f(second)h(case,)g(the)f(co)q(de)h(for)f
(put,)g(accum)o(ulate)h(or)e(RMW)i(requires)g(that)189 1140
y(the)f(written)h(data)f(b)q(e)h(\015ushed)h(from)e(the)h(origin)g(cac)o(he)g
(b)q(efore)g(the)g(RMA)g(call)g(is)h(allo)o(w)o(ed)f(to)189
1197 y(complete)c(at)f(the)h(target.)18 b(In)c(the)f(\014rst)f(case,)h(a)g
(memory)f(sync)h(ma)o(y)f(b)q(e)i(needed)g(to)e(\015ush)i(stores)189
1253 y(out)g(of)h(the)g(store)g(queue.)189 1328 y(An)f(additional)h(problem)g
(o)q(ccurs)f(at)f(windo)o(w)h(b)q(oundaries,)h(when)g(the)f(gran)o(ularit)o
(y)f(of)g(lo)q(cal)i(or)189 1385 y(remote)g(accesses)h(implies)i(that)e(an)f
(up)q(date)i(inside)h(the)e(windo)o(w)g(ma)o(y)f(cause)h(a)g(store)f(outside)
189 1441 y(the)h(windo)o(w,)g(and)h(vice-v)o(ersa.)24 b(This)17
b(problem)g(can)f(b)q(e)h(solv)o(ed)g(b)o(y)f(k)o(eeping)h(a)f(safet)o(y)g
(area)g(at)189 1498 y(the)g(windo)o(w)g(b)q(oundaries,)h(as)f(describ)q(ed)i
(in)f(the)f(previous)h(implemen)o(tation)g(note.)23 b(In)17
b(e\013ect,)189 1554 y(the)j(system)g(main)o(tains)h(t)o(w)o(o)e(copies)i(of)
f(this)h(safet)o(y)e(area,)i(one)g(used)f(for)g(remote)g(accesses,)189
1610 y(and)d(the)h(other)f(used)h(for)f(lo)q(cal)h(accesses.)27
b(When)18 b(a)f(call)i(to)e Fl(MPI)p 1369 1610 V 16 w(WINDO)o(W)p
1581 1610 V 16 w(IN)g Fu(o)q(ccurs,)h(the)189 1667 y(external)h(cop)o(y)g(is)
h(copied)g(on)o(to)e(the)i(in)o(ternal)g(cop)o(y;)g(when)g(a)f(call)h(to)e
Fl(MPI)p 1554 1667 V 16 w(WINDO)o(W)p 1766 1667 V 16 w(OUT)189
1723 y Fu(o)q(ccurs,)d(the)g(in)o(ternal)h(cop)o(y)f(is)h(copied)g(on)o(to)f
(the)g(external)h(cop)o(y)l(.)189 1799 y(The)d(same)g(soft)o(w)o(are)e
(approac)o(h)i(to)g(coherence)h(can)f(b)q(e)h(used)g(on)f(systems)g(where)h
(shared)f(mem-)189 1855 y(ory)g(is)i(a)o(v)m(ailable,)h(in)f(cases)f(where)h
(the)f(windo)o(w)h(is)f(not)g(in)i(shared)e(memory)l(.)19 b(One)c(will)h
(reserv)o(e)189 1911 y(in)k(shared)f(memory)g(a)g(public)j(cop)o(y)d(of)g
(the)g(windo)o(w.)33 b(Lo)q(cal)20 b(accesses)g(to)f(the)g(windo)o(w)h(ac-)
189 1968 y(cess)14 b(the)h(priv)m(ate)g(cop)o(y)l(,)f(whereas)h(RMA)g(remote)
e(accesses)i(access)g(the)f(public)j(cop)o(y)e(in)g(shared)189
2024 y(memory)l(.)31 b(When)20 b Fl(MPI)p 612 2024 V 16 w(WINDO)o(W)p
824 2024 V 16 w(IN)f Fu(is)g(called,)j(then)d(the)h(public,)i(shared)d(cop)o
(y)g(is)h(copied)189 2081 y(in)o(to)e(the)h(priv)m(ate)h(cop)o(y;)g(when)f
Fl(MPI)p 854 2081 V 16 w(WINDO)o(W)p 1066 2081 V 16 w(OUT)h
Fu(is)f(called,)i(then)e(the)g(priv)m(ate)h(cop)o(y)e(is)189
2137 y(copied)k(on)o(to)d(the)i(public,)j(shared)d(cop)o(y)l(.)37
b(V)l(arious)21 b(optimizations)g(\(similar)h(to)e(those)g(used)189
2194 y(in)e(DSVM)g(systems\))f(can)i(b)q(e)f(applied)i(to)e(a)o(v)o(oid)f
(full)j(windo)o(w)e(cop)o(ying.)29 b(\()p Fi(End)18 b(of)h(advic)n(e)g(to)189
2250 y(implementors.)p Fu(\))166 2432 y Fh(Alternativ)o(es)o(:)34
b Fk(A)14 b(p)q(ossible)g(alternativ)o(e)f(is)g(to)h(drop)f(the)h(condition)f
(that)h(a)f(windo)o(w)f(can)i(b)q(e)g(up)q(dated)75 2482 y(only)d(lo)q(cally)
e(or)j(only)e(remotely)h(during)g(an)g(ep)q(o)q(c)o(h.)18 b(E.g.,)10
b(b)o(y)h(allo)o(wing)e(b)q(oth)j(t)o(yp)q(es)g(of)e(up)q(dates,)j(if)d(they)
i(access)75 2532 y(distinct)i(lo)q(cations.)166 2582 y(The)g(di\016cult)o(y)m
(,)d(as)i(compared)g(to)g(the)h(main)d(prop)q(osal,)i(is)g(that)g(it)g(is)g
(not)g(p)q(ermissible)g(an)o(ymore)f(to)h(cop)o(y)75 2632 y(the)j(en)o(tire)g
(windo)o(w)e(when)h(windo)o(w)p 667 2632 13 2 v 15 w(in)f(or)h(windo)o(w)p
921 2632 V 14 w(out)h(are)f(called.)21 b(This,)15 b(b)q(ecause)i(b)q(oth)e
(the)h(priv)n(ate)f(cop)o(y)75 2682 y(and)10 b(the)h(public)g(cop)o(y)f(ma)o
(y)f(ha)o(v)o(e)h(b)q(een)h(up)q(dated)h(during)e(the)h(last)f(ep)q(o)q(c)o
(h,)i(in)e(di\013eren)o(t)h(parts,)h(and)e(the)h(up)q(dates)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 28 30
bop 75 -100 a Fu(28)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fk(need)j(to)e(b)q(e)h(merged.)26 b(Supp)q(ose)17
b(one)g(main)o(tains)d(a)j(dirt)o(y)f(bit)g(for)h(eac)o(h)g(b)o(yte.)26
b(The)17 b(dirt)o(y)g(bit)f(for)g(a)g(b)o(yte)h(in)75 99 y(the)d(priv)n(ate)g
(cop)o(y)f(is)h(set)g(b)o(y)f(a)h(lo)q(cal)e(up)q(date)j(to)e(the)h(b)o(yte,)
g(and)f(the)h(dirt)o(y)g(bit)f(for)g(a)g(b)o(yte)h(in)g(the)g(public)f(cop)o
(y)75 149 y(is)j(set)g(b)o(y)g(a)f(remote)g(up)q(date)i(to)e(the)h(b)o(yte.)
24 b(When)16 b(a)f(call)g(to)h Fj(MPI)p 1169 149 13 2 v 14
w(WINDO)o(W)p 1362 149 V 15 w(OUT)f Fk(o)q(ccurs,)i(then)f(an)o(y)g(b)o(yte)
75 199 y(that)i(is)f(dirt)o(y)g(in)g(the)h(priv)n(ate)g(cop)o(y)f(is)h
(copied)f(to)h(the)g(public)f(cop)o(y)h(and)f(mark)o(ed)f(clean.)29
b(When)18 b(a)f(call)g(to)75 249 y Fj(MPI)p 152 249 V 14 w(WINDO)o(W)p
345 249 V 15 w(IN)12 b Fk(o)q(ccurs,)h(then)g(an)o(y)e(b)o(yte)h(that)g(is)g
(dirt)o(y)g(in)f(the)i(public)e(cop)o(y)h(is)g(copied)g(to)g(the)g(priv)n
(ate)g(cop)o(y)75 298 y(and)i(mark)o(ed)e(clean.)166 348 y(It)19
b(is)f(hard)h(to)f(implemen)o(t)e(this)i(proto)q(col)h(e\016cien)o(tly)f(on)h
(systems)f(with)g(non-coheren)o(t)i(cac)o(hes.)34 b(The)75
398 y(\015ush)12 b(in)f Fj(MPI)p 295 398 V 15 w(COPY)p 423
398 V 13 w(OUT)g Fk(should)h(\015ush)g(only)f(dirt)o(y)g(b)o(ytes,)h(to)g(a)o
(v)o(oid)e(o)o(v)o(erwriting)h(b)o(ytes)i(that)f(w)o(ere)g(up)q(dated)75
448 y(remotely)m(.)26 b(But,)19 b(t)o(ypically)m(,)c(a)i(dirt)o(y)g(line)g
(is)g(\015ushed)h(in)f(its)g(en)o(tiret)o(y)m(,)g(ev)o(en)h(if)f(only)f(some)
g(of)h(the)h(b)o(ytes)g(are)75 498 y(dirt)o(y)d(\(the)h(cac)o(he)g(main)o
(tains)d(only)h(one)i(dirt)o(y)f(bit)g(p)q(er)h(line\).)22
b(Th)o(us,)15 b(if)f(part)i(of)f(the)g(line)g(w)o(as)g(up)q(dated)h(b)o(y)f
(a)75 547 y(lo)q(cal)g(store)h(and)f(part)h(w)o(as)g(up)q(dated)g(b)o(y)f(a)g
(remote)h(put,)f(the)h(\015ush)g(will)e(o)o(v)o(erwrite)i(the)g(put.)24
b(Similarl)o(y)m(,)12 b(the)75 597 y(in)o(v)n(alidate)f(in)h
Fj(MPI)p 387 597 V 14 w(COPY)p 514 597 V 13 w(IN)g Fk(should)g(in)o(v)n
(alidate)f(only)h(b)o(ytes)h(that)f(where)i(remotely)d(up)q(dated;)i(ho)o(w)o
(ev)o(er,)g(one)75 647 y(can)j(only)g(in)o(v)n(alidate)e(full)h(lines.)24
b(Th)o(us,)17 b(if)e(part)h(of)g(the)h(line)e(w)o(as)h(up)q(dated)h(b)o(y)f
(a)g(lo)q(cal)f(store)i(and)f(part)g(w)o(as)75 697 y(up)q(dated)i(b)o(y)g(a)f
(remote)g(put,)h(the)g(in)o(v)n(alidate)e(will)g(cause)j(the)f(store)g(to)g
(b)q(e)g(lost.)28 b(It)18 b(w)o(ould)f(seem)g(that)h(one)75
747 y(needs)d(to)e(use)h(soft)o(w)o(are)f(tec)o(hniques)h(similar)d(to)i
(those)h(used)g(in)f(DSVM)g(systems)g(in)g(order)h(to)f(implem)o(en)o(t)e
(this)75 796 y(sc)o(heme.)166 846 y(Another)16 b(option)e(is)h(to)g(a)o(v)o
(oid)e(the)j(use)g(of)e(explicit)h(windo)o(w)p 1147 846 V 14
w(in)f(and)h(windo)o(w)p 1430 846 V 14 w(out)g(coherence)i(calls,)e(but)75
896 y(sp)q(ecify)j(that)f(con\015icting)g(lo)q(cal)g(and)g(remote)g(accesses)
j(are)d(erroneous:)27 b(suc)o(h)18 b(accesses)i(m)o(ust)c(b)q(e)i(explicitly)
75 946 y(sync)o(hronized.)166 996 y(Supp)q(ose)d(that)g(a)f(pro)q(cess)i
(stores)f(a)f(v)n(alue)g(at)g(a)g(lo)q(cation)g(in)f(a)h(windo)o(w,)g(and)g
(that)g(lo)q(cation)f(is)i(accessed)75 1046 y(using)e(an)g(RMA)g(op)q
(eration)g(\(suc)o(h)h(as)f(get\))g(b)o(y)g(another)h(pro)q(cess)h(\(or)e(p)q
(ossibly)g(the)h(same)e(pro)q(cess\).)20 b(Then)13 b(the)75
1095 y(get)h(returns)i(the)e(v)n(alue)f(of)h(the)g(store)h(pro)o(vided)f
(that)134 1170 y(1.)22 b(The)14 b(store)h(precedes)h(the)f(get.)134
1245 y(2.)22 b(An)o(y)11 b(other)h(lo)q(cal)e(\(store\))i(or)f(global)f
(\(put,)i(accum)o(ulate,)e(RMW\))g(up)q(date)i(to)f(that)h(same)e(lo)q
(cation)g(either)189 1295 y(precedes)16 b(the)f(store)f(or)g(follo)o(ws)e
(the)j(get.)166 1369 y(Supp)q(ose)e(that)g(an)f(RMA)g(op)q(eration)g(\(suc)o
(h)i(as)e(put\))h(up)q(dates)h(a)e(windo)o(w)f(lo)q(cation,)h(and)g(that)h
(lo)q(cation)e(is)75 1419 y(later)i(accessed)i(b)o(y)d(the)h(lo)q(cal)f(pro)q
(cess)j(using)d(a)g(load.)17 b(The)c(load)f(returns)i(the)f(v)n(alue)f(of)g
(the)i(put)e(pro)o(vided)h(that)134 1494 y(1.)22 b(the)14 b(put)g(precedes)j
(the)d(load.)134 1569 y(2.)22 b(An)o(y)11 b(other)h(lo)q(cal)e(\(store\))i
(or)f(global)f(\(put,)i(accum)o(ulate,)e(RMW\))g(up)q(date)i(to)f(that)h
(same)e(lo)q(cation)g(either)189 1618 y(precedes)16 b(the)f(put)f(or)f(follo)
o(ws)g(the)h(load.)166 1693 y(If)h(a)h(remote)f(RMA)h(get)g(follo)o(ws)e(a)i
(lo)q(cal)e(store)j(then)g(the)f(lo)q(cal)f(pro)q(cess)j(m)o(ust)c(ha)o(v)o
(e)i(executed)i(a)d(send-)75 1743 y(lik)o(e)h(MPI)i(call)e(after)i(the)g
(store)g(and)f(b)q(efore)h(the)f(get.)29 b(\(A)17 b(send-lik)o(e)g(op)q
(eration)g(is)h(a)e(send,)j(or)e(a)g(collectiv)o(e)75 1793
y(comm)o(unication)8 b(call,)j(an)h(RMA)f(call,)g(etc.;)h(i.e.,)f(an)g(op)q
(eration)h(that)g(is)f(kno)o(wn)g(to)h(precede)i(another)e(matc)o(hing)75
1843 y(op)q(eration)k(at)f(another)h(pro)q(cess\).)26 b(If)16
b(a)f(remote)g(RMA)h(put)g(precedes)i(a)e(lo)q(cal)f(load,)f(then)j(the)f(lo)
q(cal)f(pro)q(cess)75 1892 y(m)o(ust)9 b(ha)o(v)o(e)g(executed)j(a)e(receiv)o
(e-lik)o(e)g(MPI)g(call)f(after)h(the)g(put)g(and)g(b)q(efore)h(the)f(load.)
16 b(\(A)10 b(receiv)o(e-lik)o(e)g(op)q(eration)75 1942 y(is)16
b(a)g(receiv)o(e,)i(a)e(collectiv)o(e)h(call,)e(an)i(access)h(to)e(a)g(windo)
o(w)g(coun)o(ter,)h(a)f(get,)h(etc.;)h(i.e.,)d(an)h(op)q(eration)h(that)f(is)
75 1992 y(kno)o(wn)d(to)h(follo)o(w)e(a)h(matc)o(hing)f(op)q(eration)h(at)h
(another)g(pro)q(cess.\))20 b(The)14 b(seman)o(tics)g(of)f(the)h(last)f
(option)g(can)h(b)q(e)75 2042 y(obtained)i(b)o(y)f(treating)h(an)o(y)g
(send-lik)o(e)g(call)f(as)h(a)g(call)f(to)g Fj(MPI)p 1095 2042
V 15 w(WINDO)o(W)p 1289 2042 V 14 w(OUT)p Fk(,)g(and)h(an)o(y)f(receiv)o
(e-lik)o(e)i(MPI)75 2092 y(call)c(as)h(a)g(call)f(to)h Fj(MPI)p
441 2092 V 14 w(WINDO)o(W)p 634 2092 V 14 w(IN)g Fk(\(some)f(calls)h(can)g(b)
q(e)g(b)q(oth\).)166 2148 y(The)g(di\016cult)o(y)f(in)h(implemen)o(tatio)o(n)
d(is)j(that)g(eac)o(h)h(MPI)f(op)q(eration)g(will)e(no)o(w)i(a\013ect)h(all)d
(windo)o(ws.)18 b(This)75 2205 y(will)f(p)q(oten)o(tially)f(add)i(a)g
(signi\014can)o(t)g(o)o(v)o(erhead)g(to)g(all)f(MPI)h(calls)g(on)g
(non-coheren)o(t)h(systems.)31 b(A)18 b(p)q(ossible)75 2261
y(compromise)d(is)j(to)f(sp)q(ecify)i(that)e(con\015icting)h(accesses)i(in)o
(v)o(olving)15 b(a)j(windo)o(w)e(m)o(ust)h(b)q(e)h(co)q(ordinated)g(using)75
2317 y(sp)q(eci\014c)g(MPI)e(calls.)24 b(One)17 b(c)o(hoice)g(is)f(that)g
(accesses)j(to)d(a)g(windo)o(w)f(m)o(ust)g(b)q(e)i(sync)o(hronized)g(with)f
(MPI)g(calls)75 2374 y(whic)o(h)d(ha)o(v)o(e)h(the)g(windo)o(w)e(comm)o
(unicator)f(as)j(an)f(argumen)o(t.)k(This)c(includes)h(barrier,)g(fence,)g
(send)g(or)g(receiv)o(e)75 2430 y(and)d(windo)o(w)f(coun)o(ter)j(accesses.)19
b(Th)o(us,)12 b(only)e(MPI)i(calls)f(with)g(this)g(comm)o(unicator)d(argumen)
o(t)j(will)e(b)q(e)j(slo)o(w)o(ed)75 2487 y(do)o(wn)i(with)f(windo)o(w)g
(coherence)j(actions.)i(This)c(last)g(option)f(should)h(b)q(e)g(considered)h
(seriously)m(.)166 2626 y Fu(Windo)o(w)g(coun)o(ter)g(incremen)o(t)h(is)g
(alw)o(a)o(ys)e(atomic.)1967 46 y Fm(1)1967 103 y(2)1967 159
y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 29 31
bop 75 -100 a Fo(4.8.)29 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)g(COHERENCE)h
(OPERA)l(TIONS)406 b Fu(29)75 49 y Fp(Example)18 b(4.5)k Fu(W)l(e)16
b(presen)o(t)f(sev)o(eral)h(examples)g(that)f(explain)i(the)e(use)h(of)f
(memory)f(coherence)j(op-)75 106 y(erations.)166 162 y(Example)i(1:)25
b(pro)q(ducer-consumer;)20 b(pro)q(ducer)f(main)o(tains)g(queue)g(p)q(oin)o
(ters)f(and)h(pushes)g(data)75 219 y(to)c(the)g(consumer)g(memory)l(.)75
312 y Fq(main\(\))23 b({)75 369 y(typedef)g(struct)g({)147
425 y(float)g(somedata[6];)147 482 y(int)g(somedross[7];)147
538 y(char)g(someotherdross[8];)147 595 y(})g(SOMEOBJECT;)75
651 y(SOMEOBJECT)f(*object;)75 708 y(long)h(int)h(produced,)e(consumed;)75
821 y(blens[3])h(=)g({6,)h(7,)g(8};)75 877 y(MPI_Datatypes)e(types[3])h(=)h
({MPI_FLOAT,)e(MPI_INT,)h(MPI_CHAR};)75 933 y(MPI_Datatype)f(objecttype;)75
990 y(MPI_Comm)h(comm1,)g(comm2;)75 1103 y(MPI_Type_simple_struct\(3,)d
(blens,)j(types,)h(objecttype\);)75 1159 y(MPI_Type_commit\(objecttype)o(\);)
75 1272 y(if)g(\(myrank)e(==)i(0\))g({)f(/*)h(producer)f(code)g(*/)147
1385 y(object)g(=)g(\(SOMEOBJECT*\)malloc\(sizeof\(SO)o(MEOBJEC)o(T\)\);)147
1442 y(MPI_RMA_init\(NULL,)e(0,)j(0,)f(MPI_COMM_WORLD;)f(comm1\);)147
1498 y(MPI_RMA_init\(NULL,)f(0,)j(0,)f(MPI_COMM_WORLD;)f(comm2\);)147
1611 y(while)h(\(1\))g({)218 1667 y(produce\(object\);)218
1724 y(MPI_Put\(object,)f(1,)i(objecttype,)e(1,)i(produced\045QLEN,)910
1780 y(1,)g(objecttype,)e(1,)i(comm1\);)218 1837 y(while\()f(produced)g(>=)h
(\(consumed+QLEN\)\))290 1893 y(MPI_Get\(&consumed,)d(1,)j(MPI_LONGINT,)e(1,)
862 1950 y(0,)i(MPI_LONGINT,)e(0,)i(comm2\);)218 2006 y(produced++;)218
2063 y(})75 2175 y(elseif)f(\(myrank)g(==)h(1\))f({)h(/*)f(consumer)g(code)h
(*/)147 2288 y(object)f(=)g(\(SOMEOBJECT*\)malloc\(QLEN*size)o(of\(SOME)o
(OBJECT\))o(\);)147 2345 y(MPI_RMA_init\(&object,)e
(QLEN*sizeof\(SOMEOBJECT\))o(,)457 2401 y(sizeof\(SOMEOBJECT},)g
(MPI_COMM_WORLD,)h(comm1\);)147 2458 y(MPI_RMA_INIT\(&consumed,)e(1,)k
(sizeof\(consumed\),)d(MPI_COMM_WORLD,)h(comm2\);)147 2571
y(while)h(\(1\))g({)218 2627 y(while)g(\(consumed)g(>=)h(produced\))290
2684 y(MPI_Get_counter\(comm1,)d(&produced\);)-32 46 y Fm(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 30 32
bop 75 -100 a Fu(30)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)218 49 y Fq(MPI_Window_in\(comm1\);)218 106 y
(consume\(object+consumed\045QLE)o(N\);)218 162 y(consumed++;)218
219 y(MPI_Window_out\(comm2\))147 275 y(})75 332 y(})166 417
y Fu(Example)h(2:)j(similar)e(consumer-pro)q(ducer)f(co)q(de,)f(but)h
(consumer)f(pulls)i(data)d(from)h(pro)q(ducer.)75 502 y Fq(main\(\))23
b({)75 558 y(typedef)g(struct)g({)147 615 y(float)g(somedata[6];)147
671 y(int)g(somedross[7];)147 728 y(char)g(someotherdross[8];)147
784 y(})g(SOMEOBJECT;)75 841 y(SOMEOBJECT)f(*object;)75 897
y(long)h(int)h(produced,)e(consumed;)75 1010 y(blens[3])h(=)g({6,)h(7,)g(8};)
75 1066 y(MPI_Datatypes)e(types[3])h(=)h({MPI_FLOAT,)e(MPI_INT,)h(MPI_CHAR};)
75 1123 y(MPI_Datatype)f(objecttype;)75 1179 y(MPI_Comm)h(comm1,)g(comm2;)75
1292 y(MPI_Type_simple_struct\(3,)d(blens,)j(types,)h(objecttype\);)75
1349 y(MPI_Type_commit\(objecttype)o(\);)75 1462 y(if)g(\(myrank)e(==)i(0\))g
({)f(/*)h(producer)f(code)g(*/)147 1574 y(object)g(=)g
(\(SOMEOBJECT*\)malloc\(QLEN*size)o(of\(SOME)o(OBJECT\))o(\);)147
1631 y(MPI_RMA_init\(&object,)e(QLEN*sizeof\(SOMEOBJECT\))o(,)457
1687 y(sizeof\(SOMEOBJECT},)g(MPI_COMM_WORLD,)h(comm1\);)147
1744 y(MPI_RMA_INIT\(&produced,)e(1,)k(sizeof\(produced\),)d(MPI_COMM_WORLD,)
h(comm2\);)147 1857 y(while)h(\(1\))g({)218 1913 y(while)g(\(produced)g(>=)h
(\(consumed+QLEN\)\))290 1970 y(MPI_Get_counter\(comm1,)d(&consumed\);)218
2026 y(produce\(object+produced\045QLE)o(N\);)218 2083 y
(MPI_Window_out\(comm1\);)218 2139 y(produced++;)218 2195 y
(MPI_Window_out\(comm2\);)147 2252 y(})75 2365 y(elseif)i(\(myrank)g(==)h
(1\))f({)h(/*)f(consumer)g(code)h(*/)147 2534 y(object)f(=)g
(\(SOMEOBJECT*\)malloc\(sizeof\(SO)o(MEOBJEC)o(T\)\);)147 2591
y(MPI_RMA_init\(NULL,)e(0,)j(0,)f(MPI_COMM_WORLD;)f(comm1\);)147
2647 y(MPI_RMA_init\(NULL,)f(0,)j(0,)f(MPI_COMM_WORLD;)f(comm2\);)1967
46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 31 33
bop 75 -100 a Fo(4.8.)29 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)g(COHERENCE)h
(OPERA)l(TIONS)406 b Fu(31)147 49 y Fq(while)23 b(\(1\))g({)218
106 y(while)g(\(produced)g(<=)h(consumed\))290 162 y(MPI_Get\(&produced,)d
(1,)j(MPI_LONGINT,)e(0,)i(0,)815 219 y(1,)f(MPI_LONGINT,)g(0,)g(comm2\);)218
275 y(MPI_Get\(object,)f(1,)i(objecttype,)e(1,)i(produced\045QLEN,)910
332 y(1,)g(objecttype,)e(1,)i(comm1\);)218 388 y(consume\(object\);)218
444 y(})75 501 y(})166 595 y Fu(Example)16 b(3:)j(lo)q(cal)e(store,)d(follo)o
(w)o(ed)h(b)o(y)g(get,)g(in)h(a)f(global)g(gather)g(comm)o(unication)h
(pattern.)75 689 y Fq(...)75 745 y(double)23 b(a[1000],)g(b[1000];)75
801 y(int)g(index[1000];)75 858 y(...)75 914 y(MPI_RMA_init\(a,)f
(1000*sizeof\(double\),)f(sizeof\(double\),)934 971 y(MPI_WORLD,)i(comm\);)75
1027 y(....)75 1084 y(while)g(\(notdone\))g({)147 1140 y(update1\(a\);)147
1197 y(update2\(index\);)147 1253 y(MPI_Window_out\(comm\);)147
1310 y(MPI_Barrier\(comm\);)147 1366 y(for)g(\(i=0;)g(i<1000;)g(i++\))g({)218
1422 y(rank)g(=)h(index[i]/numproc;)218 1479 y(disp)f(=)h
(index[i]\045numproc;)218 1535 y(MPI_Get\(&b[i],)e(1,)i(MPI_REAL,)e(rank,)i
(disp,)f(1,)g(MPI_REAL,)g(comm\);)218 1592 y(})147 1648 y
(MPI_Window_in\(comm\);)147 1705 y(use\(b\);)75 1761 y(})75
1818 y(...)75 1911 y Fu(Assignmen)o(ts)14 b(to)f(arra)o(y)g
Fq(a)h Fu(are)f(not)h(allo)o(w)o(ed)g(after)f(the)h(call)h(to)e(Windo)o(w)p
1347 1911 14 2 v 17 w(out,)g(and)h(b)q(efore)h(the)f(call)h(to)75
1968 y(Windo)o(w)p 242 1968 V 17 w(in.)166 2024 y(Example)e(4:)19
b(pro)q(ducer-consumer)14 b(co)q(de.)20 b(The)13 b(pro)q(ducer)h(rep)q
(eatedly)g(pro)q(duces)g(a)e(new)i(v)o(ersion)75 2081 y(of)d(an)h(ob)s(ject.)
18 b(The)12 b(consumer)g(o)q(ccasionally)i(pic)o(ks)e(and)g(consumes)g(the)g
(latest)g(v)o(ersion)g(of)f(the)h(ob)s(ject.)75 2137 y(T)l(o)j(simplify)i
(the)e(example,)h(w)o(e)f(assume)g(a)g(homogeneous)g(en)o(vironmen)o(t.)75
2231 y Fq(main)23 b(\(\))h({)75 2288 y(struct)f({)147 2344
y(double)g(data[100];)147 2400 y(long)g(int)g(version;)147
2457 y(})g(object;)75 2513 y(double)g(localcopy[100];)75 2570
y(MPI_Comm)g(comm;)75 2683 y(MPI_RMA_init\(&object,)e(sizeof\(object\),)h(1,)
-32 46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 32 34
bop 75 -100 a Fu(32)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)767 49 y Fq(MPI_COMM_WORLD,)22 b(comm\))75 162 y(if)i(\(myrank)e(==)i
(0\))48 b({)23 b(/*)h(producer)f(*/)147 275 y(MPI_RMA_INIT\(NULL,)e(0,)j(0,)f
(MPI_WORLD,)g(comm\))147 388 y(while\(1\))f({)218 444 y(object.version++;)218
501 y(produce\(object.data\);)218 557 y(MPI_Fence\(comm,)g(1\);)218
614 y(MPI_Put\(&object.version,)f(1,)i(MPI_INT,)g(1,)481 670
y(100*sizeof\(double\),)e(1,)j(MPI_INT,)e(0,)i(comm\);)218
727 y(MPI_Fence\(comm,)e(1\);)218 783 y(MPI_Put\(&object.data,)f(100,)i
(MPI_FLOAT,)g(1,)h(0,)839 840 y(100,)f(MPI_FLOAT,)f(0,)i(comm\);)218
896 y(})75 953 y(elseif)f(\(myrank)g(==)h(1\))47 b({)24 b(/*)f(consumer)g(*/)
147 1065 y(MPI_RMA_INIT\(&object,)e(sizeof\(object\),)g(1,)j(MPI_WORLD,)e
(comm\);)147 1178 y(while\(1\))g({)218 1235 y(do)i({)290 1291
y(MPI_Window_in\(comm\);)290 1348 y(current)f(=)g(object.version;)290
1404 y(for)g(\(i=0;)g(i)h(<)g(100;)f(i++\))g(localcopy[i])g(=)g
(object.data[i];)290 1461 y(MPI_Window_in\(comm\);)290 1517
y(})218 1574 y(while)g(\(current)g(!=)h(object.version\);)218
1630 y(consume\(localcopy\);)218 1686 y(})75 1780 y Fu(The)18
b(pro)q(ducer)h(rep)q(eatedly)h(up)q(dates)e(in)i(the)e(consumer)g(memory)g
(the)g(v)o(ersion)h(n)o(um)o(b)q(er,)g(next)f(the)75 1837 y(ob)s(ject)e
(data.)24 b(Successiv)o(e)19 b(up)q(dates)e(are)f(serialized)j(b)o(y)e(calls)
h(to)e(fence.)26 b(The)17 b(consumer)g(c)o(hec)o(ks)g(the)75
1893 y(v)o(ersion)22 b(n)o(um)o(b)q(er,)h(next)e(bu\013ers)h(the)f(ob)s(ject)
g(data,)h(next)g(rec)o(hec)o(k)g(the)f(v)o(ersion)h(n)o(um)o(b)q(er.)39
b(If)22 b(the)75 1950 y(v)o(ersion)15 b(n)o(um)o(b)q(er)h(has)f(not)g(c)o
(hanged,)g(then)g(all)i(ob)s(ject)d(data)h(comes)g(from)f(the)i(same)e(v)o
(ersion.)166 2006 y(The)21 b(co)q(de)h(is)g(correct,)f(but)h(starv)m(ation)e
(is)i(p)q(ossible,)i(if)e(consumer)f(rep)q(eatedly)h(up)q(dates)g(the)75
2063 y(v)o(ersion)14 b(n)o(um)o(b)q(er)f(b)q(efore)h(the)f(consumer)h
(bu\013ered)f(a)g(new)h(v)o(ersion.)19 b(Suc)o(h)14 b(co)q(de)g(should)h(b)q
(e)f(used)f(only)75 2119 y(when)k(new)g(v)o(ersions)f(are)h(pro)q(duced)g(at)
f(m)o(uc)o(h)h(greater)e(in)o(terv)m(als)j(than)e(the)g(time)h(needed)h(to)e
(bu\013er)75 2175 y(a)f(v)o(ersion.)75 2296 y Fl(Registers)h(and)h(Com)o
(piler)c(Optimizations)44 b Fu(Another)15 b(coherence)h(problem)g(exists)g(b)
q(et)o(w)o(een)g(v)m(ariables)75 2352 y(k)o(ept)21 b(in)g(registers)g(and)g
(the)f(memory)h(v)m(alue)h(of)e(these)h(v)m(ariables.)38 b(An)21
b(RMA)g(call)g(ma)o(y)f(access)h(a)75 2408 y(v)m(ariable)c(in)g(memory)f
(\(or)f(cac)o(he\),)h(while)h(the)f(up-to-date)g(v)m(alue)i(of)d(this)i(v)m
(ariable)g(is)g(in)g(register.)22 b(A)75 2465 y(get)12 b(will)j(not)d(return)
g(the)h(latest)g(v)m(ariable)h(v)m(alue,)g(and)f(a)f(put)h(ma)o(y)f(b)q(e)h
(o)o(v)o(erwritten)f(when)h(the)g(register)75 2521 y(is)j(stored)e(bac)o(k)i
(in)g(memory)l(.)166 2578 y(MPI)k(implemen)o(tations)i(will)f(a)o(v)o(oid)f
(this)h(problem)g(for)e(standard)h(conforming)h(C)f(programs.)75
2634 y(High)13 b(qualit)o(y)h(MPI)f(F)l(ortran)e(implemen)o(tations)j(will)h
(a)o(v)o(oid)d(this)i(problem)f(for)g(standard)f(conforming)75
2691 y(F)l(ortran)i(programs)h(as)g(w)o(ell,)h(without)g(disabling)h
(compiler)g(optimizations.)22 b(Ho)o(w)o(ev)o(er,)14 b(MPI)h(imple-)1967
46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 33 35
bop 75 -100 a Fo(4.9.)34 b(FUR)l(THER)16 b(EXAMPLES)1153 b
Fu(33)75 49 y(men)o(tations)16 b(are)h(not)f(required)i(to)e(a)o(v)o(oid)h
(altogether)f(the)h(register)f(coherence)i(problem,)g(since)g(this)75
106 y(requires)d(compiler)h(supp)q(ort)e(that)g(is)h(not)f(curren)o(tly)h(a)o
(v)m(ailable)h(on)f(most)e(F)l(ortran)g(en)o(vironmen)o(ts.)20
b(In)75 162 y(order)14 b(to)f(a)o(v)o(oid)g(register)h(coherence)h(problems,)
f(users)g(should)h(restrict)e(their)h(use)h(of)e(RMA)h(windo)o(ws)75
219 y(to)h(v)m(ariables)h(stored)f(in)h Fj(COMMON)f Fu(blo)q(c)o(ks,)h(or)e
(to)h(v)m(ariables)i(that)d(w)o(ere)h(declared)i Fj(V)o(OLA)m(TILE)d
Fu(\(while)75 275 y Fj(V)o(OLA)m(TILE)e Fu(is)h(no)g(a)f(standard)h(F)l
(ortran)f(declaration,)h(it)h(is)f(supp)q(orted)g(b)o(y)g(man)o(y)g(F)l
(ortran)e(compilers.\))189 380 y Fi(R)n(ationale.)70 b Fu(Supp)q(ose)21
b(that)e(remote)h(accesses)g(are)g(executed)h(b)o(y)g(a)e(separate)h(soft)o
(w)o(are)e(or)189 436 y(hardw)o(are)d(agen)o(t,)g(concurren)o(tly)i(with)f
(main)h(pro)q(cess)f(execution.)24 b(A)16 b(remote)f(get)h(will)i(access)189
493 y(the)f(memory)g(or)f(cac)o(he)i(v)m(alue)g(of)f(a)g(v)m(ariable,)i
(while)g(the)e(latest)g(v)m(alue)i(of)d(this)i(v)m(ariable)h(ma)o(y)189
549 y(b)q(e)g(in)g(a)f(register.)29 b(Similarly)l(,)21 b(a)d(remote)f(put)i
(will)h(up)q(date)f(the)f(memory)g(or)g(cac)o(he)g(lo)q(cation)189
606 y(of)g(this)i(v)m(ariable,)h(while)g(the)e(v)m(ariable)h(is)g(held)g(in)g
(register.)32 b(The)19 b(memory)g(up)q(date)h(will)g(b)q(e)189
662 y(o)o(v)o(erwritten)14 b(when)i(the)f(register)g(spills)i(bac)o(k)e(to)g
(memory)l(.)189 737 y(The)g(same)h(problem)g(will)h(o)q(ccur)f(ev)o(en)g(if)g
(remote)f(accesses)g(are)g(executed)i(b)o(y)e(a)h(callbac)o(k)g(pro-)189
793 y(cedure)d(on)g(the)g(lo)q(cal)h(pro)q(cess.)20 b(The)13
b(v)m(ariable)h(that)e(is)i(accessed)f(remotely)g(do)q(es)g(not)g(app)q(ear)g
(in)189 850 y(the)k(argumen)o(t)f(list)h(of)g(the)f(callbac)o(k)i(pro)q
(cedure.)26 b(Therefore,)17 b(an)f(optimizing)j(compiler)f(ma)o(y)189
906 y(infer)f(that)e(the)i(pro)q(cedure)g(do)q(es)f(not)g(access)h(this)f(v)m
(ariable.)25 b(If)16 b(the)h(v)m(ariable)g(w)o(as)f(in)h(register)189
963 y(b)q(efore)f(the)f(pro)q(cedure)i(call,)g(the)e(compiler)i(ma)o(y)e(lea)
o(v)o(e)h(it)g(there.)21 b(The)16 b(pro)q(cedure)h(will)g(either)189
1019 y(lea)o(v)o(e)h(this)i(register)e(unp)q(erturb)q(ed,)j(or)d(will)i(sa)o
(v)o(e)e(it)h(on)g(the)f(stac)o(k)g(up)q(on)i(en)o(try)e(and)h(restore)189
1076 y(it)f(up)q(on)g(exit.)28 b(The)18 b(address)g(of)f(this)h(v)m(ariable)h
(is)g(not)e(kno)o(wn)h(to)f(the)g(pro)q(cedure.)29 b(In)18
b(either)189 1132 y(cases,)d(a)g(read)g(to)g(the)h(lo)q(cation)g(of)f(this)h
(v)m(ariable)h(in)g(memory)e(on)g(b)q(ehalf)i(of)e(a)g(get)g(will)i(fail)g
(to)189 1189 y(return)d(the)g(latest)h(v)m(alue)g(of)f(this)h(v)m(ariable,)h
(and)e(a)g(write)h(to)f(this)h(lo)q(cation)g(on)f(b)q(ehalf)i(of)e(a)g(put)
189 1245 y(will)i(b)q(e)g(o)o(v)o(erwritten)f(when)g(the)h(pro)q(cedure)g
(returns.)189 1320 y(A)i(lo)q(cal)h(access)f(to)f(suc)o(h)h(v)m(ariable)h(is)
g(alw)o(a)o(ys)e(separated)g(from)h(a)f(con\015icting)i(remote)f(access)189
1376 y(b)o(y)13 b(a)g(lo)q(cal)h(MPI)f(call,)i(suc)o(h)e(as)g
Fl(MPI)p 827 1376 14 2 v 16 w(WINDO)o(W)p 1039 1376 V 16 w(IN)g
Fu(or)f Fl(MPI)p 1248 1376 V 16 w(WINDO)o(W)p 1460 1376 V 16
w(OUT)p Fu(.)i(Therefore,)f(the)189 1433 y(coherence)j(problem)h(will)h(not)d
(o)q(ccur)h(in)h(a)e(standard)h(conforming)g(C)f(program.)21
b(If)16 b(a)g(v)m(ariable)189 1489 y(o)q(ccurs)22 b(in)g(a)g(comm)o
(unication)h(windo)o(w,)g(then)f(this)g(v)m(ariable)i(has)d(b)q(een)i
(dereferenced:)35 b(its)189 1546 y(address)15 b(has)h(b)q(een)h(used)f(to)f
(create)h(the)f(windo)o(w,)h(or)f(the)h(v)m(ariable)h(has)f(b)q(een)g
(heap-allo)q(cated)189 1602 y(at)21 b(a)h(kno)o(wn)f(lo)q(cation.)42
b(This)22 b(v)m(ariable)i(ma)o(y)d(b)q(e)i(accessed)f(b)o(y)g(a)g(pro)q
(cedure,)i(ev)o(en)e(if)h(the)189 1659 y(v)m(ariable)17 b(is)f(not)g(on)g
(the)f(argumen)o(t)h(list)g(of)g(the)g(pro)q(cedure.)22 b(If)16
b(the)g(MPI)g(library)h(is)f(compiled)189 1715 y(separately)l(,)c(and)f(at)f
(least)i(one)f(external)g(p)q(oin)o(ter)h(is)g(declared,)h(then)e(an)g
(optimizing)i(C)e(compiler)189 1772 y(m)o(ust)16 b(assume)g(that)g(MPI)h
(functions)g(ma)o(y)f(access)h(the)g(v)m(ariable.)25 b(Therefore,)17
b(the)f(v)m(ariable)i(is)189 1828 y(written)d(bac)o(k)g(to)f(memory)h(b)q
(efore)h(the)f(MPI)g(function)h(is)g(called.)189 1903 y(The)11
b(coherence)h(problem)g(ma)o(y)e(o)q(ccur)i(in)g(F)l(ortran)e(since)i
(explicit)h(aliasing)f(via)g(p)q(oin)o(ters)f(cannot)189 1959
y(o)q(ccur.)33 b(Ho)o(w)o(ev)o(er,)19 b(if)h(a)f(v)m(ariable)h(is)g(in)h(a)e
Fj(COMMON)g Fu(blo)q(c)o(k,)i(then)e(an)h(optimizing)h(compiler)189
2016 y(m)o(ust)12 b(assume)h(that)f(this)i(v)m(ariable)g(ma)o(y)e(b)q(e)i
(accessed)g(b)o(y)f(MPI)g(calls,)h(and)f(the)g(v)m(ariable)i(will)f(b)q(e)189
2072 y(written)f(bac)o(k)g(to)f(memory)h(b)q(efore)g(the)g(MPI)g(function)h
(is)g(called.)21 b(A)13 b(declaration)h(of)e Fj(V)o(OLA)m(TILE)189
2129 y Fu(will)22 b(force)e(the)h(v)m(ariable)h(to)e(b)q(e)i(written)e(bac)o
(k)h(to)f(memory)g(after)g(eac)o(h)h(up)q(date,)h(and)f(read)189
2185 y(from)e(memory)g(b)q(efore)h(eac)o(h)g(use,)g(so)g(that)f(the)h
(coherence)g(problem)h(disapp)q(ears.)34 b(\()p Fi(End)20 b(of)189
2242 y(r)n(ationale.)p Fu(\))1890 2301 y Fn(?)c Fm(\(June\))75
2433 y Fr(4.9)59 b(F)n(urther)20 b(Examples)75 2534 y Fp(Example)e(4.6)k
Fu(W)l(e)17 b(assume)g(that)e(t)o(w)o(o)h(datat)o(yp)q(e)g(transfer)g
(functions)h(are)f(built,)i(using)f(the)g(facili-)75 2591 y(ties)h(de\014ned)
h(for)f(datat)o(y)o(e)e(transfer)h(in)i(Chapter)e Fp(??)p Fu(.)27
b(The)18 b(function)h Fl(MPI)p 1416 2591 V 16 w(TYPE)p 1549
2591 V 17 w(SEND\(datat)o(yp)q(e,)75 2647 y(rank,)j(tag,)h(comm)m(\))18
b Fu(send)k(a)e(description)j(of)e(a)g(datat)o(yp)q(e)g(to)f(another)h(pro)q
(cess)g(and)h(the)f(function)75 2704 y Fl(MPI)p 160 2704 V
16 w(TYPE)p 293 2704 V 17 w(RECV\(t)o(yp)q(e,)16 b(rank,)g(tag,)f(comm)m(\))d
Fu(receiv)o(es)17 b(this)f(description)h(and)e(create)h(an)f(equiv)m(alen)o
(t)-32 46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272
y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554
y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40
836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40
1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345
y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40
1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853
y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40
2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361
y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40
2643 y(47)-40 2699 y(48)p eop
%%Page: 34 36
bop 75 -100 a Fu(34)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fu(datat)o(yp)q(e)f(on)h(the)g(destination)g(pro)q(cess)g
(\(the)g(same)f(datat)o(yp)q(e)g(that)g(w)o(ould)h(b)q(e)h(created)e(if)h
(the)g(desti-)75 106 y(nation)g(pro)q(cess)f(had)h(executed)h(the)e(same)g
(sequence)i(of)e(datat)o(yp)q(e)g(construction)h(calls)h(as)e(the)g(origin)75
162 y(pro)q(cess\).)166 219 y(In)19 b(the)h(follo)o(wing)f(co)q(de)h(pro)q
(cess)f(one)g(gets)g(a)f(structure)h(from)f(pro)q(cess)h(zero.)32
b(The)19 b(co)q(de)g(will)75 275 y(w)o(ork)14 b(correctly)i(in)g(a)f
(heterogeneous)g(en)o(vironmen)o(t.)75 381 y Fq(main\(\))23
b({)75 438 y(int)g(rank;)75 494 y(MPI_Comm)g(getcomm;)147 607
y(struct)g({)147 664 y(char)g(a;)147 720 y(int)g(b[10];)147
777 y(})g(str;)75 889 y(MPI_Comm_rank\(MPI_COMM_WOR)o(LD,)e(rank\);)75
946 y(MPI_RMA_Init\(&str,)g(sizeof\(str\),)i(1,)g(MPI_COMM_WORLD,)f
(getcomm\);)75 1059 y(/*)i(construct)e(local)h(datatype)g(for)h(str)f(*/)75
1115 y(int)g(disps[2];)75 1172 y(disps[0])g(=)g(0;)75 1228
y(disps[1])g(=)g(&str.b)h(-)f(&str;)75 1285 y(int)g(blens[2])g(=)h({1,)f
(10};)75 1341 y(MPI_Datatype)f(types[2])h(=)h({MPI_CHAR,)e(MPI_INT};)75
1398 y(MPI_Datatype)g(strtype;)75 1510 y(MPI_Type_struc\(2,)g(blens,)h
(disps,)g(types,)g(strtype\);)75 1567 y(MPI_Type_commit\(&strtype\);)75
1680 y(if\(rank)g(==)g(0\))h({)147 1736 y(int)f(i;)147 1849
y(/*)g(initialize)g(str)g(*/)147 1906 y(str.a)g(=)g('z';)147
1962 y(for)g(\(i=0;)g(i)h(<)g(10;)f(i++\))g(str.b[i])g(=)h(i;)147
2075 y(/*)f(send)g(datatype)g(for)h(str)f(*/)147 2131 y
(MPI_Type_send\(strtype,)d(1,)k(0,)g(getcomm\);)147 2188 y(})75
2244 y(elseif\(rank)e(==)i(1\))g({)147 2301 y(/*)f(get)h(remote)f(datatype)f
(for)i(str)f(*/)147 2357 y(MPI_Datatype)f(remstrtype;)147 2414
y(MPI_Recv\(&remstrtype,)f(0,)i(0,)h(getcomm\);)147 2527 y(/*)f(get)h(remote)
f(structure)f(*/)147 2583 y(MPI_Get\(str,)g(1,)h(srttype,)g(0,)h(0,)f(1,)h
(remstrtype,)e(1,)i(getcomm\);)147 2640 y(})75 2696 y(})1967
46 y Fm(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 35 37
bop 75 -100 a Fo(4.9.)29 b(FUR)l(THER)16 b(EXAMPLES)1158 b
Fu(35)75 49 y Fp(Example)18 b(4.7)k Fu(W)l(e)c(outline)h(b)q(elo)o(w)g(a)e
(design)i(for)e Fi(glob)n(al)h(p)n(ointers)p Fu(.)27 b(Global)18
b(p)q(oin)o(ters)h(are)e(sp)q(eci\014c)75 106 y(to)g(an)g(RMA)g(comm)o
(unicator,)g(and)h(consist)f(of)g(a)g(\(rank,)f(displacemen)o(t\))j(pair.)26
b(The)18 b(example)g(uses)75 162 y(a)f(windo)o(w,)h(with)g(displacemen)o(t)h
(one,)f(that)f(co)o(v)o(ers)g(the)h(heap)g(area)f(used)h(allo)q(cate)g
(globally)h(link)o(ed)75 219 y(structures.)75 381 y Fq(int)k(i;)75
551 y(/*)h(global)f(pointer)g(*/)75 607 y(typedef)g(struct)g({)123
664 y(int)g(rank;)123 720 y(int)g(disp;)75 777 y(})h(GPTR;)75
889 y(/*)g(local)f(to)g(global)g(pointer)g(conversion)g(\(error)g(checking)g
(omitted\))f(*/)75 946 y(void)h(local2global\(void)f(*ptr,)h(MPI_Comm)g
(comm,)g(GPTR)g(*gptr\))75 1002 y({)75 1059 y(int)g(flag,)h(disp_unit;)75
1115 y(MPI_Aint)f(addr,)g(base;)75 1172 y(void)g(*attr;)75
1285 y(MPI_Comm_rank\(comm,)e(&\(gptr->rank\)\);)75 1341 y
(MPI_Attr_get\(comm,)g(MPI_WINDOW_BASE,)h(&attr,)h(&flag\);)75
1398 y(base)g(=)h(\(MPI_Aint\)attr;)75 1454 y(MPI_Attr_get\(comm,)d
(MPI_WINDOW_DISP_UNIT,)g(&attr,)i(&flag\);)75 1510 y(disp_unit)g(=)g
(\(int\)attr;)75 1567 y(gptr->disp)f(=)i(\(\(MPI_Aint\)ptr)e(-)i
(base\)/disp_unit;)75 1623 y(return\(\);)75 1680 y(})75 1849
y(/*)g(MPI)f(Datatype)g(for)g(global)g(pointer)g(*/)75 1906
y(MPI_Datatype)f(GPTR_TYPE;)75 1962 y(MPI_Type_contiguous\(2,)f(MPI_INT,)i
(&GPTR_TYPE\);)75 2019 y(MPI_Type_commit\(&GPTR_TYPE)o(\);)75
2188 y(/*)h(nodes)f(of)g(local)h(list)f(*/)75 2244 y(typedef)g(lnode)g
(struct)g({)75 2301 y(double)g(content;)75 2357 y(struct)g(lnode)g(*next;)75
2414 y(})h(LNODE;)75 2583 y(/*)g(node)f(of)g(global)h(list;)f(the)g(list)g
(can)h(be)g(distributed)e(among)147 2640 y(several)g(processes)h(*/)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 36 38
bop 75 -100 a Fu(36)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fq(typedef)23 b(struct)g({)75 106 y(GPTR)g(next;)75
162 y(double)g(content;)75 219 y(})h(GNODE;)75 388 y(/*)g(MPI)f(Datatype)g
(for)g(global)g(node)h(*/)75 444 y(MPI_Datatype)e(GNODE_TYPE;)75
501 y(blens)h(=)h({1,)f(1};)75 557 y(types)g(=)h({GPRT_TYPE,)e(MPI_DOUBLE};)
75 614 y(MPI_Type_contiguous_struct)o(\(2,)f(blens,)i(types,)g
(&GNODE_TYPE\);)75 670 y(MPI_Type_commit\(&GNODE_TYP)o(E\);)75
840 y(/*)h(insertion)e(function.)47 b(Inserts)23 b(node)g(in)h(global)f(list)
g(*/)75 896 y(void)g(insert\()g(double)g(content,)g(GPTR)g(*gptr,)g(MPI_Comm)
g(comm\);)75 953 y({)75 1009 y(GNODE)g(*gnode;)75 1065 y(GPTR)g(me;)75
1178 y(gnode)g(=)h(\(GNODE*\)malloc\(sizeof\(GNO)o(DE\)\);)75
1235 y(gnode->content)e(=)i(content;)75 1291 y(/*)g(compute)e(global)i
(pointer)e(to)i(inserted)f(node)g(*/)75 1348 y(local2global\(gnode,)e(comm,)i
(&me\);)75 1461 y(/*)h(insert)f(*/)75 1517 y(MPI_Rmw\(me,)f
(&\(gnode->next\),)g(2,)i(MPI_INT,)266 1574 y(gptr->rank,)e(gptr->disp,)h
(MPI_SWAP,)f(0,)i(comm\);)75 1630 y(return\(\);)75 1686 y(})75
1856 y(/*)g(global)f(to)g(local)g(list)h(copying)f(function.)46
b(Creates)23 b(local)g(copy)h(of)147 1912 y(global,)e(distributed)h(list)g
(\(error)g(checking)g(and)g(locking)g(omitted\))147 1969 y(an)g(undefined)g
(rank)g(is)h(used)f(as)h(list)f(terminator)f(*/)75 2025 y(void)h
(global2local_list_copy\(GPTR)d(*ghead,)j(MPI_Comm)g(comm,)g(LNODE)h
(**head\);)75 2082 y({)75 2138 y(GNODE)f(gnode;)75 2195 y(LNODE)g(*tail;)75
2307 y(if)h(\(ghead->rank)e(==)h(MPI_UNDEFINED\))f({)123 2364
y(*head)h(=)h(NULL;)123 2420 y(return\(\);)123 2477 y(})75
2533 y(else)f({)147 2590 y(*head)g(=)g(\(LNODE*\)malloc\(sizeof\(LNODE\)\);)
147 2646 y(MPI_Get\(&gnode,)e(1,)j(GNODE_TYPE,)e(ghead->rank,)h(ghead->disp,)
337 2703 y(1,)h(GNODE_TYPE,)e(0,)i(comm\);)1967 46 y Fm(1)1967
103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967
385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959
667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893
y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119
y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345
y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570
y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796
y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022
y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248
y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474
y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699
y(48)p eop
%%Page: 37 39
bop 75 -100 a Fo(4.9.)29 b(FUR)l(THER)16 b(EXAMPLES)1158 b
Fu(37)147 49 y Fq(\(*head\)->content)21 b(=)j(gnode.content;)147
106 y(tail)f(=)h(*head;)147 162 y(while\(gnode->next.rank)c(!=)k
(MPI_UNDEFINED\))e({)218 219 y(tail->next)h(=)g
(\(LNODE*\)malloc\(sizeof\(LNODE\)\))o(;)218 275 y(tail)g(=)h(tail->next;)218
332 y(MPI_Get\(&gnode,)e(1,)i(GNODE_TYPE,)e(gnode->next.rank,)f
(gnode->next.disp,)409 388 y(1,)j(GNODE_TYPE,)e(0,)i(comm\);)218
444 y(tail->content)e(=)i(gnode.content;)218 501 y(})147 557
y(tail->next)e(=)i(NULL;)147 614 y(})75 670 y(return\(\);)75
727 y(})166 909 y Fh(Discussion:)61 b Fk(As)21 b(the)g(last)g(example)e(sho)o
(ws,)j(the)g(curren)o(t)g(design)f(allo)o(ws)e(for)h(global)f(structures)75
965 y(\(structures)g(using)d(global)f(p)q(oin)o(ters\))j(to)e(b)q(e)h
(accessed)i(lo)q(cally)c(or)i(remotely;)f(it)g(allo)o(ws)g(for)g(lo)q(cal)f
(structures)75 1022 y(\(structures)f(with)c(lo)q(cal)h(p)q(oin)o(ters\))g(to)
g(b)q(e)h(accessed)h(lo)q(cally)m(,)c(and)i(con)o(v)o(erted)h(to)f(global)f
(structures.)19 b(It)11 b(do)q(es)h(not)75 1078 y(allo)o(w)d(for)h(lo)q(cal)g
(structures)j(to)d(b)q(e)h(accessed)i(remotely)m(.)j(The)11
b(last)f(is)g(reasonably)h(easy)g(to)f(do)h(in)f(a)g(homogeneous)75
1134 y(en)o(vironmen)o(t.)24 b(An)17 b(MPI)f(extension)h(w)o(ould)f(b)q(e)h
(needed)h(to)e(supp)q(ort)h(the)g(last:)22 b(e.g.,)16 b(allo)o(wing)e(v)n
(alues)i(to)g(b)q(e)75 1191 y(sen)o(t)f(with)f(datat)o(yp)q(e)g
Fj(MPI)p 508 1191 13 2 v 14 w(POINTER)f Fk(and)h(receiv)o(ed)h(with)f(t)o(yp)
q(e)g Fj(MPI)p 1217 1191 V 15 w(INT)p Fk(,)f(on)h(a)g(windo)o(w)f(comm)o
(unicator;)e(if)75 1247 y(the)j(p)q(oin)o(ter)g(w)o(as)f(p)q(oin)o(ting)f(to)
h(a)h(lo)q(cation)e(inside)h(the)h(windo)o(w,)f(then)h(it)f(is)g(con)o(v)o
(erted,)h(at)f(the)h(receiving)g(side,)75 1304 y(in)o(to)f(a)h(displacemen)o
(t)f(within)g(this)h(windo)o(w.)75 1486 y Fp(Example)k(4.8)k
Fu(This)15 b(example)f(implemen)o(ts)h(a)e(master-sla)o(v)o(e)g(framew)o
(ork,)f(where)i(sla)o(v)o(es)g(pic)o(k)g(w)o(ork)75 1542 y(from)e(a)h(w)o
(ork)f(pile)j(and)e(execute)h(it.)19 b(A)14 b(form)e(of)h(guided)h(self-sc)o
(heduling)i(is)d(used)h(for)f(load)g(balancing,)75 1599 y(assuming)h(that)e
(task)h(execution)h(time)g(is)g(not)f(constan)o(t.)19 b(The)13
b(main)h(sla)o(v)o(e)f(routine)h(is)g(sho)o(wn)f(b)q(ello)o(w.)75
1687 y Fq(...)75 1744 y(void)23 b(function)g(slave\()147 1800
y(MPI_Comm)f(pile,)47 b(/*)24 b(communicator)e(for)i(window)f(containing)1053
1857 y(shared)g(work)h(pile)214 b(*/)147 1913 y(MPI_Comm)22
b(counter,)h(/*)h(communicator)e(for)h(window)g(containing)910
1970 y(shared)g(work)h(pile)f(counter)166 b(*/)147 2026 y(int)47
b(master,)71 b(/*)23 b(rank)g(of)h(master)739 b(*/)242 2083
y(numitems,)47 b(/*)23 b(number)g(of)h(work)f(items)g(in)h(pile)405
b(*/)242 2139 y(itemsize,)47 b(/*)23 b(size)g(of)h(work)f(item)668
b(*/)242 2195 y(minchunk,)47 b(/*)23 b(min)h(number)f(of)g(work)h(items)f
(picked)g(by)g(process)71 b(*/)242 2252 y(maxchunk,)47 b(/*)23
b(max)h(number)f(of)g(work)h(items)f(picked)g(by)g(process)71
b(*/)147 2308 y(float)23 b(bfactor,)g(/*)g(fudge)g(factor)g(for)h(load)f
(balancing)g(\(greater)g(than)552 2365 y(1,)h(and)f(larger)g(if)h(tasks)f
(exhibit)g(more)g(variance\))g(*/)147 2421 y(void)g(\(*task\))g(\()h(char)f
(*\))g(/*)h(pointer)f(to)h(work)f(function)309 b(*/)147 2478
y(\))75 2534 y({)75 2591 y(int)23 b(numproc,)g(mychunk,)g(diff,)g(mystart,)g
(i;)75 2647 y(char*)g(mypile;)-32 46 y Fm(1)-32 103 y(2)-32
159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32
441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40
724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40
1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232
y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40
1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740
y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40
2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248
y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40
2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 38 40
bop 75 -100 a Fu(38)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fq(MPI_Comm_size\(comm,)21 b(numproc\);)75
106 y(mypile)i(=)h(\(char)f(*\)malloc\(numitems*itemsize)o(/\(numpr)o
(oc*bfac)o(tor\)\);)75 219 y(while\(1\))g({)147 275 y(mychunk)f(=)i
(numitem/\(numproc*bfactor\);)147 332 y(if)f(\(mychunk)g(>)h(maxchunk\))e
(mychunk)h(=)h(maxchunk;)147 388 y(if)f(\(mychunk)g(<)h(minchunk\))e(mychunk)
h(=)h(minchunk;)147 444 y(diff)f(=)h(-mychunk;)147 501 y(MPI_Rmw\(&diff,)e
(&mystart,)g(1,)i(MPI_INT,)f(master,)f(0,)i(MPI_SUM\);)147
557 y(if)f(\(\(mystart)g(+)h(mychunk\))e(<)i(0\))g(break;)147
614 y(if)f(\(mystart)g(<)h(0\))f({)h(mychunk)f(+=)g(mystart;)g(mystart)g(=)h
(0;})147 670 y(MPI_Get\(mypile,)d(mychunk*itemsize,)h(MPI_CHAR,)h(master,)337
727 y(mystart*itemsize,)f(mychunk*itemsize,)g(MPI_CHAR,)g(pile\);)147
783 y(for)h(\(i=0;)g(i<mychunk;)g(i++\))g(task\(mypile+i*itemsize\);)147
840 y(})-117 865 y Fn(>)15 b Fm(\(June\))75 1001 y Fr(4.10)60
b(Optimization)18 b(Hints)75 1185 y Fh(Discussion:)e Fk(This)f(is)f(the)h
(text)g(of)f(a)g(prop)q(osal)g(b)o(y)g(Rolf)g Fu(The)i(application)i(can)d
(giv)o(e)h(some)g(information)75 1324 y(on)k(ho)o(w)f(it)h(will)i(use)e
(one-sided)h(comm)o(unication.)34 b(Based)21 b(on)e(these)h(hin)o(ts)h(the)e
(MPI)h(library)h(can)75 1381 y(p)q(ossibly)f(optimize)f(some)f(RMA)g
(functions.)30 b(F)l(or)17 b(this)i(purp)q(ose)g(the)f(application)i(can)e
(store)f(some)75 1437 y(attribute)e(v)m(alues)i(with)f Fl(MPI)p
592 1437 14 2 v 16 w(A)l(TTR)p 725 1437 V 17 w(PUT)f Fu(on)h(a)f(comm)o
(unicator)g(used)h(for)f(RMA)h(comm)o(unications.)75 1494 y(T)l(o)h(store)f
(an)o(y)g(com)o(binations)i(of)e(attribute)h(v)m(alues)h(for)e(eac)o(h)h(k)o
(ey)l(,)g(the)g(v)m(alues)h(can)f(b)q(e)g(added.)26 b(The)75
1550 y(giv)o(en)21 b(hin)o(ts)h(are)f(v)m(alid)h(un)o(til)g(they)f(are)g(mo)q
(di\014ed)h(b)o(y)f(a)g(later)g Fl(MPI)p 1319 1550 V 16 w(A)l(TTR)p
1452 1550 V 17 w(PUT)g Fu(or)f(canceled)j(b)o(y)75 1607 y Fl(MPI)p
160 1607 V 16 w(A)l(TTR)p 293 1607 V 17 w(DELETE)p Fu(.)c(The)h(k)o(ey)g(and)
g(attribute)g(v)m(alues)g(are)g(prede\014ned.)35 b(The)20 b(k)o(ey)g(v)m
(alues)h(itself)75 1663 y(remain)16 b(usable)g(after)e Fl(MPI)p
558 1663 V 16 w(A)l(TTR)p 691 1663 V 17 w(DELETE)p Fu(.)166
1719 y(The)k(follo)o(wing)h(attribute)f(v)m(alues)h(sp)q(ecify)g(whether)f
(the)h(application)g(will)h(use)e(the)g(one-sided)75 1776 y(commin)o(ucation)
i(in)g(the)g(same)f(manner)g(on)h(eac)o(h)f(no)q(de)h(of)f(the)h(comm)o
(unicator.)32 b(If)19 b(non)h(of)f(these)75 1832 y(v)m(alues)d(are)f(sp)q
(eci\014ed)j(then)d(eac)o(h)g(no)q(de)h(can)g(ha)o(v)o(e)e(its)i(o)o(wn)e
(pro\014le)i(\(this)g(is)f(the)h(default\).)75 1912 y(Key)g(v)m(alue:)21
b Fl(MPI)p 390 1912 V 16 w(1SIDED)p 560 1912 V 15 w(A)l(TTR)75
1989 y Fu(A)o(ttribute)15 b(v)m(alues:)434 b(Seman)o(tics)75
2063 y Fl(MPI)p 160 2063 V 16 w(1SIDED)p 330 2063 V 16 w(A)l(TTR)p
463 2063 V 17 w(GLOBAL)193 b Fu(On)24 b(eac)o(h)f(no)q(de)h(of)f(the)g(comm)o
(unicator)g(the)h(same)f(at-)843 2119 y(tributes)j(for)f(one-sided)i(comm)o
(unication)g(will)g(b)q(e)g(de-)843 2176 y(\014ned.)e(Storing)16
b(and)h(deleting)h(of)f(these)f(attributes)h(with)843 2232
y Fl(MPI)p 928 2232 V 16 w(A)l(TTR)p 1061 2232 V 17 w(PUT)28
b Fu(and)h Fl(MPI)p 1381 2232 V 16 w(A)l(TTR)p 1514 2232 V
17 w(DELETE)g Fu(can)g(b)q(e)843 2289 y(implemen)o(ted)17 b(as)d(a)h
(collectiv)o(e)i(routine.)75 2345 y Fl(MPI)p 160 2345 V 16
w(1SIDED)p 330 2345 V 16 w(A)l(TTR)p 463 2345 V 17 w(EXPOSE)195
b(MPI)p 928 2345 V 16 w(EXPOSE)25 b Fu(can)f(b)q(e)h(implemen)o(ted)g(as)f(a)
f(collectiv)o(e)843 2401 y(routine.)75 2458 y Fl(MPI)p 160
2458 V 16 w(1SIDED)p 330 2458 V 16 w(A)l(TTR)p 463 2458 V 17
w(DELIVER)182 b(MPI)p 928 2458 V 16 w(DELIVER)22 b Fu(can)g(b)q(e)g(implemen)
o(ted)i(as)e(a)f(collectiv)o(e)843 2514 y(routine)g(and)h(after)e
Fl(MPI)p 1298 2514 V 16 w(PUT)h Fu(the)g(used)h(bu\013er)f(is)h(not)843
2571 y(mo)q(di\014ed)16 b(un)o(til)h(the)e(next)g Fl(MPI)p
1401 2571 V 16 w(DELIVER)p Fu(.)166 2647 y(The)i(follo)o(wing)h(attribute)f
(v)m(alues)i(sp)q(ecify)f(the)g(RMA)f(routines)h(that)e(will)j(b)q(e)f(used)g
(only)g(after)75 2704 y(storing)c(this)h(v)m(alues.)21 b(If)15
b(non)f(of)g(these)h(v)m(alues)h(are)e(sp)q(eci\014ed)j(then)d(eac)o(h)h(RMA)
g(routine)g(can)f(b)q(e)h(used)1967 46 y Fm(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 39 41
bop 75 -100 a Fo(4.10.)29 b(OPTIMIZA)l(TION)16 b(HINTS)1124
b Fu(39)75 49 y(on)15 b(this)h(comm)o(unicator)f(\(this)g(is)h(the)f
(default\).)75 129 y(Key)h(v)m(alue:)21 b Fl(MPI)p 390 129
14 2 v 16 w(1SIDED)p 560 129 V 15 w(OP)75 205 y Fu(A)o(ttribute)15
b(v)m(alues:)434 b(Seman)o(tics)75 280 y Fl(MPI)p 160 280 V
16 w(1SIDED)p 330 280 V 16 w(OP)p 408 280 V 16 w(PUT)331 b(MPI)p
928 280 V 16 w(PUT)15 b Fu(ma)o(y)f(b)q(e)i(used)75 336 y Fl(MPI)p
160 336 V 16 w(1SIDED)p 330 336 V 16 w(OP)p 408 336 V 16 w(IPUT)318
b(MPI)p 928 336 V 16 w(IPUT)15 b Fu(ma)o(y)f(b)q(e)i(used)75
393 y Fl(MPI)p 160 393 V 16 w(1SIDED)p 330 393 V 16 w(OP)p
408 393 V 16 w(GET)334 b(MPI)p 928 393 V 16 w(GET)15 b Fu(ma)o(y)g(b)q(e)g
(used)75 449 y Fl(MPI)p 160 449 V 16 w(1SIDED)p 330 449 V 16
w(OP)p 408 449 V 16 w(IGET)321 b(MPI)p 928 449 V 16 w(IGET)15
b Fu(ma)o(y)f(b)q(e)i(used)75 505 y Fl(MPI)p 160 505 V 16 w(1SIDED)p
330 505 V 16 w(OP)p 408 505 V 16 w(A)o(CCUMULA)l(TE)124 b(MPI)p
928 505 V 16 w(A)o(CCUMULA)l(TE)15 b Fu(ma)o(y)g(b)q(e)h(used)75
562 y Fl(MPI)p 160 562 V 16 w(1SIDED)p 330 562 V 16 w(OP)p
408 562 V 16 w(IA)o(CCUMULA)l(TE)111 b(MPI)p 928 562 V 16 w(IA)o(CCUMULA)l
(TE)15 b Fu(ma)o(y)f(b)q(e)i(used)75 618 y Fl(MPI)p 160 618
V 16 w(1SIDED)p 330 618 V 16 w(OP)p 408 618 V 16 w(RMW)310
b(MPI)p 928 618 V 16 w(RMW)15 b Fu(ma)o(y)f(b)q(e)i(used)75
675 y Fl(MPI)p 160 675 V 16 w(1SIDED)p 330 675 V 16 w(OP)p
408 675 V 16 w(PUT)p 515 675 V 17 w(EA)o(CH)p 649 675 V 17
w(SAME)58 b Fu(F)l(or)13 b(eac)o(h)g(target)g(no)q(de)h Fl(MPI)p
1349 675 V 16 w(PUT)g Fu(is)g(alw)o(a)o(ys)f(called)i(with)843
731 y(the)j(same)g(bu\013er)g(address,)h(datat)o(yp)q(e)f(and)g(coun)o(t)h
(argu-)843 788 y(men)o(ts.)42 b(F)l(or)22 b(di\013eren)o(t)h(target)e(no)q
(des)i(di\013eren)o(t)g(bu\013er)843 844 y(argumen)o(ts)14
b(ma)o(y)h(b)q(e)g(used.)75 901 y Fl(MPI)p 160 901 V 16 w(1SIDED)p
330 901 V 16 w(OP)p 408 901 V 16 w(PUT)p 515 901 V 17 w(EVERY)p
675 901 V 18 w(SAME)31 b Fu(F)l(or)10 b(ev)o(ery)i(target)e(no)q(de)i
Fl(MPI)p 1358 901 V 16 w(PUT)f Fu(is)h(alw)o(a)o(ys)f(called)i(with)843
957 y(the)18 b(same)g(bu\013er)g(address,)h(datat)o(yp)q(e)f(and)g(coun)o(t)h
(argu-)843 1014 y(men)o(ts.)35 b(F)l(or)20 b(di\013eren)o(t)g(target)g(no)q
(des)h(the)f(same)g(bu\013er)843 1070 y(argumen)o(ts)14 b(ma)o(y)h(b)q(e)g
(used.)166 1146 y(The)20 b(follo)o(wing)g(attribute)f(v)m(alues)i(sp)q(ecify)
g(whic)o(h)f(RMA)g(sp)q(eci\014c)h(sync)o(hronization)g(metho)q(ds)75
1203 y(are)13 b(used.)20 b(If)14 b(non)g(of)g(these)g(v)m(alues)h(are)e(sp)q
(eci\014ed)j(then)e(eac)o(h)g(metho)q(d)g(can)g(b)q(e)g(used)g(on)g(this)g
(comm)o(u-)75 1259 y(nicator)h(\(this)g(is)h(the)f(default\).)75
1339 y(Key)h(v)m(alue:)21 b Fl(MPI)p 390 1339 V 16 w(1SIDED)p
560 1339 V 15 w(SYNC)75 1415 y Fu(A)o(ttribute)15 b(v)m(alues:)552
b(Seman)o(tics)75 1490 y Fl(MPI)p 160 1490 V 16 w(1SIDED)p
330 1490 V 16 w(SYNC)p 462 1490 V 17 w(COUNTER)273 b(MPI)p
1046 1490 V 16 w(GET)p 1150 1490 V 16 w(COUNTER)17 b Fu(ma)o(y)d(b)q(e)i
(used.)75 1546 y Fl(MPI)p 160 1546 V 16 w(1SIDED)p 330 1546
V 16 w(SYNC)p 462 1546 V 17 w(THRESHOLD)218 b(MPI)p 1046 1546
V 16 w(SET)p 1145 1546 V 16 w(COUNTER)p 1373 1546 V 18 w(THRESHOLD)43
b Fu(ma)o(y)f(b)q(e)961 1602 y(used.)75 1659 y Fl(MPI)p 160
1659 V 16 w(1SIDED)p 330 1659 V 16 w(SYNC)p 462 1659 V 17 w(THRESHOLD)p
746 1659 V 17 w(NOPOLL)24 b(MPI)p 1046 1659 V 16 w(SET)p 1145
1659 V 16 w(COUNTER)p 1373 1659 V 18 w(THRESHOLD)43 b Fu(ma)o(y)f(b)q(e)961
1715 y(used)15 b(and)g(a)g Fl(MPI)p 1275 1715 V 15 w(W)l(AIT)g
Fu(with)g(its)g(request)g(handle)961 1772 y(should)h(not)f(b)q(e)h(implemen)o
(ted)h(b)o(y)e(p)q(olling.)75 1828 y Fl(MPI)p 160 1828 V 16
w(1SIDED)p 330 1828 V 16 w(SYNC)p 462 1828 V 17 w(BARRIER)298
b(MPI)p 1046 1828 V 16 w(BARRIER)25 b Fu(with)f(an)g(RMA)h(comm)o(unicator)
961 1885 y(ma)o(y)14 b(b)q(e)i(used.)75 1941 y Fl(MPI)p 160
1941 V 16 w(1SIDED)p 330 1941 V 16 w(SYNC)p 462 1941 V 17 w(FENCE)344
b(MPI)p 1046 1941 V 16 w(FENCE)15 b Fu(ma)o(y)g(b)q(e)h(used.)166
2017 y(If)i(an)h(application)g(giv)o(e)g(some)f(of)g(these)g(hin)o(ts)h(then)
f(it)h(m)o(ust)e(not)h(use)h(other)e(RMA)i(metho)q(ds)75 2074
y(than)c(it)h(has)f(sp)q(eci\014ed.)75 2180 y Fp(Example)j(4.9)k
Fi(On)14 b(e)n(ach)g(no)n(de)g(of)h(an)f(RMA)g(c)n(ommunic)n(ator)h(the)f
(fol)r(lowing)g(applic)n(ation)h(runs.)k(F)m(or)75 2237 y(e)n(ach)c(tar)n
(get)g(always)g(the)g(same)g(bu\013er)h(ar)n(guments)e(ar)n(e)h(use)n(d)g
(with)h Fl(MPI)p 1341 2237 V 16 w(PUT)p Fi(.)f(The)f(synchr)n(onization)75
2293 y(after)f(the)g Fl(MPI)p 337 2293 V 16 w(PUT)g Fi(is)f(done)h(with)g(c)n
(ounters,)g(b)n(e)n(c)n(ause)f(then)h(the)g(wait)g(c)n(an)f(b)n(e)g(c)n
(ompute)n(d)i(lo)n(c)n(al)r(ly.)k(The)75 2350 y(applic)n(ation)12
b(waits)f(on)h(the)f(thr)n(eshold)h(but)g(do)n(es)f(not)h(set)f
Fl(MPI)p 1127 2350 V 16 w(1SIDED)p 1297 2350 V 16 w(SYNC)p
1429 2350 V 17 w(THRESHOLD)p 1713 2350 V 17 w(NOPOLL)75 2406
y Fi(b)n(e)n(c)n(ause)18 b(the)i(applic)n(ation)f(exp)n(e)n(cts)g(that)h
(ther)n(e)f(is)g(never)g(a)h(longer)e(wait)i(at)g Fl(MPI)p
1525 2406 V 16 w(W)l(AIT)f Fi(due)h(to)g(the)75 2462 y(overlapping)c(of)h
(the)f(c)n(ommunic)n(ation)g(and)g(the)h(se)n(c)n(ond)e(c)n(omputing)h(p)n
(erio)n(d.)170 2569 y Fq(int)24 b(attr_val;)170 2625 y(attr_val)f(=)h
(MPI_1SIDED_ATTR_GLOBAL)d(+)j(MPI_1SIDED_ATTR_DELIVER)o(;)170
2682 y(MPI_Attr_put\(comm,)e(MPI_1SIDED_ATTR,)g(&attr_val\);)-32
46 y Fm(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 40 42
bop 75 -100 a Fu(40)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)170 49 y Fq(attr_val)23 b(=)h(MPI_1SIDED_OP_PUT)d(+)j
(MPI_1SIDED_OP_PUT_EACH_SAME;)170 106 y(MPI_Attr_put\(comm,)e(MPI_1SIDED_OP,)
g(&attr_val\);)170 162 y(attr_val)h(=)h(MPI_1SIDED_SYNC_THRESHOLD;)170
219 y(MPI_Attr_put\(comm,)e(MPI_1SIDED_SYNC,)g(&attr_val\);)170
332 y(while\(...\))h({)218 388 y(/*)h(compute)f(some)g(data)g(for)h(other)f
(nodes)g(and)h(store)f(it)g(into)h(buff)f(*/)218 444 y(MPI_Put\(&buff_a,)f
(...,)h(rank_a,)g(...,)g(1,)h(comm\);)218 501 y(MPI_Put\(&buff_b,)e(...,)h
(rank_b,)g(...,)g(1,)h(comm\);)218 557 y(...)218 614 y(/*)g(second)f
(computing)f(period)h(\(buff_a)g(and)h(buff_b)f(are)g(not)h(modified\)*/)218
670 y(...)218 727 y(/*)g(wait)f(until)g(from)h(n)f(other)g(nodes)h(the)f
(data)g(is)h(there)f(*/)218 783 y(MPI_Set_counter_threshold\(c)o(omm,)e(n,)i
(request\);)218 840 y(MPI_Wait\(request,)f(status\);)218 896
y(MPI_Deliver\(comm\);)170 953 y(})166 1135 y Fh(Discussion:)34
b Fk(Are)14 b(these)i(hin)o(ts)e(useful)g(for)f(optimization?)75
1185 y(They)19 b(are)f(useful)g(for)g(sp)q(ecial)h(hardw)o(are)f(systems,)h
(e.g.)31 b(with)18 b Fj(MPI)p 1217 1185 13 2 v 14 w(1SIDED)p
1372 1185 V 14 w(OP)p 1444 1185 V 14 w(PUT)p 1542 1185 V 14
w(EA)o(CH)p 1664 1185 V 15 w(SAME)g Fk(sys-)75 1234 y(tems)e(can)h(use)h
(virtual)e(c)o(hannels,)h(with)g Fj(MPI)p 826 1234 V 14 w(1SIDED)p
981 1234 V 14 w(A)m(TTR)p 1103 1234 V 15 w(GLOBAL)f Fk(&)h
Fj(MPI)p 1417 1234 V 14 w(1SIDED)p 1572 1234 V 14 w(A)m(TTR)p
1694 1234 V 15 w(DELIVER)75 1284 y Fk(systems)c(without)g(reliable)f(in)o
(terconnect)j(can)e(implemen)o(t)d(fast)j(one-sided)g(comm)o(unication.)i
(The)e(absence)i(of)75 1334 y Fj(MPI)p 152 1334 V 14 w(1SIDED)p
307 1334 V 14 w(SYNC)p 428 1334 V 15 w(THRESHOLD)p 687 1334
V 16 w(NOPOLL)e Fk(allo)o(ws)g(that)g Fj(MPI)p 1166 1334 V
15 w(W)m(AIT)f Fk(on)i(a)f(threshold)h(request)i(handle)d(is)75
1384 y(implemen)o(ted)f(b)o(y)h(p)q(olling,)f(a)i(go)q(o)q(d)f(strategy)i(on)
e(shared)i(memory)d(systems.)166 1434 y(Are)j(v)o(endor)f(sp)q(eci\014c)h
(metho)q(ds)f(b)q(etter?)75 1484 y(Y)m(es,)g(but)f(only)g(if)g(they)h(need)h
(really)d(additional)g(functions.)18 b(If)13 b(those)i(hin)o(ts)e(are)h
(enough)g(then)g(for)f(p)q(ortabilit)o(y)75 1533 y(reasons)i(a)e
(standardized)i(in)o(terface)g(is)e(b)q(etter)j(for)d(the)i(users.)166
1583 y(And)f(when)g(additional)e(hin)o(ts)i(seem)g(to)g(b)q(e)g(useful?)75
1633 y(If)g(v)o(endors)h(w)o(an)o(t)f(to)h(add)f(additional)f(hin)o(ts)h(to)g
(the)h(list)f(ab)q(o)o(v)o(e)h(then)g(this)f(should)h(b)q(e)g(p)q(ossible)f
(un)o(til)g(the)h(last)75 1683 y(draft)f(of)f(MPI-2)h(is)f(written.)19
b(And)14 b(also)f(in)h(MPI-2.1.)166 1733 y(SYNC)p 283 1733
V 15 w(THRESHOLD)p 562 1733 V 15 w(POLL)g(or)g(SYNC)p 869 1733
V 16 w(THRESHOLD)p 1149 1733 V 15 w(NOPOLL?)75 1782 y(Is)j(it)f(b)q(etter)i
(to)e(de\014ne)i Fj(MPI)p 546 1782 V 14 w(1SIDED)p 701 1782
V 14 w(SYNC)p 822 1782 V 15 w(THRESHOLD)p 1081 1782 V 15 w(POLL)f
Fk(with)f(the)h(meaning)e(that)h(w)o(aiting)f(on)h(a)75 1832
y(threshold)d(ma)o(y)d(b)q(e)j(implem)o(en)o(ted)e(b)o(y)g(p)q(olling?)16
b(No,)c(b)q(ecause)i(without)e(an)o(y)f(hin)o(t)h(implemen)o(tation)d(b)o(y)j
(p)q(olling)75 1882 y(or)f(in)o(terrupt)h(should)f(b)q(e)h(p)q(ossible.)17
b(P)o(olling)9 b(w)o(as)i(addressed)i(b)o(y)e(a)g(hin)o(t)g(b)q(ecause)i(its)
e(e\016ciency)h(dep)q(ends)g(on)f(the)75 1932 y(a)o(v)o(erage)j(w)o(aiting)e
(p)q(erio)q(d.)19 b(\(And)14 b(therefore)h(there)g(is)f(no)g(hin)o(t)f(ab)q
(out)h(implemen)o(tation)c(with)k(an)g(in)o(terrupt.\))166
1982 y(When)g(should)g(the)g(hin)o(ts)g(b)q(e)h(giv)o(en?)75
2031 y(A)o(t)c(an)o(y)g(time)f(or)h(only)f(direct)i(after)f
Fj(MPI)p 723 2031 V 14 w(RMA)p 828 2031 V 15 w(INIT)g Fk(and)g(b)q(efore)h
(the)f(\014rst)h(RMA)f(is)g(done.)17 b(There)c(is)d(no)h(reason)75
2081 y(for)i(a)f(restriction.)19 b(Esp)q(ecially)13 b(if)f
Fj(MPI)p 696 2081 V 14 w(1SIDED)p 851 2081 V 14 w(OP)p 923
2081 V 14 w(PUT)p 1021 2081 V 15 w(EA)o(CH)p 1144 2081 V 14
w(SAME)g Fk(is)h(giv)o(en)g(there)h(should)f(b)q(e)g(a)g(c)o(hance)75
2131 y(to)h(recon\014gure)h(\(e.g.)j(for)c(load)e(balacing\))h(the)i(c)o
(hannels)f(b)o(y)g(deleting)g(this)g(hin)o(t)f(and)h(storing)g(it)f(again.)
166 2188 y(Whic)o(h)g(sync)o(hronization)h(metho)q(d)f(should)h(b)q(e)g(used)
h(in)f(the)g(example?)75 2244 y(Because)19 b Fj(MPI)p 314 2244
V 14 w(BARRIER)p Fk(,)14 b Fj(MPI)p 603 2244 V 15 w(IBARRIER)g
Fk(or)i Fj(MPI)p 946 2244 V 15 w(FENCE)g Fk(in)g(com)o(bination)e(with)i(a)h
(an)o(y)f(other)h(sync)o(hro-)75 2300 y(nization)11 b(causes)i(an)e
(additional)f(net)o(w)o(ork)i(tra\016c)g(and)f(b)q(ecause)j
Fj(MPI)p 1187 2300 V 14 w(BARRIER)c Fk(causes)j(additional)d(idle)h(times)75
2357 y(in)16 b(some)f(user)i(pro)q(cesses,)i(and)d(after)g(reading)g(the)h
(one-sided)f(c)o(hapter,)h(it)f(seems)g(that)h(the)f(use)h(of)f(coun)o(ters)
75 2413 y(and)i(w)o(aiting)f(on)h(the)g(threshold)h(is)f(the)h(fastest)g
(metho)q(d.)30 b(The)19 b(user)g(probably)f(o)o(v)o(erlo)q(oks)f(that)h(up)q
(dating)75 2470 y(the)f(coun)o(ters)h(is)e(also)g(an)g(additional)e(o)o(v)o
(erhead.)26 b(Nev)o(ertheless)19 b(coun)o(ters)f(seem)e(to)g(b)q(e)h(the)g
(fastest)g(but)g(not)75 2526 y(simplest)c(metho)q(d.)k(Therefore)e(it)f(is)g
(used)g(for)g(this)g(optimization)d(example.)-117 2617 y Fn(?)k
Fm(\(June\))1967 46 y(1)1967 103 y(2)1967 159 y(3)1967 215
y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967
498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724
y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959
1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959
1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959
1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959
1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959
1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959
2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959
2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959
2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 41 43
bop 75 381 a Fs(Bibliograph)m(y)952 2828 y Fu(41)p eop
%%Trailer
end
userdict /end-hook known{end-hook}if
%%EOF
--boundary=_0--

• Next message: Shane Hebert: "RE: we should wait for 1sided implementations"
• Previous message: Marc Snir: "Re: we should wait for 1sided implementations"