draft of chapter 4 for next meeting -- pls comment

Marc Snir (snir@watson.ibm.com)
Thu, 09 May 1996 02:08:09 -0400

Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Dennis Cottel: "Re: MPI handlers need preemption"
Previous message: (no name): "Re: MPI handlers need preemption"
Next in thread: Rolf Rabenseifner: "Re: draft of chapter 4 for next meeting -- pls comment"

%!PS-Adobe-2.0
%%Creator: dvips 5.47 (RS/6000 1.0) Copyright 1986-91 Radical Eye Software
%%Title: temp.dvi
%%Pages: 43 1
%%BoundingBox: 0 0 612 792
%%EndComments
%%BeginProcSet: tex.pro
/TeXDict 250 dict def TeXDict begin /N /def load def /B{bind def}N /S /exch
load def /X{S N}B /TR /translate load N /isls false N /vsize 10 N /@rigin{
isls{[0 1 -1 0 0 0]concat}if 72 Resolution div 72 VResolution div neg scale
Resolution VResolution vsize neg mul TR matrix currentmatrix dup dup 4 get
round 4 exch put dup dup 5 get round 5 exch put setmatrix}N /@letter{/vsize 10
N}B /@landscape{/isls true N /vsize -1 N}B /@a4{/vsize 10.6929133858 N}B /@a3{
/vsize 15.5531 N}B /@ledger{/vsize 16 N}B /@legal{/vsize 13 N}B /@manualfeed{
statusdict /manualfeed true put}B /@copies{/#copies X}B /FMat[1 0 0 -1 0 0]N
/FBB[0 0 0 0]N /nn 0 N /IE 0 N /ctr 0 N /df-tail{/nn 8 dict N nn begin
/FontType 3 N /FontMatrix fntrx N /FontBBox FBB N string /base X array
/BitMaps X /BuildChar{CharBuilder} N /Encoding IE N end dup{/foo setfont}2
array copy cvx N load 0 nn put /ctr 0 N[}B /df{/sf 1 N /fntrx FMat N df-tail}
B /dfs{div /sf X /fntrx[sf 0 0 sf neg 0 0]N df-tail}B /E{pop nn dup definefont
setfont}B /ch-width{ch-data dup length 5 sub get} B /ch-height{ch-data dup
length 4 sub get} B /ch-xoff{128 ch-data dup length 3 sub get sub} B /ch-yoff{
ch-data dup length 2 sub get 127 sub} B /ch-dx{ch-data dup length 1 sub get} B
/ch-image{ch-data dup type /stringtype ne{ctr get /ctr ctr 1 add N}if}B /id 0
N /rw 0 N /rc 0 N /gp 0 N /cp 0 N /G 0 N /sf 0 N /CharBuilder{save 3 1 roll S
dup /base get 2 index get S /BitMaps get S get /ch-data X pop /ctr 0 N ch-dx 0
ch-xoff ch-yoff ch-height sub ch-xoff ch-width add ch-yoff setcachedevice
ch-width ch-height true[1 0 0 -1 -.1 ch-xoff sub ch-yoff .1 add]{ch-image}
imagemask restore}B /D{/cc X dup type /stringtype ne{]}if nn /base get cc ctr
put nn /BitMaps get S ctr S sf 1 ne{dup dup length 1 sub dup 2 index S get sf
div put}if put /ctr ctr 1 add N}B /I{cc 1 add D}B /bop{userdict /bop-hook
known{bop-hook}if /SI save N @rigin 0 0 moveto}N /eop{clear SI restore
showpage userdict /eop-hook known{eop-hook}if}N /@start{userdict /start-hook
known{start-hook}if /VResolution X /Resolution X 1000 div /DVImag X /IE 256
array N 0 1 255{IE S 1 string dup 0 3 index put cvn put} for}N /p /show load N
/RMat[1 0 0 -1 0 0]N /BDot 260 string N /rulex 0 N /ruley 0 N /v{/ruley X
/rulex X V}B /V statusdict begin /product where{pop product dup length 7 ge{0
7 getinterval(Display)eq}{pop false}ifelse}{false}ifelse end{{gsave TR -.1 -.1
TR 1 1 scale rulex ruley false RMat{BDot}imagemask grestore}}{{gsave TR -.1
-.1 TR rulex ruley scale 1 1 false RMat{BDot}imagemask grestore}}ifelse B /a{
moveto}B /delta 0 N /tail{dup /delta X 0 rmoveto}B /M{S p delta add tail}B /b{
S p tail}B /c{-4 M}B /d{-3 M}B /e{-2 M}B /f{-1 M}B /g{0 M}B /h{1 M}B /i{2 M}B
/j{3 M}B /k{4 M}B /w{0 rmoveto}B /l{p -4 w}B /m{p -3 w}B /n{p -2 w}B /o{p -1 w
}B /q{p 1 w}B /r{p 2 w}B /s{p 3 w}B /t{p 4 w}B /x{0 S rmoveto}B /y{3 2 roll p
a}B /bos{/SS save N}B /eos{clear SS restore}B end
%%EndProcSet
TeXDict begin 1000 300 300 @start /Fa 1 59 df<70F8F8F87005057D840C>58
D E /Fb 1 1 df<FFFFC0FFFFC012027D871A>0 D E /Fc 4 62 df<03060C1C183830706060E0
E0E0E0E0E0E0E0E0E06060703038181C0C0603081E7E950D>40 D<C0603038181C0C0E06060707
070707070707070706060E0C1C18383060C0081E7E950D>I<1F003FC061E0C0E0E070E0700070
007000E000C001C0030006000C30183030707FE0FFE0FFE00C137E9211>50
D<7FFFE0FFFFF0000000000000000000000000000000000000FFFFF07FFFE0140A7E8B19>61
D E /Fd 6 113 df<0C000C001C003C00FC004C000C000C000C000C000C000C000C000C000C00
0C007FC07FC00A127E910F>49 D<7C00FF00E380418001801F807F80E180C180C180E3807FF03C
F00C0D7E8C0F>97 D<0E000E000E000000000000007E007E000600060006000600060006000600
06000600FFE0FFE00B137E920F>105 D<0380038003800000000000003F803F80018001800180
01800180018001800180018001800180018001804180E380FF007C0009197F920F>I<FEE0FFF0
3BB033303330333033303330333033303330FFFCFFFC0E0D808C0F>109
D<F780FFC038E0387030303030303030303030387038E03FC037003000300030003000FC00FC00
0C137F8C0F>112 D E /Fe 1 81 df<FFFFFFFFE0FFFFFFFFF07800001FF07C000003F03E0000
00F81F000000380F0000001C0F8000000C07C000000C03C000000603E000000001F000000000F8
0000000078000000007C000000003E000000001E000000001F000000000F8000000007C0000000
03C000000003C000000001800000000380000000030000000006000000000C0000000018000000
00380000000030000000006000000000C0000006018000000C038000000C030000001C06000000
380C000000F818000001F83800001FF03FFFFFFFF07FFFFFFFF0FFFFFFFFE0272A7E7F2C>80
D E /Ff 34 121 df<FFE0FFE0FFE00B037F8C10>45 D<F0F0F0F004047B830E>I<00C001C007
C0FFC0FFC0FBC003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0
03C003C003C003C003C003C003C003C003C0FFFFFFFFFFFF10227CA118>49
D<03F0000FFC001FFE003C1F003807807007C07003C0F003E0E001E06001E02001E00001E00001
E00001E00003C00003C0000780000780000F00001E00003C0000780000F00001E00001C0000380
000700000E00001C0000380000700000FFFFE0FFFFE0FFFFE013227EA118>I<01F00007FC001F
FF003E0F003807807003C02003C02003C00003C00003C00003C0000780000780000F00001E0003
FC0003F80003FE00000F000007800003C00003C00001E00001E00001E00001E00001E08001E0C0
03C0E003C07007803C0F801FFF000FFC0003F00013237EA118>I<001F00001F00002F00002F00
006F0000EF0000CF0001CF0001CF00038F00038F00078F00070F000F0F000E0F001E0F003C0F00
3C0F00780F00780F00F00F00FFFFF8FFFFF8FFFFF8000F00000F00000F00000F00000F00000F00
000F00000F00000F0015217FA018>I<FFFFE0FFFFE0FFFFE00001E00003C0000380000780000F
00000E00001E00001C00003C0000380000780000780000F00000F00000E00001E00001E00001E0
0003C00003C00003C00003C00003C0000780000780000780000780000780000780000780001321
7EA018>55 D<FFFC00FFFF80FFFFC0F007F0F001F0F00078F0003CF0003CF0003CF0003CF0003C
F00038F00078F000F0F003E0FFFFC0FFFF00FFFFC0F00FE0F001F8F00078F0003CF0001CF0001E
F0001EF0001EF0001EF0001EF0003CF0007CF000F8F003F0FFFFE0FFFFC0FFFE0017237BA220>
66 D<FFFFF0FFFFF0FFFFF0F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000FFFFE0FFFFE0FFFFE0F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000F00000F00000F00000F00000FFFFF8FFFFF8FFFFF815237BA21D>69
D<000FF000003FFE0000FFFF8001F81F8003E00780078001000F0000001E0000001E0000003C00
00003C000000780000007800000078000000F0000000F0000000F0000000F0000000F0000000F0
000000F000FFC0F000FFC0F000FFC0780003C0780003C0780003C03C0003C03C0003C01E0003C0
1E0003C00F0003C0078003C003E003C001F80FC000FFFFC0003FFF00000FF8001A257DA321>71
D<FC00007EFE0000FEFE0000FEFE0000FEF70001DEF70001DEF70001DEF78003DEF380039EF380
039EF3C0079EF3C0079EF1C0071EF1C0071EF1E00F1EF0E00E1EF0E00E1EF0F01E1EF0F01E1EF0
701C1EF0783C1EF0783C1EF038381EF03C781EF03C781EF01C701EF01C701EF01EF01EF00EE01E
F00EE01EF00FE01EF007C01EF007C01EF007C01EF000001E1F237BA22A>77
D<FC001EFE001EFE001EFE001EF7001EF7001EF3801EF3801EF3C01EF1C01EF1C01EF1E01EF0E0
1EF0F01EF0F01EF0781EF0781EF0381EF03C1EF03C1EF01E1EF01E1EF00E1EF00F1EF0071EF007
1EF0079EF0039EF0039EF001DEF001DEF000FEF000FEF000FEF0007E17237BA222>I<001FC000
007FF00001FFFC0003F07E0007C01F000F800F801F0007C01E0003C03C0001E03C0001E0780000
F0780000F0780000F070000070F0000078F0000078F0000078F0000078F0000078F0000078F000
0078F0000078F0000078780000F0780000F0780000F07C0001F03C0001E03E0003E01E0003C01F
0007C00F800F8007C01F0003F07E0001FFFC00007FF000001FC0001D257DA324>I<FFFE00FFFF
80FFFFC0F003E0F000F0F00078F00078F0003CF0003CF0003CF0003CF0003CF0003CF00078F000
78F000F0F003E0FFFFC0FFFF80FFFE00F00000F00000F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000F0000016237BA21F>I<FFFC00FFFF80FFFFC0F0
03E0F000F0F00078F00038F0003CF0003CF0003CF0003CF0003CF00038F00078F000F0F003E0FF
FFC0FFFF80FFFE00F01E00F00F00F00700F00780F00380F003C0F001E0F001E0F000F0F000F0F0
0078F00038F0003CF0001EF0001EF0000F18237BA21F>82 D<F0001F0001E0F0003F8001E0F000
3F8001E078003F8001C078003B8003C078007BC003C078007BC003C03C0073C007803C00F1C007
803C00F1E007803C00F1E007801E00E1E00F001E01E0E00F001E01E0F00F001E01E0F00F000F01
C0F01E000F03C0701E000F03C0781E000703C0781C00078380383C00078380383C000787803C3C
000387003C38000387001C380003C7001C780003CF001C780001CE001E700001CE000E700001CE
000E700001EE000EF00000EC000EE00000FC0007E00000FC0007E00000FC0007E00000780007C0
002B237FA22E>87 D<07E01FF83FFC3C1E301E200F000F000F000F01FF0FFF3FFF7F0F780FF00F
F00FF00FF80F7C3F7FFF3FFF1F8F10167E9517>97 D<F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000F00000F1F000F7FC00FFFE00FC1F00F80F00F007
80F00780F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F00780F00780F80F00FC3E
00FFFE00F7F800F3F00012237CA219>I<01FC0007FF000FFF801F07803C018078000078000070
0000F00000F00000F00000F00000F00000F000007800007800007800403C00C01F07C00FFFC007
FF8001FC0012167E9516>I<0003C00003C00003C00003C00003C00003C00003C00003C00003C0
0003C00003C00003C00003C003F3C00FFFC01FFFC03F0FC03C07C07803C07803C0F003C0F003C0
F003C0F003C0F003C0F003C0F003C0F003C07803C07803C03C07C03E0FC01FFFC00FFBC003E3C0
12237EA219>I<03F00007FC001FFE003E0F003C0780780380780380F001C0FFFFC0FFFFC0FFFF
C0F00000F00000F000007000007800007800003C00801F07800FFF8007FF0001F80012167E9516
>I<01F0F807FFF80FFFF81F1F801E0F003C07803C07803C07803C07803C07801E0F001F1F001F
FE001FFC0039F0003800003800003C00003FFE001FFFC01FFFE03FFFF07801F0F800F8F00078F0
0078F00078F000787800F03E03E01FFFC00FFF8001FC0015217F9518>103
D<F0F0F0F0000000000000000000F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004237D
A20B>105 D<F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F01F80F03F00F03E00F07C00F0F800F1F000F3E000F7C000FFC000FFC000FFE000
FFE000FDF000F8F800F07800F07C00F03E00F01E00F01F00F00F00F00F80F007C012237CA218>
107 D<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004
237DA20B>I<F0F807C0F3FE1FF0FFFF7FF8FE0FF078FC0FE07CF807C03CF807C03CF007803CF0
07803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803C
F007803CF007803CF007803CF007803C1E167C9529>I<F1F8F7FCFFFEFE1EF80FF80FF00FF00F
F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00F10167C9519>I<01FC0007
FF000FFF801F07C03C01E07800F07800F0700070F00078F00078F00078F00078F00078F0007878
00F07800F07C01F03E03E01F07C00FFF8007FF0001FC0015167F9518>I<F1F000F7FC00FFFE00
FC3F00F80F00F00780F00780F007C0F003C0F003C0F003C0F003C0F003C0F003C0F007C0F00780
F00F80F80F00FC3E00FFFE00F7F800F3F000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F0000012207C9519>I<F0E0F3E0F7E0FFE0FF00FC00F800F800F000F000F000F0
00F000F000F000F000F000F000F000F000F000F0000B167C9511>114 D<07F01FFE3FFF3C0F78
03780078007C003F003FF01FF80FFC01FE001F000F000F400FE00FF81EFFFE3FFC0FF010167F95
13>I<0F000F000F000F000F000F00FFF8FFF8FFF80F000F000F000F000F000F000F000F000F00
0F000F000F000F000F000F080F1C07FC07F803E00E1C7F9B12>I<F00FF00FF00FF00FF00FF00F
F00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF83F7FFF7FEF1F8F10167C9519>I<
7801F07C01E03E03C01E07C00F0780078F0007DE0003FC0001FC0000F80000700000F80001FC00
03DC00039E00078F000F07801E07801E03C03C01E07800F0F800F81516809516>120
D E /Fg 6 111 df<70F8F8F87005057C840D>58 D<70F8FCFC7C0C0C0C181838306040060E7C
840D>I<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E
00F00E00F00E18F01C30F03C30707C3078FC603FCFC00F078015147E9318>97
D<07803F803F000700070007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F
700F700F700FE01EE01EE01EE03CE03CE038607071E03FC01F0010207E9F14>I<00007C0000FE
00019E00039E00030C000700000700000700000700000E00000E00000E0000FFF001FFF0000E00
001C00001C00001C00001C00001C00003800003800003800003800003800007000007000007000
00700000700000E00000E00000E00000E00000C00001C00031C000798000F300007E00003C0000
17297E9F16>102 D<0F07C0001F8FE00031D8700061F0380061E0380061C03800C3C070000380
700003807000038070000700E0000700E0000700E0000701C1800E01C3000E0383000E0386000E
018E001C01FC000C00F00019147F931B>110 D E /Fh 3 64 df<C0000CE0001C700038380070
1C00E00E01C006018007038003870001CE0000FC0000780000780000FC0001CE00038700070380
0601800E01C01C00E0380070700038E0001CC0000C16187A9623>2 D<FFFFFFF0FFFFFFF00006
000000060000000600000006000000060000000600000006000000060000000600000006000000
060000000600000006000000060000000600000006000000060000000600000006000000060000
0006000000060000000600000006000000060000000600001C1C7D9B23>62
D<0006000000060000000600000006000000060000000600000006000000060000000600000006
000000060000000600000006000000060000000600000006000000060000000600000006000000
060000000600000006000000060000000600000006000000060000FFFFFFF0FFFFFFF01C1C7D9B
23>I E /Fi 17 119 df<387CFEFEFE7C3800000000387CFEFEFE7C3807127D910D>58
D<00038000000380000007C0000007C0000007C000000FE000000FE000001FF000001BF000001B
F0000031F8000031F8000061FC000060FC0000E0FE0000C07E0000C07E0001803F0001FFFF0003
FFFF8003001F8003001F8006000FC006000FC00E000FE00C0007E0FFC07FFEFFC07FFE1F1C7E9B
24>65 D<FFFFF800FFFFFF000FC01FC00FC007E00FC003F00FC001F80FC001F80FC000FC0FC000
FC0FC000FC0FC000FE0FC000FE0FC000FE0FC000FE0FC000FE0FC000FE0FC000FE0FC000FE0FC0
00FC0FC000FC0FC000FC0FC001F80FC001F80FC003F00FC007E00FC01FC0FFFFFF00FFFFF8001F
1C7E9B25>68 D<FFC00003FFFFE00007FF0FE00007F00DF0000DF00DF0000DF00DF0000DF00CF8
0019F00CF80019F00C7C0031F00C7C0031F00C3E0061F00C3E0061F00C1F00C1F00C1F00C1F00C
1F00C1F00C0F8181F00C0F8181F00C07C301F00C07C301F00C03E601F00C03E601F00C01FC01F0
0C01FC01F00C01FC01F00C00F801F00C00F801F0FFC0701FFFFFC0701FFF281C7E9B2D>77
D<0FF8001FFE003E1F803E07803E07C01C07C00007C003FFC01FFFC03F87C07E07C0FC07C0FC07
C0FC07C0FC0FC07E1FC03FFBF80FE1F815127F9117>97 D<03FC000FFE001F1F003E1F007C1F00
7C0E00FC0000FC0000FC0000FC0000FC0000FC00007C00007E01803E03801F07000FFE0003F800
11127E9115>99 D<01FC000FFF001F0F803E07C07C03C07C03E0FC03E0FFFFE0FFFFE0FC0000FC
0000FC00007C00007E00603E00C01F81C00FFF0001FC0013127F9116>101
D<03F0F00FFFF83E1F383C0F307C0F807C0F807C0F807C0F803C0F003E1F001FFC0033F0003000
007000003800003FFF003FFFC01FFFE03FFFF07801F8F00078F00078F000787800F03E03E01FFF
C003FE00151B7F9118>103 D<1E003F003F007F003F003F001E0000000000000000000000FF00
FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F00FFE0FFE00B1E7F9D0E
>105 D<FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F
001F001F001F001F001F001F001F001F001F00FFE0FFE00B1D7F9C0E>108
D<FF1FC0FF7FE01FE1F01F80F81F80F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00
F81F00F81F00F81F00F8FFE7FFFFE7FF18127F911B>110 D<01FC000FFF801F07C03E03E07C01
F07C01F0FC01F8FC01F8FC01F8FC01F8FC01F8FC01F87C01F07C01F03E03E01F07C00FFF8001FC
0015127F9118>I<FE3E00FE7F001ECF801F8F801F8F801F07001F00001F00001F00001F00001F
00001F00001F00001F00001F00001F0000FFF000FFF00011127F9114>114
D<1FD83FF87038E018E018F000FF807FE07FF01FF807FC007CC01CC01CE01CF038FFF0CFC00E12
7E9113>I<030003000300070007000F000F003F00FFFCFFFC1F001F001F001F001F001F001F00
1F001F001F0C1F0C1F0C1F0C0F9807F003E00E1A7F9913>I<FF07F8FF07F81F00F81F00F81F00
F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F01F80F03F80FFEFF03F8
FF18127F911B>I<FFC1FCFFC1FC1F00601F80E00F80C00FC0C007C18007C18003E30003E30001
F60001F60001FE0000FC0000FC0000780000780000300016127F9119>I
E /Fj 10 58 df<1F003F8060C04040C060C060C060C060C060C060C060C06060C060C03F801F
000B107F8F0F>48 D<18007800F80098001800180018001800180018001800180018001800FF80
FF8009107E8F0F>I<3F007F80F1C0F0E06060006000E000C00180030006001C0038606060FFC0
FFC00B107F8F0F>I<1F003F8071C071C031C001800F800F0001C000E060E0F0E0F0E0F1C07F80
1F000B107F8F0F>I<070007000F001F001B003B0033006300E300FFE0FFE00300030003001FE0
1FE00B107F8F0F>I<60807F807F007C00600060006F007F8070C060E000E060E0E0E0E1C07F80
3F000B107F8F0F>I<07801FC039C061C06000C000DF80FFC0E060C060C060C060606060C03F80
1F000B107F8F0F>I<60007FE07FE0C0C0C1800180030006000E000C000C001C001C001C001C00
1C0008000B117E900F>I<1F003F8061C060C060C079C03F801F803FC063E0C0E0C060C06060C0
3F801F000B107F8F0F>I<1F003F8060C0C0C0C060C060C060C0E07FE03F60006000C070C07180
7F003E000B107F8F0F>I E /Fk 33 123 df<00003F03E00000FF87F00001C78CF00003879CF0
0003031860000700380000070038000007003800000E003800000E007000000E007000000E0070
0000FFFFFF8001FFFFFF00001C00E000001C00E000001C00E000001C00E000003800E000003801
C000003801C000003801C000003801C000007001C0000070038000007003800000700380000070
038000007003800000E007000000E007000000E007000000E007000001C006000001C00E000001
C00E000031860C00007B8F180000F31E3800007E0FF000003C07C000002429829F1C>11
D<1C3C3C3C3C0C0C1818307060C080060E7D840E>44 D<70F8F8F0E005057B840E>46
D<000007000000070000000F0000000F0000001F0000003F0000003F0000006F0000006F000000
CF000000CF0000018F0000038F0000030F0000060F0000060F00000C0F80000C07800018078000
180780003FFF80007FFF800060078000C0078000C0078001800780018007800300078007000780
0F0007807FC07FF8FFC07FF81D207E9F22>65 D<01FFFFFE01FFFFFC001E003C001E001C001E00
1C003C000C003C000C003C000C003C001800780C1800780C1800780C0000781C0000F0380000FF
F80000FFF80000F0380001E0300001E0300001E0303001E0306003C0006003C0006003C000C003
C001C007800180078003800780070007801F00FFFFFF00FFFFFE001F1F7D9E1F>69
D<01FFFFFC01FFFFF8001E0078001E0038001E0038003C0018003C0018003C0018003C00300078
0C3000780C3000780C0000781C0000F0380000FFF80000FFF80000F0380001E0300001E0300001
E0300001E0300003C0000003C0000003C0000003C0000007800000078000000780000007800000
FFFC0000FFF800001E1F7D9E1E>I<01FF0000FFC001FF0000FFC0001F0001F800001F0001F800
001F0003780000370003F00000370006F0000037000CF0000037000CF00000670019E000006380
19E00000638031E00000638031E00000C38063C00000C380C3C00000C380C3C00000C38183C000
01838187800001838307800001838307800001838607800003038C0F000003038C0F0000030398
0F00000301D80F00000601F01E00000601F01E00000601E01E00000E01C01E0000FFE1C3FFE000
FFC183FFC0002A1F7D9E29>77 D<0001FC000007FF00001F07C0003801E000F000E001E0007003
C0007007800078070000380F0000381E0000381E0000383C0000383C0000387800007878000078
7800007878000078F00000F0F00000F0F00000E0F00001E0F00001C0F00003C070000780780007
0078000F0038001E003C003C001E00F0000F83E00007FF800001FC00001D217A9F23>79
D<01FFFF0001FFFFC0001E01E0001E00F0001E0078003C0078003C0078003C0078003C00780078
00F0007800F0007801E0007803C000F00F0000FFFE0000FFF80000F03C0001E01E0001E00E0001
E00F0001E00F0003C01E0003C01E0003C01E0003C01E0007803C0007803C1807803C1807803C30
FFF81E30FFF00FE0000007C01D207D9E21>82 D<0FFFFFF01FFFFFE01E0781E0180780E0380780
60300F0060300F0060600F0060600F00C0C01E00C0C01E00C0001E0000001E0000003C0000003C
0000003C0000003C00000078000000780000007800000078000000F0000000F0000000F0000000
F0000001E0000001E0000001E0000003E00000FFFF0000FFFF00001C1F789E21>84
D<00F18003FDC0078F800E07801C07803C07803C0700780700780700780700F00E00F00E00F00E
00F00E30F01C60F03C60707C6078FCC03FCFC00F078014147C9317>97 D<07803F803F00070007
0007000E000E000E000E001C001C001CF01FFC3F1E3E0E3C0F380F700F700F700F700FE01EE01E
E01EE03CE03CE038607071E03FC01F0010207B9F15>I<007E0001FF000383800F07801E07801C
07003C0200780000780000780000F00000F00000F00000F00000F00000700200700700381E001F
F80007E00011147C9315>I<0000780003F80003F00000700000700000700000E00000E00000E0
0000E00001C00001C000F1C003FDC0078F800E07801C07803C07803C0700780700780700780700
F00E00F00E00F00E00F00E30F01C60F03C60707C6078FCC03FCFC00F078015207C9F17>I<007C
0001FF000783000F01801E01803C01803C0300780E007FFC007FE000F00000F00000F00000F000
007000007002007807003C1E001FF80007E00011147C9315>I<0000F80001FC0003BC00033C00
0718000700000700000E00000E00000E00000E00000E0001FFE001FFE0001C00001C00001C0000
380000380000380000380000380000700000700000700000700000700000700000E00000E00000
E00000E00001C00001C00001C0000180003380007B8000F300007E00003C00001629829F0E>I<
003C6000FF7001E3E00381E00701E00F01E00F01C01E01C01E01C01E01C03C03803C03803C0380
3C03803C07003C0F001C1F001E3F000FFE0003CE00000E00000E00001C00001C00301C00783800
F0F0007FE0003F8000141D7E9315>I<01E0000FE0000FC00001C00001C00001C0000380000380
00038000038000070000070000073E00077F000EC3800F81C00F01C00E01C01E03801C03801C03
801C0380380700380700380700380E18700E30700E30701C60700C60E00FC060078015207D9F17
>I<006000F000F000E000000000000000000000000000000F001F80318031C063806380C38007
00070007000E000E000E001C301C601C6038C018C01F800F000C1F7D9E0E>I<03C01FC01F8003
800380038007000700070007000E000E000E000E001C001C001C001C0038003800380038007000
700070007180E300E300E300E6007E003C000A207C9F0C>108 D<1E07C0F8003F1FE1FC0033B8
730E0063E076070063C03C07006380380700C780780E000700700E000700700E000700700E000E
00E01C000E00E01C000E00E01C000E00E038601C01C038C01C01C038C01C01C071801C01C03180
3803803F001801801E0023147D9325>I<1E07C03F1FE033B87063E07063C038638038C7807007
00700700700700700E00E00E00E00E00E00E01C31C01C61C01C61C038C1C018C3801F81800F018
147D931A>I<007C0001FF000383800F01C01E01C01C01E03C01E07801E07801E07801E0F003C0
F003C0F003C0F00780F00700700F00701E003838001FF00007C00013147C9317>I<03C1E007E7
F8067E3C0C7C1C0C781E0C701E18E01E00E01E00E01E00E01E01C03C01C03C01C03C01C07803C0
7803C07003C0E003E3C0077F80071E000700000700000E00000E00000E00000E00001C0000FFC0
00FFC000171D809317>I<1E0F003F3F8033F1C063C1C063C3C06383C0C7838007000007000007
00000E00000E00000E00000E00001C00001C00001C00001C000038000018000012147D9313>
114 D<00FC03FE07070E0F0E0F0E0E1E000F800FF007F803FC003E001E701EF01CF01CE0386070
3FE01F8010147D9313>I<018001C0038003800380038007000700FFF0FFF00E000E000E000E00
1C001C001C001C0038003800380038307060706070C071803F001E000C1C7C9B0F>I<0F00601F
80703180E031C0E06380E06380E0C381C00701C00701C00701C00E03800E03800E03800E038C0E
07180E07180E0F180E1F3007F3F003E1E016147D9318>I<0F01C01F83C03183E031C1E06380E0
6380E0C380C00700C00700C00700C00E01800E01800E01800E03000E03000E06000E06000F0C00
07F80001E00013147D9315>I<0F0060E01F8071E03180E1F031C0E0F06380E0706380E070C381
C0600701C0600701C0600701C0600E0380C00E0380C00E0380C00E0381800E0381800E0781000E
078300070F860007F9FC0001F0F8001C147D931E>I<0787800FCFC018F8E03070E06071E06071
E0C0E1C000E00000E00000E00001C00001C00001C00071C060F380C0F380C0E38180C7C3007CFE
00387C0013147D9315>I<0F00601F80703180E031C0E06380E06380E0C381C00701C00701C007
01C00E03800E03800E03800E03800E07000E07000E0F000E1F0007FE0003EE00000E00000E0000
1C00781C0078380070700060E0003FC0001F0000141D7D9316>I<00E06003F0C007FFC00E1F80
0C0300000600000C0000180000300000600000C0000180000300000600C00C01801801803F8700
7FFE0060FC00C0780013147E9313>I E /Fl 44 121 df<01F1C003F1C007F1C00701C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C0FFF1C0FFF1C0FFF1C00E01C00E01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0121D809C16>
13 D<00E001C0038007000E000E001C001C003800380038007000700070007000E000E000E000
E000E000E000E000E000E000E000E000E00070007000700070003800380038001C001C000E000E
000700038001C000E00B2A7E9E10>40 D<E000700038001C000E000E0007000700038003800380
01C001C001C001C000E000E000E000E000E000E000E000E000E000E000E000E001C001C001C001
C0038003800380070007000E000E001C0038007000E0000B2A7E9E10>I<7878787838307060E0
05097D830C>44 D<010007003F00FF00FF00C70007000700070007000700070007000700070007
00070007000700070007000700070007000700FFF8FFF8FFF80D1C7C9B15>49
D<001C0000003E0000003E0000002E0000006700000067000000E7800000C7800000C3800001C3
C0000183C0000181C0000381E0000381E0000700F0000700F0000600F0000E0078000FFFF8000F
FFF8001FFFFC001C003C0018003C0038001E0038001E0070001F0070000F0070000F00E0000780
191D7F9C1C>65 D<FFF800FFFF00FFFF80F00FC0F003E0F000F0F000F0F000F0F000F0F000F0F0
01E0F007C0FFFF80FFFE00FFFF80F01FC0F003E0F000F0F00078F00078F00078F00078F00078F0
00F0F001F0F007E0FFFFC0FFFF80FFFC00151D7C9C1C>I<003FC000FFF003FFF007C0700F8000
1E00003E00003C0000780000780000780000F00000F00000F00000F00000F00000F00000F00000
F00000F000007800007800007800003C00003E00001E00000F800807C07803FFF800FFF0003F80
151F7D9D1B>I<FFFC00FFFF00FFFF80F00FE0F003F0F000F0F00078F00078F0003CF0003CF000
1CF0001EF0001EF0001EF0001EF0001EF0001EF0001EF0001EF0003CF0003CF0007CF00078F000
F0F001F0F007E0FFFFC0FFFF00FFFC00171D7C9C1E>I<FFFFC0FFFFC0FFFFC0F00000F00000F0
0000F00000F00000F00000F00000F00000F00000FFFF80FFFF80FFFF80F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F00000FFFFC0FFFFC0FFFFC0121D7C9C19>I<
FFFF80FFFF80FFFF80F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
FFFF00FFFF00FFFF00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F00000F00000111D7C9C18>I<003F8001FFF003FFF807C0F80F00181E00003E00003C00
00780000780000780000F00000F00000F00000F00000F00000F00000F007F8F007F8F007F87800
387800387800383C00383E00381E00380F003807C07803FFF801FFF0003F80151F7D9D1C>I<F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0FFFFF0FF
FFF0FFFFF0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F0
00F0F000F0F000F0141D7C9C1D>I<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0
F0F0F0F0F0041D7C9C0C>I<F000F000F000F000F000F000F000F000F000F000F000F000F000F0
00F000F000F000F000F000F000F000F000F000F000F000F000FFFEFFFEFFFE0F1D7C9C16>76
D<FC0007E0FC0007E0FC0007E0EE000DE0EE000DE0EE000DE0E70019E0E70019E0E70019E0E780
39E0E38031E0E3C071E0E3C071E0E1C061E0E1C061E0E1E0E1E0E1E0E1E0E0E0C1E0E0F1C1E0E0
7181E0E07181E0E07181E0E03B01E0E03B01E0E03B01E0E01E01E0E01E01E0E01E01E0E00001E0
1B1D7C9C24>I<FC0070FC0070FE0070EE0070EF0070E70070E70070E78070E38070E3C070E3C0
70E1E070E1E070E0E070E0F070E07070E07870E07870E03C70E03C70E01C70E01E70E00E70E00E
70E00F70E00770E007F0E003F0E003F0141D7C9C1D>I<003F000001FFE00003FFF00007C0F800
0F807C001E001E003E001F003C000F00780007807800078078000780F00003C0F00003C0F00003
C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C078000780780007807C000F803C00
0F003E001F001F003E000F807C0007C0F80003FFF00001FFE000003F00001A1F7E9D1F>I<FFFC
00FFFF00FFFF80F007C0F003E0F001E0F000F0F000F0F000F0F000F0F000F0F001E0F003E0F007
C0FFFF80FFFF00FFFC00F00000F00000F00000F00000F00000F00000F00000F00000F00000F000
00F00000F00000141D7C9C1B>I<FFF800FFFF00FFFF80F007C0F001E0F000E0F000F0F000F0F0
00F0F000F0F000E0F001E0F007C0FFFF80FFFF00FFF800F03C00F01C00F01E00F00F00F00F00F0
0780F00780F003C0F003C0F001E0F000F0F000F0F00078151D7C9C1B>82
D<03F8000FFE001FFF003E0F00780300780000F00000F00000F00000F00000F800007C00007F00
003FE0001FFC0007FE0001FF00001F800007800007C00003C00003C00003C00003C00003C0C007
80E00780FC1F007FFE001FFC0007F000121F7E9D17>I<FFFFFF80FFFFFF80FFFFFF80001E0000
001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E00
00001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E
0000001E0000001E0000001E0000001E0000001E0000191D7F9C1C>I<F00070F00070F00070F0
0070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F00070F0
0070F00070F00070F00070F00070F00070F000F07800E07801E03C03C01F07800FFF8007FE0001
F800141E7C9C1D>I<F0000380F00003807800070078000700780007003C000E003C000E003E00
0E001E001C001E001C000F0038000F0038000F003800078070000780700003C0600003C0E00003
C0E00001E0C00001E1C00001E1C00000F1800000F380000073000000730000007B0000003E0000
003E0000001C0000191D7F9C1C>I<F000F00070F000F80070F000F800707800B8006078019C00
E078019C00E078019C00E03C039C01C03C031E01C03C030E01C01E030E01801E070F03801E070F
03801E060F03800F060703000F0E0787000F0C078700070C038600070C038600079C038E000798
03CE00039801CC00039801CC00039801CC0001D001C80001D000D80001F000F80001E000F80000
E0007000241D7F9C27>I<78000E007C001E003C003C001E0038000F0070000F00F0000781E000
03C1C00001C3C00001E7800000F70000007E0000003E0000003C0000003C0000007E0000007700
0000E7800001E3800003C1C0000381E0000700F0000F00F8000E0078001C003C003C003E007800
1F0070000F00F0000F80191D7F9C1C>I<F80001E07C0001C03E0003801E0007801F0007000F80
0E0007801E0007C01C0003E03C0001E0380001F0700000F0F0000078E000007DC000003FC00000
1F8000001F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000
000F0000000F0000000F0000000F00001B1D809C1C>I<0FC03FF07FF87038401C001C001C00FC
0FFC3FFC781CE01CE01CE01CF07C7FFC7FDC3F1C0E127E9114>97 D<E000E000E000E000E000E0
00E000E000E000E000E000E3E0EFF0FFF8F87CF01CE01EE00EE00EE00EE00EE00EE00EE01CF01C
F87CFFF8EFF0E3C00F1D7D9C15>I<07E00FF81FFC3C1C70047000E000E000E000E000E000E000
700070043C1C1FFC0FF807E00E127E9112>I<07C01FE03FF078787018601CFFFCFFFCFFFCE000
E000E000700070043C1C3FFC1FF807E00E127E9112>101 D<03C3C00FFFC01FFFC01C3800381C
00381C00381C00381C00381C001C38001FF8001FF0003BC0003800003800001FFC001FFF003FFF
80700780E001C0E001C0E001C0F003C07C0F803FFF001FFE0007F800121B7F9115>103
D<F0F0F0F000000000000000707070707070707070707070707070707070041D7E9C0A>105
D<E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0E0031D7D9C0A>108
D<E3F03F00EFF8FF80FFFDFFC0F81F81E0F00F00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E
00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E0E00E00E01B127D9124
>I<E3E0EFF0FFF8F83CF01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01C0E
127D9115>I<03F0000FFC001FFE003C0F00780780700380E001C0E001C0E001C0E001C0E001C0
F003C07003807807803C0F001FFE000FFC0003F00012127F9115>I<078E1FEE3FFE7C3E781E70
0EE00EE00EE00EE00EE00EE00EF00E701E7C3E3FFE1FEE0F8E000E000E000E000E000E000E000E
000E0F1A7E9115>113 D<E380E780EF80FC00F800F000F000E000E000E000E000E000E000E000
E000E000E000E00009127D910E>I<1FC03FF07FF0F030E000E000F0007F003FC01FE000F00038
00388038F078FFF07FE01FC00D127F9110>I<1C001C001C001C001C001C00FFE0FFE0FFE01C00
1C001C001C001C001C001C001C001C001C001C001C201FF00FF007C00C187F970F>I<E01CE01C
E01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE01CE07CFFFC7FDC3F1C0E127D9115>I<
E007E0077006700E700E381C381C381C1C381C381C380E700E700660076007E003C003C010127F
9113>I<7003807807003C0E001C1C000E1C0007380003F00001E00001C00001E00003F0000738
000E18000E1C001C0E00380700700380F003C01212809113>120 D E /Fm
61 123 df<003F0F0000FFBF8003C3F3C00703E3C00703C1800E01C0000E01C0000E01C0000E01
C0000E01C0000E01C000FFFFFC00FFFFFC000E01C0000E01C0000E01C0000E01C0000E01C0000E
01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0000E01C0007F87FC00
7F87FC001A1D809C18>11 D<003F0000FF8003C1C00703C00703C00E01800E00000E00000E0000
0E00000E0000FFFFC0FFFFC00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C07F87F87F87F8151D809C17>I<003FC000FFC003C3C00703
C00701C00E01C00E01C00E01C00E01C00E01C00E01C0FFFFC0FFFFC00E01C00E01C00E01C00E01
C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C07FCFF87FCFF8151D
809C17>I<003F03F00000FFCFF80003C0FC1C000701F03C000701F03C000E00E018000E00E000
000E00E000000E00E000000E00E000000E00E00000FFFFFFFC00FFFFFFFC000E00E01C000E00E0
1C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00E01C000E00
E01C000E00E01C000E00E01C000E00E01C000E00E01C007FC7FCFF807FC7FCFF80211D809C23>
I<7070F8F8FCFCFCFC7C7C0C0C0C0C0C0C181818183030606040400E0D7F9C15>34
D<00F0000001F8000003980000030C0000070C0000070C0000070C0000070C0000070C00000718
0000073800000730000007607FE003C07FE003801F0003800C0007C00C000DC0180019E0180038
E0300070F06000F0706000F038C000F03CC000F01F8000F00F006078070060780F80C03C39E1C0
1FF0FF8007C03E001B1F7E9D20>38 D<00C00180030006000E000C001C00180038003000300070
00700060006000E000E000E000E000E000E000E000E000E000E000E000E0006000600070007000
30003000380018001C000C000E0006000300018000C00A2A7D9E10>40 D<C0006000300018001C
000C000E000600070003000300038003800180018001C001C001C001C001C001C001C001C001C0
01C001C001C0018001800380038003000300070006000E000C001C00180030006000C0000A2A7E
9E10>I<70F0F8F8781818183030706040050D7D840C>44 D<FFE0FFE0FFE00B0380890E>I<70F8
F8F87005057D840C>I<030007003F00FF00C70007000700070007000700070007000700070007
0007000700070007000700070007000700070007000700FFF8FFF80D1C7C9B15>49
D<07C01FF03878603C601EF01EF80FF80FF80F700F000F000E001E001C003C0078007000E001C0
038007000E030C03180330067FFEFFFEFFFE101C7E9B15>I<70F8F8F870000000000000000070
F8F8F87005127D910C>58 D<70F8F8F870000000000000000070F0F8F878181818303070604005
1A7D910C>I<0FE03FF8703C601CF01EF01EF01E001E003C007800E001C0018003800300030003
00030003000200000000000000000007000F800F800F8007000F1D7E9C14>63
D<00060000000F0000000F0000000F0000001F8000001F8000001F8000001F80000033C0000033
C0000033C0000061E0000061E0000061E00000C0F00000C0F00000C0F000018078000180780001
FFF80003FFFC0003003C0003003C0006001E0006001E0006001E001F001F00FFC0FFF0FFC0FFF0
1C1D7F9C1F>65 D<FFFFC0FFFFF00F00F80F003C0F001C0F001E0F001E0F001E0F001E0F001C0F
003C0F00780FFFF00FFFE00F00F80F003C0F001E0F001E0F000F0F000F0F000F0F000F0F000F0F
001E0F003E0F007CFFFFF8FFFFC0181C7E9B1D>I<001F808000FFE18003F0338007801B800F00
0F801E0007801C0003803C000380780003807800018070000180F0000180F0000000F0000000F0
000000F0000000F0000000F0000000F00000007000018078000180780001803C0001801C000300
1E0003000F00060007800C0003F0380000FFF000001F8000191E7E9C1E>I<FFFFC000FFFFF000
0F007C000F001E000F000F000F0007000F0003800F0003C00F0003C00F0001C00F0001E00F0001
E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001E00F0001C00F0001C00F0003C00F00
03800F0007800F000F000F001E000F007C00FFFFF000FFFFC0001B1C7E9B20>I<FFFFFCFFFFFC
0F007C0F001C0F000C0F000E0F00060F03060F03060F03060F03000F07000FFF000FFF000F0700
0F03000F03000F03030F03030F00030F00060F00060F00060F000E0F001E0F007CFFFFFCFFFFFC
181C7E9B1C>I<FFF3FFC0FFF3FFC00F003C000F003C000F003C000F003C000F003C000F003C00
0F003C000F003C000F003C000F003C000FFFFC000FFFFC000F003C000F003C000F003C000F003C
000F003C000F003C000F003C000F003C000F003C000F003C000F003C000F003C00FFF3FFC0FFF3
FFC01A1C7E9B1F>72 D<FFF0FFF00F000F000F000F000F000F000F000F000F000F000F000F000F
000F000F000F000F000F000F000F000F000F000F000F00FFF0FFF00C1C7F9B0F>I<FFF800FFF8
000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00
000F00000F00000F00180F00180F00180F00180F00380F00300F00700F00F00F01F0FFFFF0FFFF
F0151C7E9B1A>76 D<FF8000FF80FFC001FF800FC001F8000FC001F8000DE00378000DE0037800
0DE00378000CF00678000CF00678000CF00678000C780C78000C780C78000C780C78000C3C1878
000C3C1878000C3C1878000C1E3078000C1E3078000C1E3078000C0F6078000C0F6078000C0F60
78000C07C078000C07C078000C07C078001E03807800FFC387FF80FFC387FF80211C7E9B26>I<
FF00FFC0FF80FFC00F801E000FC00C000FC00C000DE00C000CF00C000CF00C000C780C000C780C
000C3C0C000C1E0C000C1E0C000C0F0C000C0F0C000C078C000C07CC000C03CC000C01EC000C01
EC000C00FC000C00FC000C007C000C003C000C003C001E001C00FFC01C00FFC00C001A1C7E9B1F
>I<003F800000FFE00003E0F80007803C000E000E001E000F003C00078038000380780003C078
0003C0700001C0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0F00001E0
780003C0780003C0780003C03C0007803C0007801E000F000F001E0007803C0003E0F80000FFE0
00003F80001B1E7E9C20>I<FFFF80FFFFE00F00F00F00380F003C0F001E0F001E0F001E0F001E
0F001E0F001E0F003C0F00380F00F00FFFE00FFF800F00000F00000F00000F00000F00000F0000
0F00000F00000F00000F0000FFF000FFF000171C7E9B1C>I<FFFF0000FFFFE0000F00F0000F00
38000F003C000F001E000F001E000F001E000F001E000F001E000F003C000F0038000F00F0000F
FFE0000FFFC0000F01E0000F00F0000F0078000F0078000F0078000F0078000F0078000F007800
0F0078000F0078300F003830FFF03C60FFF01FE0000007C01C1D7E9B1F>82
D<07E0801FF9803C1F80700780700380E00380E00180E00180E00180F00000F000007C00007FC0
003FF8001FFE0007FF0000FF80000F800003C00003C00001C0C001C0C001C0C001C0E00180E003
80F00300FC0E00CFFC0083F800121E7E9C17>I<7FFFFFC07FFFFFC0780F03C0700F01C0600F00
C0E00F00E0C00F0060C00F0060C00F0060C00F0060000F0000000F0000000F0000000F0000000F
0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F000000
0F0000000F000003FFFC0003FFFC001B1C7F9B1E>I<FFF0FFC0FFF0FFC00F001E000F000C000F
000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C00
0F000C000F000C000F000C000F000C000F000C000F000C000F000C000F000C0007001800078018
000380300001C0300000E0E000007FC000001F00001A1D7E9B1F>I<FFE01FF0FFE01FF00F0007
800F0003000F000300078006000780060007C00E0003C00C0003C00C0003E01C0001E0180001E0
180000F0300000F0300000F03000007860000078600000786000003CC000003CC000003CC00000
1F8000001F8000001F8000000F0000000F0000000F0000000600001C1D7F9B1F>I<FFE0FFE1FF
FFE0FFE1FF1F001E007C0F001E00300F003F00300F003F00300F803F0070078067806007806780
60078067806003C0E780C003C0C3C0C003C0C3C0C001E0C3C18001E181E18001E181E18001E181
E18000F381F30000F300F30000F300F300007B00F600007E007E00007E007E00007E007E00003C
003C00003C003C00003C003C00001C0038000018001800281D7F9B2B>I<FFF00FFCFFF00FFC07
8003C007C0038003C0030003E0060001F0060000F00C0000F81C0000781800007C3800003C3000
001E6000001F6000000FC000000FC0000007800000078000000780000007800000078000000780
0000078000000780000007800000078000007FF800007FF8001E1C809B1F>89
D<08081818303060606060C0C0C0C0C0C0F8F8FCFCFCFC7C7C38380E0D7B9C15>92
D<0FE0001FF8003C3C003C1E00180E00000E00001E0007FE001FFE003E0E00780E00F00E00F00E
60F00E60F01E60783E603FFFC01F878013127F9115>97 D<FC0000FC00001C00001C00001C0000
1C00001C00001C00001C00001C00001C00001C7E001DFF001F87801E01C01C01E01C00E01C00F0
1C00F01C00F01C00F01C00F01C00F01C00E01C01E01E01C01F078019FF00187C00141D7F9C17>
I<03F00FF81E3C383C78187000F000F000F000F000F000F000780078063C061E0C0FF803E00F12
7F9112>I<001F80001F8000038000038000038000038000038000038000038000038000038003
E3800FFB801E0F80380780780380700380F00380F00380F00380F00380F00380F0038070038078
03803807801E1F800FFBF007E3F0141D7F9C17>I<03E00FF01C38381C781E700EFFFEFFFEF000
F000F000F000700078063C061E0C0FF803E00F127F9112>I<007801FC039E071E0E0C0E000E00
0E000E000E000E00FFE0FFE00E000E000E000E000E000E000E000E000E000E000E000E000E000E
007FE07FE00F1D809C0D>I<00038007E7C00FFDC03C3DC0381C00781E00781E00781E00781E00
381C003C3C003FF00037E0007000007000003000003FFC001FFF003FFF80700780E001C0E001C0
E001C0E001C07003803C0F001FFE0007F800121C7F9215>I<FC0000FC00001C00001C00001C00
001C00001C00001C00001C00001C00001C00001C7C001DFF001F07001E03801E03801C03801C03
801C03801C03801C03801C03801C03801C03801C03801C03801C0380FF9FF0FF9FF0141D7F9C17
>I<18003C007C003C001800000000000000000000000000FC00FC001C001C001C001C001C001C
001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<FC0000FC00001C00001C00
001C00001C00001C00001C00001C00001C00001C00001C7FC01C7FC01C3E001C18001C30001C60
001CC0001DE0001FE0001E70001C78001C38001C3C001C1C001C0E001C0F00FF9FE0FF9FE0131D
7F9C16>107 D<FC00FC001C001C001C001C001C001C001C001C001C001C001C001C001C001C00
1C001C001C001C001C001C001C001C001C001C001C00FF80FF80091D7F9C0C>I<FC7E07E000FD
FF9FF8001F83B838001E01E01C001E01E01C001C01C01C001C01C01C001C01C01C001C01C01C00
1C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C001C01C01C00FF8FF8FF
80FF8FF8FF8021127F9124>I<FC7C00FDFF001F07001E03801E03801C03801C03801C03801C03
801C03801C03801C03801C03801C03801C03801C0380FF9FF0FF9FF014127F9117>I<03F0000F
FC001E1E00380700780780700380F003C0F003C0F003C0F003C0F003C0F003C070038078078038
07001E1E000FFC0003F00012127F9115>I<FC7E00FDFF001F87801E03C01C01E01C01E01C00F0
1C00F01C00F01C00F01C00F01C00F01C01E01C01E01E03C01F07801DFF001C7C001C00001C0000
1C00001C00001C00001C0000FF8000FF8000141A7F9117>I<03E1800FF9801E1F803C07807807
80780380F00380F00380F00380F00380F00380F003807803807807803C07801E1F800FFB8007E3
80000380000380000380000380000380000380001FF0001FF0141A7F9116>I<FDE0FFF01F781E
781E301C001C001C001C001C001C001C001C001C001C001C00FFC0FFC00D127F9110>I<1F903F
F07070E030E030E030F8007F803FE00FF000F8C038C038E038E038F070DFE08FC00D127F9110>
I<0C000C000C000C000C001C001C003C00FFE0FFE01C001C001C001C001C001C001C001C001C30
1C301C301C301C301E600FC007800C1A7F9910>I<FC1F80FC1F801C03801C03801C03801C0380
1C03801C03801C03801C03801C03801C03801C03801C03801C07800C0F800FFBF003E3F014127F
9117>I<FF0FE0FF0FE01C07801C03000E06000E06000E0600070C00070C00071C000398000398
0003F80001F00001F00000E00000E00000E00013127F9116>I<FF3FCFE0FF3FCFE01C0F07801C
0F03001C1F03000E1B06000E1B86000E1B86000E318E000731CC000731CC000760CC0003E0F800
03E0F80003E0F80001C0700001C0700001C070001B127F911E>I<7F8FF07F8FF00F0780070600
038E0001DC0001D80000F00000700000780000F80001DC00038E00030E000607000F0380FF8FF8
FF8FF81512809116>I<FF0FE0FF0FE01C07801C03000E06000E06000E0600070C00070C00071C
0003980003980003F80001F00001F00000E00000E00000E00000C00000C00000C000F18000F180
00C700007E00003C0000131A7F9116>I<7FFC7FFC7838707060F060E061C063C00380070C0F0C
0E0C1C1C3C1838187078FFF8FFF80E127F9112>I E /Fn 54 123 df<00F8F001F8F003F8F007
80F00700F00F00F00F00F00F00F00F00F00F00F00F00F00F00F0FFF8F0FFF8F0FFF8F00F00F00F
00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F00F
00F00F00F00F00F01420809F18>13 D<007000E001C00380078007000E001E001E003C003C003C
0078007800780078007000F000F000F000F000F000F000F000F000F000F000F000F00070007800
7800780078003C003C003C001E001E000E0007000780038001C000E000700C2E7EA112>40
D<E000700038001C001E000E0007000780078003C003C003C001E001E001E001E000E000F000F0
00F000F000F000F000F000F000F000F000F000F000E001E001E001E001E003C003C003C0078007
8007000E001E001C0038007000E0000C2E7DA112>I<0006000000060000000600000006000000
060000000600000006000000060000000600000006000000060000000600000006000000060000
00060000FFFFFFF0FFFFFFF0000600000006000000060000000600000006000000060000000600
0000060000000600000006000000060000000600000006000000060000000600001C207D9A23>
43 D<787878781830306060E0050A7D830D>I<FFC0FFC0FFC00A037F8B0F>I<03F00007F8001F
FE001E1E003C0F00380700780780780780700380F003C0F003C0F003C0F003C0F003C0F003C0F0
03C0F003C0F003C0F003C0F003C0F003C0F003C0F003C07807807807807807803807003C0F001E
1E001FFE000FFC0003F00012207E9E17>48 D<00C001C00FC0FFC0FFC0F3C003C003C003C003C0
03C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C003C0FFFEFF
FEFFFE0F1F7C9E17>I<FFFFFFF0FFFFFFF0000000000000000000000000000000000000000000
0000000000000000000000FFFFFFF0FFFFFFF01C0C7D9023>61 D<001F0000001F0000003F8000
003B8000003B8000007BC0000073C0000071C00000F1E00000E1E00000E0E00001E0F00001E0F0
0001C0F00003C0780003C078000380780007803C0007803C0007003C000FFFFE000FFFFE000FFF
FE001E000F001E000F003C000F803C0007803C000780780007C0780003C0780003C0F00003E01B
207F9F1E>65 D<FFF800FFFF00FFFF80F00FC0F003E0F001E0F000F0F000F0F000F0F000F0F000
F0F001E0F003C0F01F80FFFF00FFFF00FFFF80F007E0F001E0F000F0F00078F00078F00078F000
78F00078F00078F000F0F001F0F007E0FFFFC0FFFF80FFFC0015207B9F1E>I<001FC000FFF801
FFFC03E03C07800C0F00001E00003E00003C00007C0000780000780000780000F00000F00000F0
0000F00000F00000F00000F00000F000007800007800007800007C00003C00003E00001E00000F
000207800E03E03E01FFFC00FFF0001FC017227DA01D>I<FFFC00FFFF80FFFFC0F007E0F001F0
F000F8F00078F0003CF0003CF0001EF0001EF0000EF0000FF0000FF0000FF0000FF0000FF0000F
F0000FF0000FF0000FF0001EF0001EF0001EF0003CF0007CF000F8F001F0F007E0FFFFC0FFFF80
FFFC0018207B9F21>I<FFFFC0FFFFC0FFFFC0F00000F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000FFFF80FFFF80FFFF80F00000F00000F00000F00000F00000F000
00F00000F00000F00000F00000F00000F00000FFFFE0FFFFE0FFFFE013207B9F1B>I<FFFFC0FF
FFC0FFFFC0F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000FF
FF00FFFF00FFFF00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F0000012207B9F1A>I<001FE000FFF801FFFE03E03E07800E0F0000
1E00003E00003C00007C0000780000780000780000F00000F00000F00000F00000F00000F00000
F003FEF003FE7803FE78001E78001E7C001E3C001E3E001E1E001E0F001E07801E03E03E01FFFE
00FFF8001FC017227DA01E>I<F00078F00078F00078F00078F00078F00078F00078F00078F000
78F00078F00078F00078F00078F00078FFFFF8FFFFF8FFFFF8F00078F00078F00078F00078F000
78F00078F00078F00078F00078F00078F00078F00078F00078F00078F0007815207B9F20>I<F0
F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207C9F0D>I<F0
0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0
0000F00000F00000FFFF80FFFF80FFFF8011207B9F19>76 D<F80001F8FC0003F8FC0003F8F400
0378F6000778F6000778F6000778F7000F78F3000E78F3000E78F3801E78F3801E78F1801C78F1
C03C78F1C03C78F1C03C78F0C03878F0E07878F0E07878F0607078F070F078F070F078F030E078
F039E078F039E078F019C078F019C078F019C078F00F8078F00F8078F00F8078F00000781D207B
9F28>I<FC0078FE0078FE0078F60078F70078F70078F38078F38078F38078F3C078F1C078F1E0
78F1E078F0E078F0F078F07078F07078F07878F03878F03C78F03C78F01C78F01E78F00E78F00E
78F00E78F00778F00778F00378F003F8F003F8F001F815207B9F20>I<003F000000FFC00003FF
F00007E1F8000F807C001F003E001E001E003C000F003C000F00780007807800078078000780F0
0003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F00003C0F80007C0
7800078078000780780007803C000F003C000F001E001E001F003E000F807C0007E1F80003FFF0
0000FFC000003F00001A227DA021>I<FFF800FFFF00FFFF80F00FC0F003E0F001E0F000F0F000
F0F000F0F000F0F000F0F000F0F000F0F001E0F003E0F00FC0FFFF80FFFF00FFF800F00000F000
00F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F0000014207B9F1D
>I<FFF800FFFF00FFFF80F007C0F003E0F001E0F000F0F000F0F000F0F000F0F000F0F001E0F0
03E0F007C0FFFF80FFFF00FFF800F03C00F01E00F01E00F00F00F00F00F00780F00780F003C0F0
01C0F001E0F000F0F000F0F00078F00078F0003C16207B9F1D>82 D<01FC0007FF800FFFC01F03
C03C00C03C00007800007800007800007800007800007C00003C00003F00001FE0000FFC0007FE
0001FF00003F800007C00003C00003E00001E00001E00001E00001E00001E00001C0C003C0F007
C0FC0F807FFF001FFE0003F80013227EA019>I<FFFFFFC0FFFFFFC0FFFFFFC0001E0000001E00
00001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E
0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E000000
1E0000001E0000001E0000001E0000001E0000001E0000001E0000001E00001A207E9F1F>I<F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F0
00F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F0F000F07801E078
01E03C03C03C03C01F0F800FFF0007FE0001F80014217B9F1F>I<F00001E0F00001E0780003C0
780003C0780003C03C0007803C0007803C0007801E000F001E000F001F000F000F001E000F001E
0007801C0007803C0007803C0003C0380003C0780003C0780001E0700001E0F00001E0F00000F0
E00000F1E00000F1E0000071C000007BC000003B8000003B8000003F8000001F0000001F00001B
207F9F1E>I<F0007C000FF0007E000FF0007E000F78006E000E7800EE001E7800E7001E7800E7
001E3C00E7003C3C01E7803C3C01C7803C3C01C3803C1E01C380781E03C3C0781E0383C0780E03
81C0700F0381C0F00F0781E0F00F0701E0F0070700E0E0078700E1E0078F00F1E0078E00F1E003
8E0071C0038E0071C003CE0073C001DC007B8001DC003B8001DC003B8001DC003B8000F8003F00
00F8001F0000F8001F0028207F9F2B>I<780007807C000F003E001F001E001E000F003C000F80
7C000780780003C0F00003E1F00001F1E00000F3C000007FC000007F8000003F0000001F000000
1E0000003F0000007F8000007FC00000F3C00001F1E00001E0F00003C0F80007C0780007803C00
0F003E001F001E001E000F003C000F807C0007C0780003C0F00003E01B207F9F1E>I<F80000F8
7C0001F03C0001E03E0003E01F0003C00F0007800F800F8007C00F0003C01F0003E01E0001F03C
0000F07C00007878000078F000003CF000001DE000001FE000000FC00000078000000780000007
800000078000000780000007800000078000000780000007800000078000000780000007800000
078000000780001D20809F1E>I<07E03FF87FFC701E401F000F000F000F003F07FF1FFF7E0FF8
0FF00FF00FF00FF83F7FFF3FEF1F8F10147E9316>97 D<F00000F00000F00000F00000F00000F0
0000F00000F00000F00000F00000F00000F00000F1F000F7FC00FFFE00FC3E00F80F00F00F00F0
0780F00780F00780F00780F00780F00780F00780F00F00F00F00F81F00FC3E00FFFC00F7F800F1
E00011207D9F17>I<03F00FFC1FFE3E0E3C0278007800F000F000F000F000F000F00078007800
3C013E0F1FFF0FFE03F010147E9314>I<00078000078000078000078000078000078000078000
078000078000078000078000078007C7800FF7801FFF803E1F807C0780780780F80780F00780F0
0780F00780F00780F00780F00780F00780780780780F803E1F801FFF800FF78007C78011207E9F
17>I<03F0000FFC001FFE003E1F003C0700780700700380FFFF80FFFF80FFFF80F00000F00000
F000007000007800003C01003E07001FFF0007FE0001F80011147F9314>I<007E01FE03FE0780
07000F000F000F000F000F000F000F00FFF0FFF0FFF00F000F000F000F000F000F000F000F000F
000F000F000F000F000F000F000F000F000F20809F0E>I<03E0F00FFFF01FFFF03E3E003C1E00
780F00780F00780F00780F00780F003C1E003E3E001FFC003FF80033E0003000003800003FFE00
3FFF801FFFC03FFFE07803F0F000F0F000F0F000F0F801F07E07E03FFFC00FFF0003FC00141E7F
9317>I<F0F0F0F00000000000000000F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207D
9F0B>105 D<F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000F00000
F00000F01F00F01E00F03C00F07800F0F000F1E000F3C000F78000FFC000FFC000FFE000F9F000
F8F000F0F800F07C00F07C00F03E00F01E00F01F00F00F8011207D9F16>107
D<F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F0F004207D9F0B>
I<F0FC07E0F3FE1FF0F7FF3FF8FE0FF07CF807C03CF807C03CF007803CF007803CF007803CF007
803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF007803CF0
07803C1E147D9327>I<F1F8F3FCF7FEFC1FF80FF80FF00FF00FF00FF00FF00FF00FF00FF00FF0
0FF00FF00FF00FF00FF00F10147D9317>I<01F80007FE001FFF803F0FC03C03C07801E07801E0
F000F0F000F0F000F0F000F0F000F0F000F07801E07801E03C03C03F0FC01FFF8007FE0001F800
14147F9317>I<F1F000F7FC00FFFE00FC3E00F81F00F00F00F00F80F00780F00780F00780F007
80F00780F00780F00F00F00F00F81F00FC3E00FFFC00F7F800F1E000F00000F00000F00000F000
00F00000F00000F00000F00000F00000111D7D9317>I<03C7800FF7801FFF803E1F807C0F8078
0780780780F00780F00780F00780F00780F00780F00780F807807807807C0F803E1F801FFF800F
F78007C780000780000780000780000780000780000780000780000780000780111D7E9317>I<
F0E0F3E0F7E0FF00FC00FC00F800F800F000F000F000F000F000F000F000F000F000F000F000F0
000B147D9310>I<07F01FFC3FFC780C7800780078007C003FC01FF00FF803F8007C003C003CC0
3CF07CFFF87FF00FC00E147F9311>I<1E001E001E001E001E001E00FFF0FFF0FFF01E001E001E
001E001E001E001E001E001E001E001E001E001E001E201FF00FF007C00C1A7F9910>I<F00FF0
0FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF00FF01FF03FFFFF7FEF3F0F1014
7D9317>I<F003C0F003C07803807807807807803C0F003C0F003C0F001E0E001E1E001E1E000F
1C000F3C000F3C0007380007380007B80003F00003F00001E00012147F9315>I<F01F00F0F01F
80F0F01F80F0781B81E0783B81E0783BC1E07839C1E03C31C3C03C71C3C03C71E3C01C70E3801E
60E7801E60E7801EE0E7800EE077000EC077000EC0770007C07E0007C03E0007803E001C147F93
1F>I<F003C0F003C07807807807807C07803C0F003C0F001E0F001E1E000E1E000F1C000F1C00
073C0007380003B80003B80003B00001F00001F00000E00000E00001C00001C00003C000038000
0780007F00007E00007C0000121D7F9315>121 D<7FFF7FFF7FFF003E003C007800F800F001E0
03E007C007800F001F001E003C007C00FFFFFFFFFFFF10147F9314>I E
/Fo 32 91 df<3078F8787005057C840D>46 D<007E0001FF0003C3800701C00E01C00E01E01C
00E01C01E03C01E03C01E07801E07801E07801E07801E07801E0F003C0F003C0F003C0F003C0F0
03C0F00380F00780E00780E00700E00F00E00E00701E00701C003878003FF0000FC000131F7C9D
17>48 D<000C001C00FC0FF80F38003800380038003800700070007000700070007000E000E000
E000E000E000E001C001C001C001C001C001C003C07FFEFFFE0F1E7C9D17>I<003F0000FFC001
C1E00300F00600780600780F007C0F807C0F807C0F00780600780000F80000F00001F00001E000
03C0000780000F00001C0000380000700000C0000180000300600600601C00C03800C07FFFC0FF
FF80FFFF80161E7E9D17>I<007F0001FFC00383E00701E00700F00F00F00F01F00F01F00001E0
0001E00003C0000780000F0000FE0000F800000E000007000007800003C00003C00003C03007C0
7807C0F807C0F807C0F00F80C00F00601E00383C001FF80007E000141F7D9D17>I<0001C00003
C00003C0000780000F80001B80003B8000738000638000C7000187000307000707000E07000C07
00180E00300E00600E00E00E00FFFFF0FFFFF0001C00001C00001C00001C00001C00001C00003C
0003FFC003FFC0141E7D9D17>I<03007003FFF003FFE003FF8003FE0006000006000006000006
00000600000600000C7C000DFF000F87800E03800C01C00C01C00001E00001E00001E00001E070
03C0F003C0F003C0F00380C00780E00700600E00383C001FF0000FC000141F7D9D17>I<000F80
007FC000F0E001C0E00381E00701E00E00C01E00001C00003C000038000079FC007BFE007E0700
7C0380F80380F803C0F003C0F003C0F003C0F003C0F00780E00780E00780E00700E00F00700E00
701C003838001FF00007C000131F7C9D17>I<3000003FFFE07FFFE07FFFC0600180E00300C006
00C00C0000180000180000300000600000C00001C0000180000380000300000700000600000E00
000E00001E00001C00001C00003C00003C00003C0000780000780000780000300000131F799D17
>I<003F0000FF8001C1E00380E00700700E00700E00700E00700E00E00F00E00F81C00FC38007
F70003FC0001FC0003FF000F3F801C0F803807C03803C07001C07001C0E001C0E001C0E001C0E0
03807003007007003C1C001FF80007E000141F7D9D17>I<007E0001FF000383800701C00E01C0
1E01C01C01E03C01E03C01E07801E07801E07801E07801E07803E07803E03807C03807C01C0FC0
1FFBC00FE380000780000780000700000E00600E00F01C00F03800E07000E0E0007FC0003F0000
131F7C9D17>I<0000180000003800000038000000780000007C000000FC000000FC000001BC00
0001BC0000033C0000033E0000061E0000061E00000C1E00000C1E0000181E0000181F0000300F
0000300F0000600F00007FFF0000FFFF0000C00F00018007800180078003000780030007800600
0780060007801F0007C0FFC07FFCFFC07FFC1E207E9F22>65 D<0003F808000FFE18003F073800
7801F801F000F803C000F007C00070078000700F0000701F0000701E0000703E0000603C000060
7C0000007C0000007C0000007C000000F8000000F8000000F8000000F8000000F80000C0780000
C0780001807C0001803C0003003C0003001E0006001E000C000F80180007E0700001FFC000007F
00001D217B9F21>67 D<07FFFF0007FFFFE0003C01F0003C00F80078007C0078003C0078001E00
78001E0078001E0078001F00F0001F00F0001F00F0001F00F0001F00F0001F00F0001F01E0001E
01E0003E01E0003E01E0003E01E0003C01E0007C03C0007803C000F003C000F003C001E003C003
C003C00F8007C03F007FFFFC00FFFFE000201F7E9E23>I<07FFFFF807FFFFF8003C00F8003C00
78007800380078003800780038007800380078003800780C3000F0183000F0180000F0180000F0
380000FFF80000FFF80001E0700001E0300001E0300001E0301801E0303001E0003003C0003003
C0006003C0006003C000E003C001C003C003C007C00FC07FFFFF80FFFFFF801D1F7E9E1F>I<07
FFFFF807FFFFF8003C00F8003C0078007800380078003800780038007800380078003800780C30
00F0183000F0180000F0180000F0380000FFF80000FFF80001E0700001E0300001E0300001E030
0001E0300001E0000003C0000003C0000003C0000003C0000003C0000003C0000007C000007FFE
0000FFFE00001D1F7E9E1E>I<0001FC04000FFF0C003F039C007800FC00F000FC03E0007803C0
0078078000380F0000381F0000381E0000383E0000303E0000307C0000007C0000007C0000007C
000000F8000000F8000000F8007FFCF8007FFCF80001E0780001E0780003C07C0003C03C0003C0
3C0003C01E0003C00F0007C007800F8003E0398001FFF080003F80001E217B9F24>I<07FFC7FF
C007FFC7FFC0003C007800003C007800007800F000007800F000007800F000007800F000007800
F000007800F00000F001E00000F001E00000F001E00000F001E00000FFFFE00000FFFFE00001E0
03C00001E003C00001E003C00001E003C00001E003C00001E003C00003C007800003C007800003
C007800003C007800003C007800003C007800007C00F80007FFCFFF800FFF8FFF800221F7E9E22
>I<07FFE007FFE0003C00003C0000780000780000780000780000780000780000F00000F00000
F00000F00000F00000F00001E00001E00001E00001E00001E00001E00003C00003C00003C00003
C00003C00003C00007C000FFFC00FFFC00131F7F9E10>I<07FFF00007FFF000003C0000003C00
0000780000007800000078000000780000007800000078000000F0000000F0000000F0000000F0
000000F0000000F0000001E0000001E0000001E0000001E0018001E0018001E0030003C0030003
C0030003C0070003C0060003C00E0003C01E0007C07E007FFFFC00FFFFFC00191F7E9E1C>76
D<07FC0000FFC007FC0001FFC0003E0001F800003E00037800006E0003F000006E0006F000006E
0006F000006E000CF0000067000CF00000670018F00000C70019E00000C70031E00000C70031E0
0000C70061E00000C38061E00000C380C1E000018380C3C00001838183C00001838183C0000181
C303C0000181C303C0000181C603C0000301C60780000301CC0780000301CC0780000300F80780
000300F80780000700F00780000F80F00F80007FF0E0FFF800FFF0E1FFF8002A1F7E9E2A>I<07
FC03FFC007FC03FFC0003E007C00003E003800006F003000006F003000006F0030000067803000
00678030000063C0300000C3C0600000C3C0600000C1E0600000C1E0600000C0F0600000C0F060
000180F0C000018078C000018078C00001803CC00001803CC00001803CC00003001F800003001F
800003000F800003000F800003000F800007000780000F800700007FF0030000FFF0030000221F
7E9E22>I<0003F800001FFE00003C1F0000F0078001E003C003C001E0078001E00F8000F00F00
00F01F0000F01E0000F83E0000F83C0000F87C0000F87C0000F87C0000F87C0000F8F80001F0F8
0001F0F80001F0F80001F0F80003E0780003E0780007C07C0007C07C000F803C000F003E001E00
1E003C000F00780007C1F00003FFC00000FE00001D217B9F23>I<07FFFF0007FFFFC0003C03E0
003C01F0007800F0007800F8007800F8007800F8007800F8007800F800F001F000F001F000F001
E000F003C000F00F8000FFFE0001FFF80001E0000001E0000001E0000001E0000001E0000003C0
000003C0000003C0000003C0000003C0000003C0000007C000007FFC0000FFFC00001D1F7E9E1F
>I<07FFFC0007FFFF00003C07C0003C03E0007801E0007801F0007801F0007801F0007801F000
7801E000F003E000F003C000F0078000F01F0000FFFC0000FFF00001E0380001E03C0001E01C00
01E01C0001E01E0001E01E0003C03E0003C03E0003C03E0003C03E0003C03E0603C03E0607C03E
0C7FFC1F18FFFC0FF8000003E01F207E9E21>82 D<003F04007FCC01E0FC03807C03003C070038
0600180E00180E00180E00180E00000F00000F00000FE00007FE0003FF8001FFC0007FE00007E0
0001E00000E00000F00000F06000E06000E06000E06000E07001C0700180F80380FE0F00C7FC00
81F80016217D9F19>I<1FFFFFF81FFFFFF81E03C0F83803C03838078038300780387007801860
0780186007803860078030C00F0030000F0000000F0000000F0000000F0000000F0000001E0000
001E0000001E0000001E0000001E0000001E0000003C0000003C0000003C0000003C0000003C00
00003C0000007C00001FFFF0003FFFF0001D1F7B9E21>I<FFFC7FF8FFFC7FF807800F80078007
000F0006000F0006000F0006000F0006000F0006000F0006001E000C001E000C001E000C001E00
0C001E000C001E000C003C0018003C0018003C0018003C0018003C0018003C0018003800300038
00300038003000380060003C00C0001C00C0001E0180000F07000007FC000001F800001D20799E
22>I<FFF07FF83FF0FFF07FF83FF01F000F800F800F000F8007000F000F8006000F000F800600
0F001F800C000F001F800C000F00378018000F00378018000F0063C03000078063C030000780C3
C060000780C3C06000078183C0E000078183C0C000078383C0C000078303E1800007C701E18000
03C601E3000003C601E3000003CC01E6000003CC01E6000003D801EC000003D801EC000003F001
F8000001F000F8000001E000F8000001E000F0000001C000F0000001C000E0000001C000E00000
2C207A9E2F>87 D<03FFC1FFC003FFC1FFC0003E007C00001E007000001E006000000F00C00000
0F01C000000F83800000078300000007C600000003CC00000003FC00000001F800000001F00000
0000F000000000F800000001F800000003FC000000033C000000063C0000000C1E0000001C1E00
0000381F000000300F000000600F800000C007800001C007C000038003C0000FC003E000FFF01F
FE00FFE01FFE00221F7F9E22>I<7FF803FF80FFF803FF0007C000F80007C000E00003C000C000
03E001C00001E003800001F003000000F006000000F80E000000F80C00000078180000007C3000
00003C700000003E600000001EC00000001F800000000F800000000F000000000F000000000F00
0000000E000000001E000000001E000000001E000000001E000000001E000000001C000000003C
00000003FFE0000007FFE00000211F7B9E22>I<03FFFFC003FFFFC003F0078007C00F0007001F
0007001E000E003C000C0078000C00F8000C00F0001801E0000003C00000078000000F8000000F
0000001E0000003C0000007C00000078030000F0030001E0030003E0060003C00600078006000F
000E001F001C001E001C003C007C007801FC00FFFFF800FFFFF8001A1F7D9E1C>I
E /Fp 42 121 df<000FFE00007FFE0000F87E0001E0FE0003E0FE0007C07E0007C07E0007C03E
0007C03E0007C03E0007C03E0007C03E00FFFFFE00FFFFFE0007C03E0007C03E0007C03E0007C0
3E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007C03E0007
C03E0007C03E0007C03E003FF9FFC03FF9FFC01A20809F1D>13 D<387CFEFEFE7C3807077C860F
>46 D<00700000F0000FF000FFF000F3F00003F00003F00003F00003F00003F00003F00003F000
03F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F00003F000
03F00003F000FFFF80FFFF80111D7C9C1A>49 D<07F0001FFE00383F007C1F80FE0FC0FE0FC0FE
0FE0FE07E07C07E03807E0000FE0000FC0000FC0001F80001F00003E0000780000F00000E00001
C0600380600700600C00E01FFFE01FFFC03FFFC07FFFC0FFFFC0FFFFC0131D7D9C1A>I<01FC00
07FF000E0F801E0FC03F07E03F87E03F87E03F87E01F0FC00E0FC0000F80001F0001FC0001FC00
000F800007C00003E00003F03C03F87E03F8FF03F8FF03F8FF03F8FF03F07E03F07C07E03C0FC0
0FFF8003FC00151D7E9C1A>I<0001C00003C00007C00007C0000FC0001FC0003FC00077C00067
C000C7C00187C00387C00707C00E07C00C07C01807C03807C07007C0E007C0FFFFFEFFFFFE000F
C0000FC0000FC0000FC0000FC0000FC001FFFE01FFFE171D7F9C1A>I<3803803FFF803FFF003F
FE003FFC003FF0003F800030000030000030000030000033F8003FFE003C1F00380F80300FC000
0FC0000FE0000FE0780FE0FC0FE0FE0FE0FE0FE0FC0FC0780FC0601F80383F001FFC0007F00013
1D7D9C1A>I<003F0001FFC007E0E00F81E01F03F01E03F03E03F07C03F07C01E07C0000FC1000
FCFF00FDFFC0FF03E0FE01F0FE01F0FC01F8FC01F8FC01F8FC01F87C01F87C01F87C01F83C01F0
3E01F01E03E00F07C007FF8001FE00151D7E9C1A>I<6000007FFFF87FFFF87FFFF07FFFE07FFF
E0FFFFC0E00380C00700C00E00C00C00001C0000380000780000780000F00000F00000F00001F0
0001F00001F00003F00003F00003F00003F00003F00003F00003F00003F00001E000151E7D9D1A
>I<01FC0007FF000F07801C03C01C01E03C01E03C01E03E01E03F81E03FE3C03FF3801FFF000F
FF000FFFC007FFE01FFFE03C7FF0781FF87807F8F001F8F000F8F00078F00078F000707800707C
00E03E03C00FFF8003FC00151D7E9C1A>I<01FC000FFF001F07803E03C07C03E07C01E0FC01F0
FC01F0FC01F0FC01F8FC01F8FC01F8FC01F87C03F87C03F83E07F81FFDF807F9F80041F80001F0
3C01F07E01F07E03E07E03E07E07C03C0780381F001FFC0007F000151D7E9C1A>I<387CFEFEFE
7C38000000000000387CFEFEFE7C3807147C930F>I<07F8001FFE00381F80780F80FC0FC0FC0F
C0FC0FC0780FC0301F80001F00003E00007C0000700000E00000E00000C00000C00000C00000C0
0000C00000C00000000000000000000000000001C00007F00007F00007F00007F00007F00001C0
0012207D9F19>63 D<0000E000000000E000000001F000000001F000000001F000000003F80000
0003F800000007FC00000007FC0000000FFE0000000CFE0000000CFE000000187F000000187F00
0000307F800000303F800000703FC00000601FC00000601FC00000C01FE00000C00FE00001FFFF
F00001FFFFF000018007F000030003F800030003F800060003FC00060001FC000E0001FE00FFE0
1FFFE0FFE01FFFE0231F7E9E28>65 D<FFFFFE00FFFFFFC007E007E007E003F007E001F807E001
FC07E001FC07E001FC07E001FC07E001FC07E001F807E003F807E003F007E00FE007FFFF8007FF
FFE007E003F007E001F807E001FC07E000FC07E000FE07E000FE07E000FE07E000FE07E000FE07
E000FC07E001FC07E001F807E007F0FFFFFFE0FFFFFF001F1F7E9E25>I<FFFFFE0000FFFFFFC0
0007F007F00007F001F80007F000FC0007F0007E0007F0003F0007F0003F0007F0001F8007F000
1F8007F0001F8007F0001FC007F0001FC007F0001FC007F0001FC007F0001FC007F0001FC007F0
001FC007F0001FC007F0001FC007F0001F8007F0001F8007F0001F8007F0003F0007F0003F0007
F0007E0007F000FC0007F001F80007F007F000FFFFFFC000FFFFFE0000221F7E9E28>68
D<FFFFFFE0FFFFFFE007F007E007F001E007F000E007F0006007F0007007F0003007F0003007F0
603007F0603007F0600007F0600007F1E00007FFE00007FFE00007F1E00007F0600007F0600007
F0600C07F0600C07F0000C07F0001807F0001807F0001807F0003807F0007807F000F807F003F0
FFFFFFF0FFFFFFF01E1F7E9E22>I<FFFF80FFFF8007F00007F00007F00007F00007F00007F000
07F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F00007F000
07F00007F00007F00007F00007F00007F00007F00007F000FFFF80FFFF80111F7F9E14>73
D<03FFFE03FFFE000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000F
E0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE0000FE03C0FE07E0FE0FF0FE0FF0F
E0FF0FE0FF1FC07E3F803FFE000FF800171F809E1B>I<FFF000007FF8FFF80000FFF807F80000
FF0006FC0001BF0006FC0001BF0006FC0001BF00067E00033F00067E00033F00063F00063F0006
3F00063F00061F800C3F00061F800C3F00061F800C3F00060FC0183F00060FC0183F000607E030
3F000607E0303F000603F0603F000603F0603F000603F0603F000601F8C03F000601F8C03F0006
00FD803F000600FD803F0006007F003F0006007F003F0006003E003F0006003E003F0006003E00
3F00FFF01C07FFF8FFF01C07FFF82D1F7E9E32>77 D<FFF800FFF0FFF800FFF007FC00060007FE
00060007FF00060006FF800600067F800600063FC00600061FE00600060FF00600060FF8060006
07FC06000603FC06000601FE06000600FF060006007F860006007FC60006003FE60006001FE600
06000FF600060007FE00060003FE00060003FE00060001FE00060000FE000600007E000600003E
000600001E000600001E00FFF0000E00FFF0000600241F7E9E29>I<001FF80000FFFF0001F81F
8007E007E00FC003F01F8001F81F0000F83F0000FC7F0000FE7E00007E7E00007EFE00007FFE00
007FFE00007FFE00007FFE00007FFE00007FFE00007FFE00007FFE00007F7E00007E7F0000FE7F
0000FE3F0000FC3F8001FC1F8001F80FC003F007E007E001F81F8000FFFF00001FF800201F7D9E
27>I<FFFFFE00FFFFFF8007F00FE007F003F007F001F807F001F807F001FC07F001FC07F001FC
07F001FC07F001FC07F001F807F001F807F003F007F00FE007FFFF8007FFFE0007F0000007F000
0007F0000007F0000007F0000007F0000007F0000007F0000007F0000007F0000007F0000007F0
0000FFFF8000FFFF80001E1F7E9E24>I<03FC080FFF381E03F83C00F8780078780038F80038F8
0018FC0018FC0000FF0000FFF8007FFF007FFFC03FFFE01FFFF00FFFF803FFF8001FFC0001FC00
00FC0000FCC0007CC0007CC0007CE00078E00078F800F0FE01E0E7FFC081FF00161F7D9E1D>83
D<7FFFFFFC7FFFFFFC7C0FE07C700FE01C600FE00C600FE00CE00FE00EC00FE006C00FE006C00F
E006C00FE006000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE00000
0FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE000000FE00007FFFFC0
07FFFFC01F1E7E9D24>I<FFFF81FFE0FFFF81FFE007F0000C0007F0000C0007F0000C0007F000
0C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0
000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007F0000C0007
F0000C0007F0000C0007F0001C0003F000180003F800380001F800300000FC007000007F03E000
001FFF80000003FC0000231F7E9E28>I<FFFF1FFFC0FFE0FFFF1FFFC0FFE00FE001FC000E0007
F001FC000C0007F001FE000C0007F800FE000C0003F800FE00180003F800FF00180003FC00FF00
380001FC01FF80300001FC01BF80300000FE01BF80600000FE031FC0600000FF031FC0E000007F
031FE0C000007F060FE0C000003F860FE18000003F8E0FF18000003FCC07F18000001FCC07F300
00001FD803FB0000001FF803FF0000000FF803FE0000000FF001FE00000007F001FC00000007E0
00FC00000007E000FC00000003E000F800000003C0007800000001C00070000000018000300000
331F7F9E36>87 D<3FFFFF803FFFFF803F807F803E00FF003801FE003801FE007003FC007003F8
006007F800600FF000600FF000001FE000003FC000003FC000007F8000007F000000FF000001FE
000001FE018003FC018007F8018007F801800FF003800FE003801FE003003FC007003FC00F007F
801F00FF007F00FFFFFF00FFFFFF00191F7D9E20>90 D<07FC001FFF003F0F803F07C03F03E03F
03E00C03E00003E001FFE00FFFE03F83E07E03E07C03E0F803E0F803E0F803E0FC07E07E0DE03F
F9FE07E07E17147F9319>97 D<01FE0007FF801F0FC03E0FC03E0FC07C0FC07C0300FC0000FC00
00FC0000FC0000FC0000FC00007C00007E00003E00603F00C01F81C007FF0001FC0013147E9317
>99 D<01FE0007FF801F83E03F01F07E00F07E00F8FC00F8FC00F8FFFFF8FFFFF8FC0000FC0000
FC00007C00007E00003E00183F00380F807007FFE000FF8015147F9318>101
D<01FC3C07FFFE0F07DE1E03DE3E03E03E03E03E03E03E03E03E03E01E03C00F07800FFF0019FC
001800001800001C00001FFF801FFFF00FFFF83FFFFC7C007C70003EF0001EF0001EF0001E7800
3C78003C3F01F80FFFE001FF00171E7F931A>103 D<1C003F007F007F007F003F001C00000000
000000000000000000FF00FF001F001F001F001F001F001F001F001F001F001F001F001F001F00
1F001F001F00FFE0FFE00B217EA00E>105 D<FF00FF001F001F001F001F001F001F001F001F00
1F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F001F
00FFE0FFE00B207E9F0E>108 D<FE0FE03F80FE3FF0FFC01E70F9C3E01EC07F01F01F807E01F0
1F807E01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01
F01F007C01F01F007C01F01F007C01F01F007C01F01F007C01F0FFE3FF8FFEFFE3FF8FFE27147D
932C>I<FE0FC0FE3FE01E61F01EC0F81F80F81F00F81F00F81F00F81F00F81F00F81F00F81F00
F81F00F81F00F81F00F81F00F81F00F81F00F8FFE3FFFFE3FF18147D931D>I<01FF0007FFC01F
83F03E00F83E00F87C007C7C007CFC007EFC007EFC007EFC007EFC007EFC007E7C007C7C007C3E
00F83E00F81F83F007FFC001FF0017147F931A>I<FF1FC0FF7FF01FE1F81F80FC1F007E1F007E
1F003E1F003F1F003F1F003F1F003F1F003F1F003F1F003E1F007E1F007C1F80FC1FC1F81F7FE0
1F1F801F00001F00001F00001F00001F00001F00001F0000FFE000FFE000181D7E931D>I<FE3E
00FE7F801ECFC01F8FC01F8FC01F8FC01F03001F00001F00001F00001F00001F00001F00001F00
001F00001F00001F00001F0000FFF000FFF00012147E9316>114 D<0180018001800380038003
8007800F803F80FFFCFFFC0F800F800F800F800F800F800F800F800F800F800F860F860F860F86
0F8607CC03F801F00F1D7F9C14>116 D<FF07F8FF07F81F00F81F00F81F00F81F00F81F00F81F
00F81F00F81F00F81F00F81F00F81F00F81F00F81F00F81F01F81F01F80F07F807FEFF03F8FF18
147D931D>I<FFE1FF00FFE1FF000F80700007C0E00007E0C00003E1800001F3800000FF000000
7E0000003E0000003F0000007F8000006F800000C7C0000183E0000381F0000701F8000E00FC00
FF81FF80FF81FF8019147F931C>120 D E /Fq 81 126 df<70F8F8F8F8F8F8F8F8F8F8F8F8F8
F8F8F870000000000070F8F8F870051C779B18>33 D<3803007C07807C0F80FE0F80EE0F00EE1F
00EE1F00EE1E00EE3E00FE3E007C3C007C7C00387C0000780000F80000F80000F00001F00001F0
0001E00003E00003E00003C00007C00007C3800787C00F87C00F8FE00F0EE01F0EE01F0EE01E0E
E03E0FE03E07C03C07C018038013247E9F18>37 D<03C00007E0000FF0001E78001C38001C3800
1C38001C38001C7BF01CF3F01EE3F00FE7800FC7000F87000F0F001F0E003F8E007B9E0073DC00
F1DC00E1FC00E0F800E07870F0F87071FE707FFFF03FCFE01F03C0141C7F9B18>I<387C7E7E3E
0E0E0E1E3C78F8F0C0070E789B18>I<007000F001F003C007800F001E001C003C003800780070
007000F000E000E000E000E000E000E000E000E000F00070007000780038003C001C001E000F00
078003C001F000F000700C24799F18>I<6000F00078003C001E000F000780038003C001C001E0
00E000E000F00070007000700070007000700070007000F000E000E001E001C003C0038007800F
001E003C007800F00060000C247C9F18>I<01C00001C00001C00001C000C1C180F1C780F9CF80
7FFF001FFC0007F00007F0001FFC007FFF00F9CF80F1C780C1C18001C00001C00001C00001C000
11147D9718>I<00600000F00000F00000F00000F00000F00000F00000F0007FFFC0FFFFE0FFFF
E07FFFC000F00000F00000F00000F00000F00000F00000F00000600013147E9718>I<3C7E7F7F
7F3F0F0E1E7CF870080C788518>I<7FFF00FFFF80FFFF807FFF0011047D8F18>I<78FCFCFCFC78
0606778518>I<000300000780000F80000F80000F00001F00001F00003E00003E00003C00007C
00007C0000F80000F80000F00001F00001F00003E00003E00007C00007C0000780000F80000F80
001F00001F00001E00003E00003E00007C00007C0000780000F80000F80000F000006000001124
7D9F18>I<01F00007FC000FFE001F1F001C07003803807803C07001C07001C0E000E0E000E0E0
00E0E000E0E000E0E000E0E000E0E000E0E000E0F001E07001C07001C07803C03803801C07001F
1F000FFE0007FC0001F000131C7E9B18>I<01800380078007800F803F80FF80FB806380038003
80038003800380038003800380038003800380038003800380038003807FFCFFFE7FFC0F1C7B9B
18>I<07F8001FFE003FFF007C0F807003C0F001E0F000E0F000E0F000E00000E00000E00001E0
0001C00003C0000780000F00001E00003C0000780000F00003E00007C0000F80001E00E03C00E0
7FFFE0FFFFE07FFFE0131C7E9B18>I<07F8001FFE003FFF007C0F807803C07801C03001C00001
C00003C0000780000F8003FF0003FE0003FF00000F800003C00001C00001E00000E00000E0F000
E0F001E0F001C0F003C07C0F803FFF001FFE0007F800131C7E9B18>I<001F00003F00007F0000
770000F70001E70001C70003C7000787000707000E07001E07003C0700380700780700F00700FF
FFF8FFFFF8FFFFF8000700000700000700000700000700000700007FF000FFF8007FF0151C7F9B
18>I<007E0001FF0007FF800FC3C01F03C03C03C0380180780000700000700000F3F800EFFE00
FFFF00FE0F80F803C0F001C0F001E0F000E0F000E0F000E07000E07001E07801C03C03C01E0F80
0FFF0007FE0003F800131C7E9B18>54 D<E00000FFFFE0FFFFE0FFFFE0E003C0E00780000F0000
0E00001E00001C00003C0000780000700000700000F00000E00000E00001E00001C00001C00001
C00003C000038000038000038000038000038000038000038000131D7E9C18>I<03F8000FFE00
1FFF003E0F803803807001C07001C07001C07001C03803803C07801FFF0007FC000FFE001F1F00
3C07807001C0F001E0E000E0E000E0E000E0E000E07001C07803C03E0F801FFF000FFE0003F800
131C7E9B18>I<3C7E7E7E7E3C0000000000000000387C7E7E7E3E0E1E3C78F060071A789318>
59 D<000300000F80001F80003F0000FE0001FC0003F00007E0001FC0003F80007E0000FC0000
FC00007E00003F80001FC00007E00003F00001FC0000FE00003F00001F80000F8000030011187D
9918>I<7FFFC0FFFFE0FFFFE0FFFFE0000000000000000000000000FFFFE0FFFFE0FFFFE07FFF
C0130C7E9318>I<600000F80000FC00007E00003F80001FC00007E00003F00001FC0000FE0000
3F00001F80001F80003F0000FE0001FC0003F00007E0001FC0003F80007E0000FC0000F8000060
000011187D9918>I<00700000F80000F80000D80000D80001DC0001DC0001DC00018C00038E00
038E00038E00038E000306000707000707000707000707000FFF800FFF800FFF800E03800E0380
1C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>65 D<FFFC00FFFF00FFFF801C03C01C01C01C
00E01C00E01C00E01C00E01C01E01C01C01C07C01FFF801FFF001FFFC01C03C01C00E01C00F01C
00701C00701C00701C00701C00F01C00E01C03E0FFFFC0FFFF80FFFE00141C7F9B18>I<01FCE0
03FEE007FFE00F07E01E03E03C01E07800E07000E07000E0F00000E00000E00000E00000E00000
E00000E00000E00000E00000F000007000E07000E07800E03C01E01E01C00F07C007FF8003FF00
01FC00131C7E9B18>I<7FF800FFFE007FFF001C0F801C03C01C03C01C01E01C00E01C00E01C00
F01C00701C00701C00701C00701C00701C00701C00701C00701C00F01C00E01C00E01C01E01C01
C01C03C01C0F807FFF00FFFE007FF800141C7F9B18>I<FFFFF0FFFFF0FFFFF01C00701C00701C
00701C00701C00001C00001C0E001C0E001C0E001FFE001FFE001FFE001C0E001C0E001C0E001C
00001C00001C00381C00381C00381C00381C0038FFFFF8FFFFF8FFFFF8151C7F9B18>I<FFFFE0
FFFFE0FFFFE01C00E01C00E01C00E01C00E01C00001C00001C1C001C1C001C1C001FFC001FFC00
1FFC001C1C001C1C001C1C001C00001C00001C00001C00001C00001C00001C0000FFC000FFC000
FFC000131C7E9B18>I<01F9C007FFC00FFFC01F0FC01C03C03C03C07801C07001C07001C0F000
00E00000E00000E00000E00000E00000E00FF0E01FF0E00FF0F001C07001C07003C07803C03C03
C01C07C01F0FC00FFFC007FDC001F9C0141C7E9B18>I<7F07F0FF8FF87F07F01C01C01C01C01C
01C01C01C01C01C01C01C01C01C01C01C01C01C01FFFC01FFFC01FFFC01C01C01C01C01C01C01C
01C01C01C01C01C01C01C01C01C01C01C01C01C07F07F0FF8FF87F07F0151C7F9B18>I<7FFF00
FFFF807FFF0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C000
01C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0007FFF00FFFF80
7FFF00111C7D9B18>I<01FFC003FFC001FFC0000E00000E00000E00000E00000E00000E00000E
00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00600E00F00E
00F01E00F03C007FFC003FF8001FE000121C7D9B18>I<7F07F0FF87F87F07F01C03C01C07801C
07001C0F001C1E001C3C001C38001C78001CF0001DF0001DF8001FF8001FBC001F1C001E1E001E
0E001C0F001C07001C07801C03801C03C01C01C07F03F0FF87F87F03F0151C7F9B18>I<7FE000
FFE0007FE0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0000
0E00000E00000E00000E00000E00000E00000E00700E00700E00700E00700E00707FFFF0FFFFF0
7FFFF0141C7F9B18>I<FC01F8FE03F8FE03F83B06E03B06E03B06E03B06E03B8EE03B8EE0398C
E0398CE039DCE039DCE039DCE038D8E038D8E038F8E03870E03870E03800E03800E03800E03800
E03800E03800E0FE03F8FE03F8FE03F8151C7F9B18>I<7E07F0FF0FF87F07F01D81C01D81C01D
81C01DC1C01CC1C01CC1C01CE1C01CE1C01CE1C01C61C01C71C01C71C01C31C01C39C01C39C01C
39C01C19C01C19C01C1DC01C0DC01C0DC01C0DC07F07C0FF87C07F03C0151C7F9B18>I<0FF800
3FFE007FFF00780F00700700F00780E00380E00380E00380E00380E00380E00380E00380E00380
E00380E00380E00380E00380E00380E00380E00380E00380F00780700700780F007FFF003FFE00
0FF800111C7D9B18>I<FFFE00FFFF80FFFFC01C03C01C01E01C00E01C00701C00701C00701C00
701C00701C00E01C01E01C03C01FFFC01FFF801FFE001C00001C00001C00001C00001C00001C00
001C00001C0000FF8000FF8000FF8000141C7F9B18>I<0FF8003FFE007FFF00780F00700700F0
0780E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E00380E0
0380E00380E1E380E1E380F0E78070F700787F007FFF003FFE000FFC00001C00001E00000E0000
0F0000070000070011227D9B18>I<7FF800FFFE007FFF001C0F801C03801C03C01C01C01C01C0
1C01C01C03C01C03801C0F801FFF001FFE001FFE001C0F001C07801C03801C03801C03801C0380
1C03801C039C1C039C1C039C7F03FCFF81F87F00F0161C7F9B18>I<07F3801FFF803FFF807C1F
80700780F00380E00380E00380E00000F000007800003F00001FF0000FFE0001FF00001F800003
C00001E00000E00000E06000E0E000E0E001E0F001C0FC07C0FFFF80FFFF00E7FC00131C7E9B18
>I<7FFFF8FFFFF8FFFFF8E07038E07038E07038E0703800700000700000700000700000700000
700000700000700000700000700000700000700000700000700000700000700000700000700007
FF0007FF0007FF00151C7F9B18>I<FF83FEFF83FEFF83FE1C00701C00701C00701C00701C0070
1C00701C00701C00701C00701C00701C00701C00701C00701C00701C00701C00701C00701C0070
1E00F00E00E00F01E007C7C003FF8001FF00007C00171C809B18>I<FF07F8FF07F8FF07F81C01
C01C01C01C01C01C01C00E03800E03800E03800E03800F0780070700070700070700070700038E
00038E00038E00038E00018C0001DC0001DC0001DC0000D80000F80000F800007000151C7F9B18
>I<FE03F8FE03F8FE03F87000707000707000703800E03800E03800E03800E03800E038F8E038
F8E039DCE039DCE019DCC019DCC019DCC0198CC01D8DC01D8DC01D8DC01D8DC00D8D800D05800F
07800F07800E0380151C7F9B18>I<7F8FE07F9FE07F8FE00E07000F0700070E00078E00039C00
03DC0001F80001F80000F00000F00000700000F00000F80001F80001DC00039E00038E00070F00
0707000E07800E03801E03C07F07F0FF8FF87F07F0151C7F9B18>I<FF07F8FF07F8FF07F81C01
C01E03C00E03800F0780070700070700038E00038E0001DC0001DC0001FC0000F80000F8000070
0000700000700000700000700000700000700000700000700001FC0003FE0001FC00151C7F9B18
>I<3FFFE07FFFE07FFFE07001C07003C0700780700700000F00001E00001C00003C0000780000
700000F00001E00001C00003C0000780000700000F00001E00E01C00E03C00E07800E07000E0FF
FFE0FFFFE0FFFFE0131C7E9B18>I<FFF8FFF8FFF8E000E000E000E000E000E000E000E000E000
E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E0
00E000FFF8FFF8FFF80D24779F18>I<FFF8FFF8FFF80038003800380038003800380038003800
380038003800380038003800380038003800380038003800380038003800380038003800380038
00380038FFF8FFF8FFF80D247F9F18>93 D<7FFF00FFFF80FFFF807FFF0011047D7F18>95
D<1FE0003FF8007FFC00783E00300F0000070000070001FF000FFF003FFF007F0700780700F007
00E00700E00700F00F00783F007FFFF03FFBF00FE1F014147D9318>97 D<7E0000FE00007E0000
0E00000E00000E00000E00000E00000E3E000EFF800FFFC00FE3E00F80F00F00700F00780E0038
0E00380E00380E00380E00380F00380F00780F00700F80F00FC3E00FFFC00EFF80067E00151C80
9B18>I<01FE0007FF001FFF803F07803C0300780000700000F00000E00000E00000E00000E000
00F000007000007801C03C01C03F07C01FFF8007FF0001FC0012147D9318>I<001F80003F8000
1F8000038000038000038000038000038003F3800FFB801FFF803E1F80780F80700780F00780E0
0380E00380E00380E00380E00380E00780F00780700780780F803E3F801FFFF00FFBF803E3F015
1C7E9B18>I<03F0000FFC001FFE003E1F00780780700380F003C0E001C0E001C0FFFFC0FFFFC0
FFFFC0F000007000007801C03C01C03F07C01FFF8007FF0001FC0012147D9318>I<001FC0007F
E000FFE001F1E001C0C001C00001C00001C0007FFFC0FFFFC0FFFFC001C00001C00001C00001C0
0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0007FFF007FFF007FFF
00131C7F9B18>I<03F1F007FFF80FFFF81E1F303C0F003807003807003807003807003807003C
0F001E1E001FFC003FF8003BF0003800003C00001FFF001FFFC03FFFE07801F0F00078E00038E0
0038E00038F000787800F07E03F03FFFE00FFF8003FE00151F7F9318>I<7E0000FE00007E0000
0E00000E00000E00000E00000E00000E3F000EFF800FFFC00FE1E00F80E00F00E00F00E00E00E0
0E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFFE7FE7FC3FC171C80
9B18>I<03800007C00007C00007C0000380000000000000000000000000007FC000FFC0007FC0
0001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C00001C0
0001C000FFFF00FFFF80FFFF00111D7C9C18>I<0038007C007C007C003800000000000000000F
FC1FFC0FFC001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C001C
001C001C001C001C003C6038F078FFF07FE03FC00E277E9C18>I<FE0000FE0000FE00000E0000
0E00000E00000E00000E00000E3FF00E7FF00E3FF00E07800E0F000E1E000E3C000E78000EF000
0FF8000FFC000F9C000F1E000E0F000E07800E03800E03C0FFC7F8FFC7F8FFC7F8151C7F9B18>
I<7FE000FFE0007FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0
0000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0007FFF
C0FFFFE07FFFC0131C7E9B18>I<7DF1F000FFFBF8007FFFFC001F1F1C001E1E1C001E1E1C001C
1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C001C1C1C00
1C1C1C007F1F1F00FFBFBF807F1F1F001914819318>I<7E3F00FEFF807FFFC00FE1E00F80E00F
00E00F00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E07FC3FCFF
E7FE7FC3FC1714809318>I<01F0000FFE001FFF003E0F803803807001C07001C0E000E0E000E0
E000E0E000E0E000E0F001E07001C07803C03C07803E0F801FFF000FFE0001F00013147E9318>
I<7E3E00FEFF807FFFC00FE3E00F80F00F00700F00780E00380E00380E00380E00380E00380F00
380F00780F00700F80F00FC3E00FFFC00EFF800E7E000E00000E00000E00000E00000E00000E00
000E00007FC000FFE0007FC000151E809318>I<03F3800FFB801FFF803E1F80780F80700780F0
0780E00380E00380E00380E00380E00380E00380F00780700780780F803E1F801FFF800FFB8003
F380000380000380000380000380000380000380000380003FF8003FF8003FF8151E7E9318>I<
7F87E0FF9FF87FBFF803FC7803F03003E00003C00003C00003C000038000038000038000038000
0380000380000380000380007FFE00FFFF007FFE0015147F9318>I<0FF7003FFF007FFF00F81F
00E00700E00700F007007C00007FF0001FFC0007FE00001F00600780E00380F00380F00780FC0F
00FFFF00FFFE00E7F80011147D9318>I<0180000380000380000380000380007FFFC0FFFFC0FF
FFC00380000380000380000380000380000380000380000380000380000380400380E00380E003
81E003C3C001FFC000FF80007E0013197F9818>I<7E07E0FE0FE07E07E00E00E00E00E00E00E0
0E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E00E01E00F07E007FFFC03FFFE
01FCFC1714809318>I<7F8FF0FF8FF87F8FF01E03C00E03800E03800E03800707000707000707
00038E00038E00038E00038E0001DC0001DC0001DC0000F80000F80000700015147F9318>I<FF
8FF8FF8FF8FF8FF83800E03800E03800E01C01C01C01C01C71C01CF9C01CF9C01CD9C01CD9C00D
DD800DDD800DDD800D8D800F8F800F8F8007070015147F9318>I<7F8FF07F9FF07F8FF0070700
078E00039E0001DC0001F80000F80000700000F00000F80001DC00039E00038E000707000F0780
7F8FF0FF8FF87F8FF015147F9318>I<7F8FF0FF8FF87F8FF00E01C00E03800E03800703800707
00070700038700038600038E0001CE0001CE0000CC0000CC0000DC000078000078000078000070
0000700000700000F00000E00079E0007BC0007F80003F00001E0000151E7F9318>I<3FFFF07F
FFF07FFFF07001E07003C0700780000F00001E00007C0000F80001F00003E0000780000F00701E
00703C0070780070FFFFF0FFFFF0FFFFF014147F9318>I<0007E0003FE0007FE000FC0000E000
00E00000E00000E00000E00000E00000E00000E00000E00000E00000E00001E0007FC000FF8000
FF80007FC00001E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E000
00E00000FC00007FE0003FE00007E013247E9F18>I<7C0000FF8000FFC00007E00000E00000E0
0000E00000E00000E00000E00000E00000E00000E00000E00000E00000F000007FC0003FE0003F
E0007FC000F00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0
0007E000FFC000FF80007C000013247E9F18>125 D E /Fr 44 123 df<F8F8F8F8F805057B84
10>46 D<007E0001FF8007FFE00FFFF01FC3F81F00F83E007C3E007C3C003C7C003E7C003E7800
1E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF8001FF800
1FF8001FF8001FF8001F78001E7C003E7C003E7C003E7C003E3E007C3F00FC1F00F81FC3F80FFF
F007FFE001FF80007E0018297EA71D>48 D<00180000380000F80007F800FFF800FFF800FFF800
F8F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F800
00F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F80000F800
00F80000F80000F8007FFFF07FFFF07FFFF07FFFF014287CA71D>I<00FE0003FF800FFFC01FFF
E01F07F03C01F87800FC78007C70007CF0003EF0003E60003E20003E00003E00003E00003E0000
7C00007C0000780000F80001F00001E00003E00007C0000F80001F00003E00007C0000F80001F0
0003E00003C0000780000F00001E00003C00007FFFFE7FFFFE7FFFFE7FFFFE17287EA71D>I<00
7E0003FF8007FFE00FFFF01F81F83E00F87C007C78007C30007C10007C00007C00007C00007800
00F80000F80001F00007E000FFC000FF8000FFC000FFF00001F80000FC00007C00003E00003E00
001F00001F00001F00001F00001F80001FC0003EE0003EF0007E7C00FC3F01F81FFFF00FFFE003
FFC000FE0018297EA71D>I<0007E0000FE0000BE0001BE0001BE0003BE0003BE00073E00073E0
00F3E000F3E001E3E001E3E003E3E003C3E007C3E00F83E00F83E01F03E01F03E03E03E03E03E0
7C03E07C03E0F803E0FFFFFFFFFFFFFFFFFFFFFFFF0003E00003E00003E00003E00003E00003E0
0003E00003E00003E00003E018277EA61D>I<3FFFF83FFFF83FFFF83FFFF83E00003E00003E00
003E00003E00003E00003E00003E00003E00003E00003E3E003EFF803FFFE03FFFF03FC1F03F80
F83F007C3E007C00007C00003E00003E00003E00003E00003E00003E00003E20003E60007C7000
7CF800F87C01F87F07F03FFFE00FFFC007FF0001FC0017287EA61D>I<000FE0003FF800FFF801
FFF803F81807E0000F80000F80001F00003E00003E00003C00007C00007C02007C3FC0F8FFE0F9
FFF0FBFFF8FF80FCFE007CFE003EFC003EFC003FFC001FF8001FF8001FF8001FF8001F78001F7C
001F7C001F7C001E3C003E3E003E1E007C1F00FC0FC1F807FFF003FFE001FF80007E0018297EA7
1D>I<FFFFFFFFFFFFFFFFFFFFFFFF00000F00001E00003C0000780000780000F00001F00001E0
0003E00003C00007C0000F80000F80000F00001F00001E00003E00003E00003C00007C00007C00
007C0000F80000F80000F80000F80000F80000F00001F00001F00001F00001F00001F00001F000
01F00018277EA61D>I<007E0003FFC007FFE00FFFF01F81F83F00FC3E007C7E007E7C003E7C00
3E7C003E7C003E7C003E3E007C3E007C1F00F80FC3F007FFE003FFC003FFC00FFFF01F81F83F00
FC3E007C7C003E7C003EF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E7E007E3F00
FC1F81F80FFFF007FFE003FFC0007E0018297EA71D>I<007E0001FF8007FFC00FFFE01F83F01F
00F83E00787C007C7C003C78003EF8003EF8003EF8001EF8001FF8001FF8001FF8001FF8003FFC
003F7C003F7C007F3E007F3F01FF1FFFDF0FFF9F07FF1F03FC1E00403E00003E00003E00007C00
007C0000F80000F80001F01803E01E0FC03FFF801FFF000FFE0003F80018297EA71D>I<0003F0
00000003F000000007F800000007F800000007780000000F7C0000000F3C0000000F3C0000001F
3E0000001E3E0000001E1E0000003E1F0000003C1F0000007C0F8000007C0F800000780F800000
F807C00000F807C00000F007C00001F003E00001F003E00001E003E00003E001F00003E001F000
03C001F00007C000F80007FFFFF80007FFFFF8000FFFFFFC000FFFFFFC000F00007C001F00003E
001F00003E001E00003E003E00001F003E00001F003C00001F007C00000F807C00000F80F80000
0FC0F8000007C0F8000007C0222A7EA927>65 D<0003FE00000FFFC0003FFFF000FFFFF001FE03
F003F8006007E0002007C000000F8000001F8000001F0000003E0000003E0000007C0000007C00
00007C0000007C000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8
000000F8000000F80000007C0000007C0000007C0000007C0000003E0000003E0000001F000000
1F8000000F80000007C0000007E0000803F8003801FE01F800FFFFF8003FFFF0000FFFC00003FE
001D2C7CAA25>67 D<FFFFFFC0FFFFFFC0FFFFFFC0FFFFFFC0F8000000F8000000F8000000F800
0000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8
000000FFFFFF00FFFFFF00FFFFFF00FFFFFF00F8000000F8000000F8000000F8000000F8000000
F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00F8000000FFFFFFE0FFFFFFE0FFFFFFE0FFFFFFE01B2A7BA923>69 D<FFFFFF80FFFFFF80FFFF
FF80FFFFFF80F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8
000000F8000000F8000000F8000000F8000000F8000000F8000000FFFFFC00FFFFFC00FFFFFC00
FFFFFC00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F800
0000192A7BA921>I<0003FE00000FFFC0003FFFF8007FFFFC01FE01FC03F8007C07E0001807C0
00000F8000001F8000001F0000003E0000003E0000007E0000007C0000007C0000007C000000F8
000000F8000000F8000000F8000000F8000000F8000000F8000000F8003FFCF8003FFCF8003FFC
7C003FFC7C00007C7C00007C7E00007C3E00007C3E00007C1F00007C1F80007C0F80007C07C000
7C07E0007C03F8007C01FE01FC007FFFFC003FFFF8000FFFC00003FE001E2C7CAA27>I<F80000
7CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800
007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CFFFFFFFCFFFFFFFCFF
FFFFFCFFFFFFFCF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007C
F800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF800007CF80000
7CF800007CF800007C1E2A7BA929>I<F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8
F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8052A7BA910>I<FE000000FFFE000000FFFE0000
00FFFF000001FFFB000001DFFB800003DFFB800003DFFB800003DFF9C000079FF9C000079FF9C0
00079FF9E0000F9FF8E0000F1FF8E0000F1FF8F0001F1FF870001E1FF870001E1FF878003E1FF8
38003C1FF83C007C1FF83C007C1FF81C00781FF81E00F81FF81E00F81FF80E00F01FF80F01F01F
F80F01F01FF80701E01FF80783E01FF80783E01FF80383C01FF803C7C01FF803C7C01FF801C780
1FF801EF801FF801EF801FF800EF001FF800FF001FF8007E001FF8007E001FF8007E001FF80000
001F282A7BA933>77 D<0003FC0000000FFF0000003FFFC00000FFFFF00001FE07F80003F801FC
0007E0007E000FC0003F000F80001F001F00000F801F00000F803E000007C03E000007C07C0000
03E07C000003E07C000003E078000001E0F8000001F0F8000001F0F8000001F0F8000001F0F800
0001F0F8000001F0F8000001F0F8000001F0F8000001F0F8000001F07C000003E07C000003E07C
000003E07E000007E03E000007C03E000007C01F00000F801F80001F800F80001F000FC0003F00
07E0007E0003F801FC0001FE07F80000FFFFF000003FFFC000001FFF80000003FC0000242C7DAA
2B>79 D<FFFFC000FFFFF800FFFFFE00FFFFFF00F8007F80F8001FC0F80007C0F80003E0F80003
E0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80003E0F80003E0F800
07C0F8001FC0F8007F80FFFFFF00FFFFFE00FFFFF800FFFFC000F8000000F8000000F8000000F8
000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000F8000000
F8000000F8000000F8000000F80000001C2A7BA925>I<FFFFE000FFFFFC00FFFFFE00FFFFFF80
F8003FC0F8000FE0F80003F0F80001F0F80000F0F80000F8F80000F8F80000F8F80000F8F80000
F8F80000F0F80001F0F80003F0F8000FE0F8003FC0FFFFFF80FFFFFE00FFFFFC00FFFFE000F801
F000F801F000F800F800F800F800F8007C00F8003C00F8003E00F8001E00F8001F00F8000F80F8
000F80F80007C0F80007C0F80003E0F80001E0F80001F0F80000F8F80000F8F800007C1E2A7BA9
26>82 D<007FC00001FFF80007FFFE000FFFFF001FC07F003F000F007E0006007C0000007C0000
00F8000000F8000000F8000000F8000000F8000000FC0000007E0000007F0000003F8000001FF8
00000FFF000007FFE00003FFF80000FFFC00000FFE000000FF0000003F0000001F8000000F8000
000FC0000007C0000007C0000007C0000007C0000007C0000007C000000F8060000F80F0001F00
FC003F00FF80FE007FFFFC001FFFF80007FFE00000FF80001A2C7DAA21>I<F80001F80001F0F8
0003F80001F0FC0003FC0001F07C0003FC0003E07C0003BC0003E07C0007BC0003E03E00079E00
03C03E00079E0007C03E00079E0007C03E000F9F0007C01F000F1F0007801F000F0F000F801F00
0F0F000F801F001F0F800F800F801E0F800F000F801E07801F000F801E07801F000F803E07C01F
0007C03C07C01E0007C03C03C03E0007C03C03C03E0007C07C03E03E0003E07803E03C0003E078
01E07C0003E07801E07C0001E0F801F0780001E0F001F0780001F0F000F0F80001F0F000F0F800
00F0F000F0F00000F1E000F8F00000F9E00078F00000F9E00079F0000079E00079E0000079C000
79E000007BC0003DE000007BC0003DE000003BC0003DC000003F80003FC000003F80001FC00000
1F80001F8000001F00001F8000342A7FA937>87 D<01FC000FFF003FFF803FFFC03C07E03003E0
0001F00001F00001F00001F00001F00003F001FFF007FFF01FFFF03FC1F07E01F0FC01F0F801F0
F801F0F801F0FC03F07E0FF07FFFF03FFFF01FF9F00FE1F0141B7E9A1C>97
D<007F8001FFE007FFF80FFFF81FC0783F00103E00007C00007C00007C0000F80000F80000F800
00F80000F80000F80000F800007C00007C00007C00003E00003F00181FC0780FFFF807FFF801FF
E0007F80151B7E9A1A>99 D<00007C00007C00007C00007C00007C00007C00007C00007C00007C
00007C00007C00007C00007C00007C00007C00FC7C03FF7C07FFFC0FFFFC1FC1FC3F00FC3E007C
7C007C7C007CFC007CF8007CF8007CF8007CF8007CF8007CF8007CF8007CF8007C7C007C7C007C
7E00FC3F00FC1F83FC1FFFFC0FFF7C03FE7C01F87C162A7EA91E>I<00FC0003FF0007FF800FFF
C01F87E03F01E03E00F07C00F07C0070F80078FFFFF8FFFFF8FFFFF8FFFFF8F80000F80000F800
007800007C00007C00003E00003F00181FC0F80FFFF807FFF801FFE0007F00151B7E9A1A>I<00
1FC0007FC000FFC001FFC003F00003E00007C00007C00007C00007C00007C00007C00007C00007
C00007C000FFFE00FFFE00FFFE00FFFE0007C00007C00007C00007C00007C00007C00007C00007
C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007
C00007C00007C000122A7FA912>I<F80000F80000F80000F80000F80000F80000F80000F80000
F80000F80000F80000F80000F80000F80000F80000F87E00F9FF00FBFF80FFFFC0FF0FC0FE07E0
FC03E0FC03E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0
F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0132A7BA91E>104
D<F8F8F8F8F800000000000000000000F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8
F8F8F8F8052A7CA90E>I<F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8F8
F8F8F8F8F8F8F8F8F8F8F8F8F8F8052A7CA90E>108 D<F83F007E00F9FF83FF00FBFFC7FF80FF
FFEFFFC0FF07FE0FC0FE03FC07E0FC01F803E0FC01F803E0F801F003E0F801F003E0F801F003E0
F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003
E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F003E0F801F0
03E0231B7B9A30>I<F87E00F9FF00FBFF80FFFFC0FF0FC0FE07E0FC03E0FC03E0F803E0F803E0
F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0
F803E0F803E0F803E0F803E0131B7B9A1E>I<007E0001FF8007FFE00FFFF01F81F83F00FC3E00
7C7C003E7C003E78001EF8001FF8001FF8001FF8001FF8001FF8001FF8001F7C003E7C003E7C00
3E3E007C3F00FC1FC3F80FFFF007FFE003FFC0007E00181B7E9A1D>I<F87E00F9FF80FFFFC0FF
FFE0FF0FE0FC03F0F801F8F800F8F800F8F800FCF8007CF8007CF8007CF8007CF8007CF8007CF8
007CF800FCF800F8F801F8F801F0FC03F0FE0FE0FFFFC0FFFF80FBFF00F8FC00F80000F80000F8
0000F80000F80000F80000F80000F80000F80000F80000F80000F8000016277B9A1E>I<F838F8
F8F9F8FBF8FFC0FF00FE00FE00FC00FC00F800F800F800F800F800F800F800F800F800F800F800
F800F800F800F800F800F8000D1B7B9A14>114 D<07F8001FFF003FFF807FFF807C0780F80100
F80000F80000F80000FC00007F00007FF0003FFC001FFE0007FF0000FF80001FC0000FC00007C0
0007C04007C0E007C0FC1F80FFFF807FFF001FFE0007F800121B7E9A16>I<07C00007C00007C0
0007C00007C00007C00007C000FFFF80FFFF80FFFF80FFFF8007C00007C00007C00007C00007C0
0007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C00007C0
8007E38003FFC003FFC001FF0000FC0012227FA115>I<F803E0F803E0F803E0F803E0F803E0F8
03E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F803E0F8
03E0F803E0F807E0F807E0FC1FE0FFFFE07FFBE03FF3E01F83E0131B7B9A1E>I<F800F800F8F8
01F800F87C01FC00F07C01DC01F07C01DC01F07C03DC01F03E039E03E03E039E03E03E079E03E0
3E079E03E01F078F07C01F070F07C01F070F07C00F0F0F07800F8F078F800F8F078F800F8E078F
80079E078F00079E03CF0007DE03DF0003DC03DE0003DC03DE0003DC01DE0003D801DE0001F801
FC0001F801FC0001F800FC00251B7F9A28>119 D<7C001F803E003F003F003E001F807C000F80
FC0007C0F80007E1F00003F3E00001F7E00000FFC000007F8000007F0000003E0000003E000000
3F0000007F800000FFC00001F3E00003E1E00003E1F00007C0F8000F807C001F807C001F003E00
3E001F007C001F80FC000FC01A1B809A1B>I<F8001FFC001F7C003E7C003E3E003E3E007C3F00
7C1F007C1F00F80F80F80F80F00F81F007C1F007C1E003C1E003E3E003E3C001E3C001E3C000F7
8000F78000F780007700007700003F00003E00003E00003E00003C00003C000078000078000078
0000F00001F0007FE0007FE0007FC0007F000018277F9A1B>I<FFFFF0FFFFF0FFFFF0FFFFE000
07E00007C0000F80001F80001F00003E00007E00007C0000F80001F80001F00003E00007E00007
C0000F80001F80001F00003E00007E00007FFFF8FFFFF8FFFFF8FFFFF8151B7E9A19>I
E /Fs 22 122 df<FFFFFFFFFFFFC00000FFFFFFFFFFFFFC0000FFFFFFFFFFFFFF8000FFFFFFFF
FFFFFFE000FFFFFFFFFFFFFFF800001FFF000003FFFC00001FFF0000007FFE00001FFF0000003F
FF00001FFF0000001FFF80001FFF0000000FFFC0001FFF0000000FFFC0001FFF00000007FFE000
1FFF00000007FFE0001FFF00000007FFE0001FFF00000003FFF0001FFF00000003FFF0001FFF00
000003FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000003FFF0001FFF00000003
FFF0001FFF00000003FFF0001FFF00000007FFE0001FFF00000007FFE0001FFF00000007FFC000
1FFF0000000FFFC0001FFF0000000FFF80001FFF0000001FFF00001FFF0000003FFE00001FFF00
00007FFC00001FFF000001FFF800001FFF00000FFFE000001FFFFFFFFFFF8000001FFFFFFFFFFC
0000001FFFFFFFFFFF8000001FFFFFFFFFFFF000001FFF000000FFFC00001FFF0000003FFF0000
1FFF0000000FFF80001FFF00000007FFE0001FFF00000003FFF0001FFF00000001FFF0001FFF00
000001FFF8001FFF00000000FFFC001FFF00000000FFFC001FFF000000007FFE001FFF00000000
7FFE001FFF000000007FFE001FFF000000007FFF001FFF000000007FFF001FFF000000007FFF00
1FFF000000007FFF001FFF000000007FFF001FFF000000007FFF001FFF000000007FFF001FFF00
0000007FFF001FFF000000007FFE001FFF00000000FFFE001FFF00000000FFFE001FFF00000001
FFFC001FFF00000001FFFC001FFF00000003FFF8001FFF00000007FFF0001FFF0000000FFFE000
1FFF0000003FFFC0001FFF000001FFFF80FFFFFFFFFFFFFFFF00FFFFFFFFFFFFFFFC00FFFFFFFF
FFFFFFF000FFFFFFFFFFFFFF8000FFFFFFFFFFFFF8000048477CC653>66
D<000000007FFC00000E0000000FFFFFC0001E0000007FFFFFF8003E000003FFFFFFFE007E0000
0FFFFFFFFF80FE00003FFFF8007FC1FE0000FFFF80000FF3FE0003FFFC000003FFFE0007FFF000
0001FFFE000FFFC00000007FFE001FFF800000003FFE003FFF000000001FFE007FFC000000000F
FE00FFFC000000000FFE01FFF80000000007FE03FFF00000000003FE03FFE00000000003FE07FF
E00000000001FE0FFFC00000000001FE0FFFC00000000000FE1FFF800000000000FE1FFF800000
0000007E3FFF8000000000007E3FFF0000000000007E3FFF0000000000003E7FFF000000000000
3E7FFF0000000000003E7FFF0000000000003E7FFE000000000000007FFE00000000000000FFFE
00000000000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000
00000000FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000000000
00FFFE00000000000000FFFE00000000000000FFFE00000000000000FFFE000000000000007FFE
000000000000007FFE000000000000007FFF000000000000007FFF0000000000003E7FFF000000
0000003E3FFF0000000000003E3FFF0000000000003E3FFF8000000000003E1FFF800000000000
7E1FFF8000000000007C0FFFC000000000007C0FFFC00000000000FC07FFE00000000000FC03FF
E00000000000F803FFF00000000001F801FFF80000000003F000FFFC0000000003F0007FFE0000
000007E0003FFF000000000FC0001FFF800000001F80000FFFC00000003F000007FFF0000000FE
000003FFFC000001FC000000FFFF80000FF80000003FFFF8007FE00000000FFFFFFFFFC0000000
03FFFFFFFF00000000007FFFFFFC00000000000FFFFFE00000000000007FFE00000047497AC754
>I<00000003FFF0000000000000007FFFFF80000000000003FFFFFFF000000000000FFFFFFFFC
00000000003FFF003FFF0000000000FFF80007FFC000000003FFE00001FFF000000007FF800000
7FF80000000FFF0000003FFC0000001FFE0000001FFE0000003FFC0000000FFF0000007FF80000
0007FF800000FFF000000003FFC00001FFF000000003FFE00003FFE000000001FFF00003FFC000
000000FFF00007FFC000000000FFF80007FFC000000000FFF8000FFF80000000007FFC000FFF80
000000007FFC001FFF80000000007FFE001FFF00000000003FFE003FFF00000000003FFF003FFF
00000000003FFF003FFF00000000003FFF007FFF00000000003FFF807FFE00000000001FFF807F
FE00000000001FFF807FFE00000000001FFF807FFE00000000001FFF80FFFE00000000001FFFC0
FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFF
C0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001F
FFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE00000000001FFFC0FFFE0000000000
1FFFC0FFFE00000000001FFFC07FFE00000000001FFF807FFF00000000003FFF807FFF00000000
003FFF807FFF00000000003FFF803FFF00000000003FFF003FFF00000000003FFF003FFF800000
00007FFF001FFF80000000007FFE001FFF80000000007FFE001FFF80000000007FFE000FFFC000
000000FFFC000FFFC000000000FFFC0007FFE000000001FFF80003FFE000000001FFF00003FFF0
00000003FFF00001FFF000000003FFE00000FFF800000007FFC000007FFC0000000FFF8000007F
FE0000001FFF8000003FFF0000003FFF0000000FFFC00000FFFC00000007FFE00001FFF8000000
03FFF80007FFF000000000FFFF003FFFC0000000007FFFFFFFFF80000000000FFFFFFFFC000000
000003FFFFFFF00000000000007FFFFF8000000000000003FFF0000000004A497AC757>79
D<00007FF00007000007FFFF000F00001FFFFFE01F00007FFFFFF83F0000FFFFFFFC7F0003FFC0
0FFFFF0007FE0000FFFF000FFC00003FFF000FF800001FFF001FF000000FFF003FE0000007FF00
3FE0000003FF007FE0000001FF007FC0000000FF007FC0000000FF00FFC00000007F00FFC00000
007F00FFC00000003F00FFE00000003F00FFE00000003F00FFE00000001F00FFF00000001F00FF
F80000001F00FFFC0000001F00FFFE00000000007FFF80000000007FFFF8000000007FFFFF8000
00003FFFFFFC0000003FFFFFFFC000001FFFFFFFF800001FFFFFFFFF00000FFFFFFFFFC00007FF
FFFFFFE00003FFFFFFFFF00001FFFFFFFFFC0000FFFFFFFFFC00003FFFFFFFFE00000FFFFFFFFF
000003FFFFFFFF8000007FFFFFFF80000003FFFFFFC00000003FFFFFC000000001FFFFE0000000
003FFFE0000000000FFFE00000000003FFF00000000001FFF00000000001FFF0F800000000FFF0
F8000000007FF0F8000000007FF0F8000000007FF0F8000000003FF0FC000000003FF0FC000000
003FF0FC000000003FE0FE000000003FE0FE000000003FE0FF000000007FE0FF800000007FC0FF
800000007FC0FFC0000000FF80FFF0000001FF80FFF8000001FF00FFFE000003FE00FFFFC0000F
FC00FF7FFC007FF800FE3FFFFFFFF000FC0FFFFFFFC000F803FFFFFF0000F0007FFFFC0000E000
03FFE0000034497AC741>83 D<0003FFF0000000003FFFFF00000000FFFFFFE0000001FFFFFFF8
000003FF003FFC000007FF800FFE000007FF8007FF000007FF8003FF800007FF8003FFC00007FF
8001FFC00007FF8001FFC00003FF0000FFE00001FE0000FFE00000FC0000FFE00000000000FFE0
0000000000FFE00000000000FFE00000000000FFE00000000000FFE000000000FFFFE00000007F
FFFFE0000007FFFFFFE000003FFFFCFFE00000FFFE00FFE00003FFF000FFE00007FFC000FFE000
0FFF0000FFE0001FFE0000FFE0003FFC0000FFE0007FFC0000FFE0007FF80000FFE000FFF80000
FFE000FFF00000FFE000FFF00000FFE000FFF00000FFE000FFF00001FFE000FFF00001FFE000FF
F80003FFE0007FFC0007FFE0007FFC000FFFE0003FFE001F7FF8001FFF80FE7FFFF007FFFFFC3F
FFF003FFFFF01FFFF0007FFFC00FFFF00007FE0003FFF0342E7DAD38>97
D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC0
0000000003FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC000000000
01FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC000
00000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001
FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00FFE000001FFC0FFFF
C00001FFC3FFFFF00001FFCFFFFFFC0001FFFFE01FFF0001FFFF0007FF8001FFFC0001FFC001FF
F80000FFE001FFF00000FFF001FFE000007FF001FFC000007FF801FFC000003FFC01FFC000003F
FC01FFC000003FFC01FFC000001FFE01FFC000001FFE01FFC000001FFE01FFC000001FFF01FFC0
00001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF
01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000
001FFE01FFC000001FFE01FFC000003FFE01FFC000003FFC01FFC000003FFC01FFC000003FF801
FFC000007FF801FFE000007FF001FFF00000FFE001FFF80001FFE001FFFC0003FFC001FF7F0007
FF0001FE3FC03FFE0001FC0FFFFFF80001FC03FFFFF00001F800FFFF800001F0001FF800003848
7CC741>I<00001FFF80000001FFFFF8000007FFFFFE00001FFFFFFF00007FF801FF8000FFE003
FFC003FFC003FFC007FF8003FFC007FF0003FFC00FFE0003FFC01FFE0003FFC01FFC0001FF803F
FC0000FF003FFC00007E007FFC000000007FF8000000007FF800000000FFF800000000FFF80000
0000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FFF800000000FF
F800000000FFF800000000FFF800000000FFF8000000007FFC000000007FFC000000007FFC0000
00003FFC000000003FFE000000001FFE000001F01FFF000001F00FFF000003F007FF800007E007
FFC00007C003FFE0000FC000FFF8003F80007FFF00FF00001FFFFFFC000007FFFFF0000001FFFF
C00000001FFC00002C2E7CAD34>I<0000000000FF8000000001FFFF8000000001FFFF80000000
01FFFF8000000001FFFF8000000001FFFF800000000007FF800000000003FF800000000003FF80
0000000003FF800000000003FF800000000003FF800000000003FF800000000003FF8000000000
03FF800000000003FF800000000003FF800000000003FF800000000003FF800000000003FF8000
00000003FF800000000003FF800000000003FF800000000003FF800000000003FF800000000003
FF8000001FF803FF800001FFFF03FF80000FFFFFE3FF80003FFFFFFBFF80007FFC03FFFF8001FF
E0007FFF8003FFC0003FFF8007FF80000FFF800FFF000007FF800FFE000003FF801FFE000003FF
801FFC000003FF803FFC000003FF803FFC000003FF807FFC000003FF807FF8000003FF807FF800
0003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80
FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF8000003FF80FFF80000
03FF80FFF8000003FF807FF8000003FF807FF8000003FF807FF8000003FF803FFC000003FF803F
FC000003FF803FFC000003FF801FFE000007FF800FFE000007FF800FFF00000FFF8007FF00001F
FF8003FF80007FFF8001FFE000FFFFC000FFF807FBFFFF003FFFFFE3FFFF000FFFFFC3FFFF0003
FFFE03FFFF00003FF003FFFF38487CC741>I<00003FFE00000001FFFFC000000FFFFFF800003F
FFFFFC00007FFC0FFE0000FFE003FF0001FF8000FF8003FF0000FFC007FF00007FE00FFE00003F
F01FFE00003FF01FFC00003FF03FFC00001FF83FFC00001FF87FFC00001FF87FF800001FFC7FF8
00000FFC7FF800000FFCFFF800000FFCFFFFFFFFFFFCFFFFFFFFFFFCFFFFFFFFFFFCFFFFFFFFFF
FCFFF800000000FFF800000000FFF800000000FFF800000000FFF8000000007FF8000000007FF8
000000007FFC000000007FFC000000003FFC000000003FFC0000007C1FFE0000007C0FFE000000
FC0FFF000000F807FF800001F803FFC00003F001FFE00007E000FFF8001FC0007FFF00FF80001F
FFFFFF000007FFFFFC000000FFFFF00000000FFF00002E2E7DAD35>I<00000000003F000000FF
F000FFC00007FFFE03FFE0003FFFFFCFFFF0007FFFFFFFCFF001FFE07FFE0FF003FF801FFC1FF0
07FF000FFE0FF00FFE0007FF0FF00FFE0007FF07E01FFC0003FF81801FFC0003FF80003FFC0003
FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003FFC0003F
FC0003FFC0003FFC0003FFC0003FFC0003FFC0001FFC0003FF80001FFC0003FF80000FFE0007FF
00000FFE0007FF000007FF000FFE000003FF801FFC000001FFE07FF8000003FFFFFFE0000003FF
FFFFC000000787FFFE0000000780FFF0000000078000000000000F8000000000000F8000000000
000FC000000000000FC000000000000FE000000000000FF8000000000007FFFFFFF0000007FFFF
FFFF000007FFFFFFFFE00003FFFFFFFFF80003FFFFFFFFFC0001FFFFFFFFFE0000FFFFFFFFFF00
03FFFFFFFFFF800FFFFFFFFFFF801FF000001FFFC03FC0000001FFC07F800000007FE07F800000
003FE0FF000000003FE0FF000000001FE0FF000000001FE0FF000000001FE0FF000000001FE0FF
800000003FE07F800000003FC07FC00000007FC03FE0000000FF801FF8000003FF000FFE00000F
FE0007FFE000FFFC0001FFFFFFFFF000003FFFFFFF8000000FFFFFFE000000007FFFC000003444
7DAE3A>103 D<007FC000000000FFFFC000000000FFFFC000000000FFFFC000000000FFFFC000
000000FFFFC00000000003FFC00000000001FFC00000000001FFC00000000001FFC00000000001
FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000
000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF
C00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC007FE00
0001FFC03FFFC00001FFC0FFFFF00001FFC1FFFFFC0001FFC7F01FFC0001FFCFC00FFE0001FFDF
0007FF0001FFDE0007FF0001FFFC0007FF0001FFF80003FF8001FFF80003FF8001FFF00003FF80
01FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC00003FF8001FFC000
03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001
FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF
81FFFFFF38487BC741>I<00FC0001FF0003FF8007FF800FFFC00FFFC00FFFE00FFFE00FFFE00F
FFC00FFFC007FF8003FF8001FF0000FC0000000000000000000000000000000000000000000000
0000000000000000000000000000007FC0FFFFC0FFFFC0FFFFC0FFFFC0FFFFC003FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001FFC001
FFC001FFC001FFC001FFC001FFC001FFC001FFC0FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF18497CC8
20>I<007FC000FFFFC000FFFFC000FFFFC000FFFFC000FFFFC00003FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000FFFF
FF80FFFFFF80FFFFFF80FFFFFF80FFFFFF8019487CC720>108 D<00FF8003FF000003FF000000
FFFF801FFFE0001FFFE00000FFFF807FFFF8007FFFF80000FFFF81FFFFFC01FFFFFC0000FFFF83
F01FFE03F01FFE0000FFFF87C007FF07C007FF000003FF8F0007FF8F0007FF800001FF9E0003FF
9E0003FF800001FFBC0003FFBC0003FF800001FFF80001FFF80001FFC00001FFF80001FFF80001
FFC00001FFF00001FFF00001FFC00001FFE00001FFE00001FFC00001FFE00001FFE00001FFC000
01FFE00001FFE00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC0
0001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FF
C00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001
FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC00001FFC000
01FFC00001FFC00001FFC000FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF
80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80FFFFFF80592E7BAD62>
I<00FF8007FE0000FFFF803FFFC000FFFF80FFFFF000FFFF81FFFFFC00FFFF87F01FFC00FFFF8F
C00FFE0003FF9F0007FF0001FF9E0007FF0001FFBC0007FF0001FFF80003FF8001FFF80003FF80
01FFF00003FF8001FFE00003FF8001FFE00003FF8001FFE00003FF8001FFC00003FF8001FFC000
03FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001
FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF8001FFC00003FF80FFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF81FFFFFFFFFFFF
81FFFFFFFFFFFF81FFFFFF382E7BAD41>I<00000FFE0000000001FFFFF000000007FFFFFC0000
001FFFFFFF0000007FF803FFC00000FFE000FFE00001FF80003FF00003FF00001FF80007FE0000
0FFC000FFE00000FFE000FFC000007FE001FFC000007FF003FFC000007FF803FF8000003FF803F
F8000003FF807FF8000003FFC07FF8000003FFC07FF8000003FFC0FFF8000003FFE0FFF8000003
FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8000003FFE0FFF8
000003FFE0FFF8000003FFE0FFF8000003FFE07FF8000003FFC07FF8000003FFC07FF8000003FF
C07FF8000003FFC03FFC000007FF803FFC000007FF801FFC000007FF001FFC000007FF000FFE00
000FFE0007FF00001FFC0003FF00001FF80001FFC0007FF00000FFE000FFE000007FFC07FFC000
001FFFFFFF00000007FFFFFC00000001FFFFF0000000001FFF000000332E7DAD3A>I<007FC00F
FE0000FFFFC0FFFFC000FFFFC3FFFFF000FFFFCFFFFFFC00FFFFFFE03FFF00FFFFFF000FFF8003
FFFC0007FFC001FFF80003FFE001FFF00001FFF001FFE00000FFF001FFC00000FFF801FFC00000
7FFC01FFC000007FFC01FFC000003FFC01FFC000003FFE01FFC000003FFE01FFC000003FFE01FF
C000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001F
FF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC000001FFF01FFC0
00001FFF01FFC000003FFE01FFC000003FFE01FFC000003FFE01FFC000007FFC01FFC000007FFC
01FFC000007FF801FFC00000FFF801FFE00000FFF001FFF00001FFE001FFF80003FFE001FFFC00
07FFC001FFFF000FFF0001FFFFC07FFE0001FFCFFFFFF80001FFC3FFFFF00001FFC0FFFF800001
FFC01FF8000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000
000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FFC00000000001FF
C00000000001FFC00000000001FFC00000000001FFC00000000001FFC000000000FFFFFF800000
00FFFFFF80000000FFFFFF80000000FFFFFF80000000FFFFFF8000000038427CAD41>I<00FF80
7F00FFFF81FFC0FFFF87FFF0FFFF8FFFF8FFFF9F8FFCFFFF9F1FFE03FFBE1FFE01FFBC1FFE01FF
F81FFE01FFF81FFE01FFF01FFE01FFF00FFC01FFE007F801FFE003F001FFE0000001FFE0000001
FFE0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC00000
01FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC000
0001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0000001FFC0
000001FFC00000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFFE000FFFFFFE000272E7CAD2F>
114 D<000FFE01C000FFFFC7C003FFFFFFC007FFFFFFC01FF001FFC03FC0007FC03F80003FC07F
00001FC07E00000FC0FE00000FC0FE000007C0FE000007C0FF000007C0FF800007C0FFC0000000
FFF8000000FFFFC000007FFFFE00007FFFFFE0003FFFFFF8001FFFFFFE000FFFFFFF0007FFFFFF
8001FFFFFFC0007FFFFFE0001FFFFFF00000FFFFF0000007FFF00000007FF80000001FF8F80000
0FF8F8000007F8FC000007F8FC000003F8FC000003F8FE000003F8FF000003F0FF000007F0FF80
0007F0FFC0000FE0FFF0001FC0FFFC00FF80FFFFFFFF00FC7FFFFC00F01FFFF000E003FF800025
2E7CAD2E>I<0001F000000001F000000001F000000001F000000001F000000001F000000003F0
00000003F000000003F000000007F000000007F000000007F00000000FF00000000FF00000001F
F00000003FF00000003FF00000007FF0000001FFF0000003FFF000000FFFFFFFC0FFFFFFFFC0FF
FFFFFFC0FFFFFFFFC0FFFFFFFFC0007FF00000007FF00000007FF00000007FF00000007FF00000
007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF000
00007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF00000007FF0
0000007FF00000007FF00000007FF001F0007FF001F0007FF001F0007FF001F0007FF001F0007F
F001F0007FF001F0007FF001F0007FF001F0007FF003F0003FF803E0003FF807E0001FFC07C000
1FFE0F80000FFFFF800003FFFE000000FFFC0000001FE00024427EC12D>I<007FC00000FF80FF
FFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF80FFFFC001FFFF8003FFC00007
FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FF
C00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF
8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC0
0003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF80
01FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00003FF8001FFC00007FF8001FFC000
07FF8001FFC00007FF8001FFC0000FFF8001FFC0001FFF8000FFC0003FFF8000FFE0007FFF8000
7FE000FBFFC0007FF803F3FFFF003FFFFFE3FFFF000FFFFF83FFFF0003FFFE03FFFF00007FF003
FFFF382E7BAD41>I<FFFFFF0003FFFFFFFFFF0003FFFFFFFFFF0003FFFFFFFFFF0003FFFFFFFF
FF0003FFFF01FFE000001F8000FFE000001F0000FFF000003F00007FF000003E00007FF800007E
00003FF800007C00003FFC0000FC00001FFC0000F800001FFC0000F800001FFE0001F800000FFE
0001F000000FFF0003F0000007FF0003E0000007FF8007E0000003FF8007C0000003FFC007C000
0003FFC00FC0000001FFC00F80000001FFE01F80000000FFE01F00000000FFF03F000000007FF0
3E000000007FF87E000000003FF87C000000003FF87C000000003FFCFC000000001FFCF8000000
001FFFF8000000000FFFF0000000000FFFF00000000007FFE00000000007FFE00000000003FFC0
0000000003FFC00000000003FFC00000000001FF800000000001FF800000000000FF0000000000
00FF0000000000007E0000000000007E0000000000007E0000000000007C000000000000FC0000
00000000F8000000000001F8000000000001F00000001F0003F00000003F8003E00000007FC003
E0000000FFE007C0000000FFE00FC0000000FFE00F80000000FFE01F80000000FFE03F00000000
7FC0FE000000007F83FC000000003FFFF8000000001FFFE00000000007FF800000000001FE0000
00000038427EAD3D>121 D E /Ft 8 117 df<0000000F80000000000F80000000001F80000000
003F80000000007F80000000007F8000000000FF8000000001FF8000000003FF8000000003FF80
00000007FF800000000FFF800000001FFF800000001FFF800000003DFF8000000079FF80000000
F9FF80000000F1FF80000001E1FF80000003C1FF8000000781FF8000000F81FF8000000F01FF80
00001E01FF8000003C01FF8000007C01FF8000007801FF800000F001FF800001E001FF800003E0
01FF800003C001FF8000078001FF80000F0001FF80001F0001FF80001E0001FF80003C0001FF80
00780001FF8000F80001FF8000FFFFFFFFFF80FFFFFFFFFF80FFFFFFFFFF80FFFFFFFFFF800000
03FF8000000003FF8000000003FF8000000003FF8000000003FF8000000003FF8000000003FF80
00000003FF8000000003FF8000000003FF80000007FFFFFF800007FFFFFF800007FFFFFF800007
FFFFFF8029387DB730>52 D<0000001FFE0000E0000003FFFFE001E000001FFFFFF803E000007F
FFFFFE07E00001FFFC00FF0FE00007FFC0001FDFE0000FFF000007FFE0003FFC000001FFE0007F
F0000000FFE000FFE00000007FE001FFC00000003FE003FF800000001FE007FF800000001FE007
FF000000000FE00FFE0000000007E00FFE0000000007E01FFC0000000007E01FFC0000000003E0
3FFC0000000003E03FF80000000003E07FF80000000001E07FF80000000001E07FF80000000001
E07FF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF000000000
0000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000000000FFF0000000
000000FFF0000000000000FFF0000000000000FFF0000000000000FFF00000000000007FF00000
000000007FF80000000000007FF80000000001E07FF80000000001E03FF80000000001E03FFC00
00000001E01FFC0000000001E01FFC0000000003E00FFE0000000003C00FFE0000000007C007FF
0000000007C007FF800000000F8003FF800000000F8001FFC00000001F0000FFE00000003E0000
7FF00000007C00003FFC000000F800000FFF000003F0000007FFC0000FE0000001FFFC007FC000
00007FFFFFFF000000001FFFFFFC0000000003FFFFE000000000001FFE0000003B3D7BBB46>67
D<003FFF00000001FFFFE0000007FFFFFC00000FF007FE00001FF801FF80001FF800FFC0001FF8
007FE0001FF8007FE0001FF8007FF0000FF0003FF00007E0003FF00003C0003FF0000000003FF0
000000003FF0000000003FF0000000003FF0000000FFFFF000000FFFFFF000007FF83FF00003FF
803FF00007FE003FF0001FFC003FF0003FF8003FF0007FF0003FF0007FE0003FF000FFE0003FF0
00FFC0003FF000FFC0003FF000FFC0003FF000FFC0007FF000FFC0007FF000FFE000FFF0007FE0
01DFF0003FF003DFFC001FFC0F9FFFE00FFFFE0FFFE001FFF807FFE0003FE001FFE02B267DA52F
>97 D<0003FF8000001FFFF000007FFFFC0000FF83FF0003FE00FF8007FC003F800FF8003FC01F
F8001FE01FF0001FE03FF0001FF03FF0000FF07FE0000FF07FE0000FF87FE0000FF8FFE0000FF8
FFE0000FF8FFFFFFFFF8FFFFFFFFF8FFFFFFFFF8FFE0000000FFE0000000FFE0000000FFE00000
00FFE00000007FE00000007FE00000007FF00000003FF00000783FF00000781FF80000F80FF800
00F007FC0001F003FE0003E001FF000FC000FFC07F80003FFFFE00000FFFFC000001FFC0002526
7DA52C>101 D<00FF0000000000FFFF0000000000FFFF0000000000FFFF0000000000FFFF0000
00000007FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003
FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000
000003FF000000000003FF000000000003FF000000000003FF000000000003FF000000000003FF
000000000003FF003FE0000003FF01FFFC000003FF07FFFE000003FF0F81FF800003FF3C00FF80
0003FF3800FFC00003FF7000FFC00003FFE0007FE00003FFC0007FE00003FFC0007FE00003FF80
007FE00003FF80007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE000
03FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF0000
7FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003
FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007FE00003FF00007F
E00003FF00007FE000FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80FFFFFC1FFFFF80313C
7DBB36>104 D<00FF00FF8000FFFF0FFFF800FFFF3FFFFE00FFFFFE03FF00FFFFF000FFC007FF
E0007FE003FFC0003FF003FF80003FF803FF00001FF803FF00001FFC03FF00000FFC03FF00000F
FE03FF00000FFE03FF00000FFE03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF
000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF000007FF03FF00000F
FE03FF00000FFE03FF00000FFE03FF00000FFC03FF00001FFC03FF00001FF803FF80003FF003FF
C0007FF003FFE000FFE003FFF001FF8003FFFE07FF0003FF3FFFFC0003FF0FFFF00003FF01FF00
0003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF
0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF0000000003FF000000
00FFFFFC000000FFFFFC000000FFFFFC000000FFFFFC00000030377EA536>112
D<00FE01F800FFFE07FF00FFFE1FFF80FFFE3E3FC0FFFE787FE007FE707FE003FEE07FE003FEE0
7FE003FFC07FE003FFC03FC003FF801F8003FF800F0003FF80000003FF80000003FF00000003FF
00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003
FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF00000003FF000000
03FF00000003FF00000003FF000000FFFFFE0000FFFFFE0000FFFFFE0000FFFFFE000023267EA5
28>114 D<000F0000000F0000000F0000000F0000000F0000001F0000001F0000001F0000001F
0000003F0000003F0000007F0000007F000000FF000001FF000003FF000007FF00001FFFFFF0FF
FFFFF0FFFFFFF0FFFFFFF003FF000003FF000003FF000003FF000003FF000003FF000003FF0000
03FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF000003FF00
0003FF000003FF000003FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF003C03FF
003C01FF807801FF807800FFC0F000FFE1F0003FFFE0000FFF800001FE001E377EB626>116
D E /Fu 78 124 df<000FC3E0007FEFF000E07C7801C0F8780380F83007007000070070000700
700007007000070070000700700007007000FFFFFF80FFFFFF8007007000070070000700700007
007000070070000700700007007000070070000700700007007000070070000700700007007000
0700700007007000070070007FE3FF007FE3FF001D20809F1B>11 D<001F80007FC001E0E00381
E00381E00701E0070000070000070000070000070000070000FFFFE0FFFFE00700E00700E00700
E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700
E07FC3FE7FC3FE1720809F19>I<001FE0007FE001E1E00381E00380E00700E00700E00700E007
00E00700E00700E00700E0FFFFE0FFFFE00700E00700E00700E00700E00700E00700E00700E007
00E00700E00700E00700E00700E00700E00700E00700E00700E07FE7FE7FE7FE1720809F19>I<
000F80F800007FE7FE0000E06E060001C0FC0F000380F80F000700F00F00070070000007007000
000700700000070070000007007000000700700000FFFFFFFF00FFFFFFFF000700700700070070
070007007007000700700700070070070007007007000700700700070070070007007007000700
7007000700700700070070070007007007000700700700070070070007007007007FE3FE3FF07F
E3FE3FF02420809F26>I<7038F87CFC7EFC7E7C3E0C060C060C06180C180C381C301860304020
0F0E7E9F17>34 D<70F8FCFC7C0C0C0C181838306040060E7C9F0D>39 D<006000C00180030007
0006000C001C00180038003800300070007000700060006000E000E000E000E000E000E000E000
E000E000E000E000E0006000600070007000700030003800380018001C000C0006000700030001
8000C000600B2E7DA112>I<C0006000300018001C000C0006000700030003800380018001C001
C001C000C000C000E000E000E000E000E000E000E000E000E000E000E000E000C000C001C001C0
01C00180038003800300070006000C001C00180030006000C0000B2E7DA112>I<70F8FCFC7C0C
0C0C181838306040060E7C840D>44 D<FFC0FFC0FFC00A037F8A0F>I<70F8F8F87005057C840D>
I<00030003000700060006000E000C000C001C0018001800380030003000700060006000E000C0
00C001C00180018001800380030003000700060006000E000C000C001C00180018003800300030
00700060006000E000C000C000102D7DA117>I<03F0000FFC001E1E001C0E0038070078078070
0380700380700380F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F003C0F0
03C0F003C0F003C07003807003807003807807803807001C0E001E1E000FFC0003F000121F7E9D
17>I<018003801F80FF80E3800380038003800380038003800380038003800380038003800380
0380038003800380038003800380038003800380FFFEFFFE0F1E7C9D17>I<03F0000FFC00183E
00300F00600F80C00780F007C0F807C0F803C0F803C02003C00007C0000780000780000F00000E
00001C0000380000700000E00001C0000380000300000600C00C00C01800C03001807FFF80FFFF
80FFFF80121E7E9D17>I<03F0000FFC001C1E00300F00780F80780F80780780380F80000F8000
0F00000F00001E00003C0003F80003F000001C00000E00000F000007800007800007C02007C0F8
07C0F807C0F807C0F00780600F80700F003C1E001FFC0003F000121F7E9D17>I<000E00000E00
001E00003E00003E00006E0000EE0000CE00018E00038E00030E00060E000E0E000C0E00180E00
380E00300E00600E00E00E00FFFFF0FFFFF0000E00000E00000E00000E00000E00000E00000E00
00FFE000FFE0141E7F9D17>I<3803003FFF003FFE003FF8003FE0003000003000003000003000
0030000030000031F00037FC003E0E003C07003807803003800003800003C00003C00003C06003
C0F003C0F003C0F00380C00780600700700E003C1E000FF80007E000121F7E9D17>I<007C0001
FE000783000E03800C07801C0780380300380000780000700000700000F3F800F7FE00FC0E00F8
0700F80380F80380F003C0F003C0F003C0F003C0F003C07003C07003C07003803803803807001C
07000E1E0007FC0001F000121F7E9D17>I<6000007FFFC07FFFC07FFF80600180C00300C00600
C00600000C0000180000180000300000600000600000E00000C00001C00001C00001C000038000
038000038000038000078000078000078000078000078000078000078000030000121F7D9D17>
I<03F0000FFC001E1E003807003007007003807003807003807003807807003E07003F0E001FDC
000FF80003F80007FC001E7E00383F00700F80700780E003C0E001C0E001C0E001C0E001C0F001
807003803807001E0E000FFC0003F000121F7E9D17>I<03F00007F8001E1C00380E0038070070
0700700380F00380F00380F003C0F003C0F003C0F003C0F003C07007C07007C0380FC01C0FC01F
FBC007F3C0000380000380000780000700300700780E00780E00701C003078001FF0000FC00012
1F7E9D17>I<70F8F8F8700000000000000000000070F8F8F87005147C930D>I<70F8F8F8700000
000000000000000070F8F8F878181818183030606040051D7C930D>I<7FFFFFE0FFFFFFF00000
000000000000000000000000000000000000000000000000000000000000FFFFFFF07FFFFFE01C
0C7D9023>61 D<0FC03FF070786038E03CF03CF03C603C003C007800F000E001C0018003800300
0300030003000300030003000000000000000000000007000F800F800F8007000E207D9F15>63
D<0003800000038000000380000007C0000007C0000007C000000DE000000DE000000DE0000018
F0000018F0000018F00000307800003078000030780000603C0000603C0000603C0000E01E0000
C01E0000FFFE0001FFFF0001800F0001800F0003800F800300078003000780070007C0070003C0
0F8003C0FFE03FFEFFE03FFE1F207F9F22>65 D<FFFFE000FFFFF80007803E0007801F0007800F
0007800F8007800F8007800F8007800F8007800F8007800F0007801F0007803E0007807C0007FF
F80007FFFC0007801E0007800F0007800F8007800780078007C0078007C0078007C0078007C007
8007C00780078007800F8007801F0007803E00FFFFFC00FFFFF0001A1F7E9E20>I<001FC04000
7FF0C001F839C003C00DC0078007C00F0003C01E0003C03E0001C03C0001C07C0001C07C0000C0
780000C0F80000C0F8000000F8000000F8000000F8000000F8000000F8000000F8000000F80000
00780000C07C0000C07C0000C03C0000C03E0001801E0001800F0003000780030003C00E0001F8
1C00007FF000001FC0001A217D9F21>I<FFFFE000FFFFFC0007803E0007800F00078007800780
03C0078003E0078001E0078001F0078000F0078000F0078000F8078000F8078000F8078000F807
8000F8078000F8078000F8078000F8078000F8078000F0078000F0078001F0078001E0078001E0
078003C00780078007800F0007803E00FFFFFC00FFFFE0001D1F7E9E23>I<FFFFFF00FFFFFF00
07801F000780070007800300078003000780038007800180078001800780C1800780C1800780C0
000780C0000781C00007FFC00007FFC0000781C0000780C0000780C0000780C0600780C0600780
0060078000C0078000C0078000C0078001C0078001C0078003C007800F80FFFFFF80FFFFFF801B
1F7E9E1F>I<FFFFFF00FFFFFF0007801F00078007000780030007800300078003800780018007
8001800780C1800780C1800780C0000780C0000781C00007FFC00007FFC0000781C0000780C000
0780C0000780C0000780C000078000000780000007800000078000000780000007800000078000
0007800000FFFE0000FFFE0000191F7E9E1E>I<000FC020007FF86001F81CE003E006E0078003
E00F0001E01E0001E01E0000E03C0000E07C0000E07C00006078000060F8000060F8000000F800
0000F8000000F8000000F8000000F8000000F8007FFCF8007FFC780001E07C0001E07C0001E03C
0001E01E0001E01E0001E00F0001E0078003E003E003E001F80E60007FFC20000FE0001E217D9F
24>I<FFF8FFF8FFF8FFF807800F0007800F0007800F0007800F0007800F0007800F0007800F00
07800F0007800F0007800F0007800F0007800F0007FFFF0007FFFF0007800F0007800F0007800F
0007800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F0007800F000780
0F00FFF8FFF8FFF8FFF81D1F7E9E22>I<FFFCFFFC078007800780078007800780078007800780
078007800780078007800780078007800780078007800780078007800780078007800780FFFCFF
FC0E1F7F9E10>I<FFFC1FFCFFFC1FFC078007C0078007000780060007800C0007801800078030
00078070000780E0000781C000078380000787000007878000078F8000079BC00007B1E00007E1
E00007C0F0000780F0000780780007803C0007803C0007801E0007801E0007800F000780078007
800780078007C0FFFC3FFCFFFC3FFC1E1F7E9E23>75 D<FFFE00FFFE0007800007800007800007
800007800007800007800007800007800007800007800007800007800007800007800007800007
800007800607800607800607800607800E07800E07800C07801C07803C0780FCFFFFFCFFFFFC17
1F7E9E1C>I<FF80001FF8FFC0003FF807C0003F0007C0003F0006E0006F0006E0006F0006E000
6F00067000CF00067000CF00067000CF000638018F000638018F000638018F00061C030F00061C
030F00061C030F00060E060F00060E060F00060E060F0006070C0F0006070C0F0006070C0F0006
03980F000603980F000603980F000601F00F000601F00F000601F00F000F00E00F00FFF0E1FFF8
FFF0E1FFF8251F7E9E2A>I<FF807FF8FFC07FF807C0078007E0030007E0030006F0030006F003
0006780300063C0300063C0300061E0300061E0300060F0300060F0300060783000607C3000603
C3000601E3000601E3000600F3000600F30006007B0006007B0006003F0006001F0006001F0006
000F0006000F000F000700FFF00700FFF003001D1F7E9E22>I<001F800000FFF00001E0780007
C03E000F801F000F000F001E0007803C0003C03C0003C07C0003E07C0003E0780001E0F80001F0
F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0F80001F0780001E07C0003
E07C0003E03C0003C03E0007C01E0007800F000F000F801F0007C03E0001F0F80000FFF000001F
80001C217D9F23>I<FFFFE000FFFFF80007807C0007801E0007801F0007800F0007800F800780
0F8007800F8007800F8007800F8007800F0007801F0007801E0007807C0007FFF80007FFE00007
800000078000000780000007800000078000000780000007800000078000000780000007800000
0780000007800000FFFC0000FFFC0000191F7E9E1F>I<FFFF8000FFFFF0000780F80007803C00
07803E0007801E0007801F0007801F0007801F0007801F0007801E0007803E0007803C000780F8
0007FFF00007FF80000781C0000780E00007807000078070000780780007807800078078000780
7C0007807C0007807C0007807C0C07807E0C07803E0CFFFC1F18FFFC0FF0000003E01E207E9E21
>82 D<07E0800FF9801C1F80300F80700780600380E00380E00180E00180E00180F00000F00000
7800007F00003FF0001FFC000FFE0003FF00001F800007800003C00003C00001C0C001C0C001C0
C001C0C001C0E00180F00380F80700FE0E00CFFC0081F80012217D9F19>I<7FFFFFE07FFFFFE0
780F01E0700F00E0600F0060600F0060E00F0070C00F0030C00F0030C00F0030C00F0030000F00
00000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F0000000F
0000000F0000000F0000000F0000000F0000000F0000000F0000000F000007FFFE0007FFFE001C
1F7E9E21>I<FFFC7FF8FFFC7FF807800780078003000780030007800300078003000780030007
800300078003000780030007800300078003000780030007800300078003000780030007800300
0780030007800300078003000780030007800300078003000380020003C0060003C0060001C00C
0000E0180000783000003FE000000FC0001D207E9E22>I<FFF007FEFFF007FE0F8001F00F0000
E0078000C0078000C007C001C003C0018003C0018003E0038001E0030001E0030000F0060000F0
060000F0060000780C0000780C0000780C00003C1800003C1800003C1800001E3000001E300000
1F7000000F6000000F6000000FE0000007C0000007C0000003800000038000000380001F207F9E
22>I<FFF07FF83FF0FFF07FF83FF00F0007800F800F00078003000F000780030007800FC00600
07800FC0060007800FC00600078019E00E0003C019E00C0003C019E00C0003C039E00C0001E030
F0180001E030F0180001E030F0180000F06078300000F06078300000F06078300000F0E07C3000
0078C03C60000078C03C60000078C03C6000003D801EC000003D801EC000003D801EC000003F00
0FC000001F000F8000001F000F8000001F000F8000000E00070000000E00070000000E00070000
2C207F9E2F>I<7FF83FF87FF83FF807C00F8003C0060001E00E0001F00C0000F0180000783800
007C3000003C7000003E6000001EC000000FC000000F8000000780000007C0000007E000000DE0
00001DF0000018F8000038780000307C0000603C0000E01E0000C01F0001800F00038007800300
07C00F8007C0FFE01FFEFFE01FFE1F1F7F9E22>I<FEFEC0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0
C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0C0FEFE072D7CA10D>91
D<0804180C3018703860306030C060C060C060F87CFC7EFC7E7C3E381C0F0E7B9F17>I<FEFE06
060606060606060606060606060606060606060606060606060606060606060606060606060606
06FEFE072D7FA10D>I<1FE0003FF000783800781C00300E00000E00000E00000E0003FE001FFE
003E0E00780E00F80E00F00E30F00E30F00E30F01E30783F703FEFE00F878014147E9317>97
D<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E3F
000EFF800FC3C00F00E00E00F00E00700E00780E00780E00780E00780E00780E00780E00780E00
700E00F00F00F00F01E00FC3C00CFF800C3E0015207F9F19>I<03F80FFC1E1E3C1E380C780070
00F000F000F000F000F000F0007000780038033C031F0E0FFC03F010147E9314>I<000380003F
80003F8000038000038000038000038000038000038000038000038000038003E3800FFB801E0F
803C0780780380780380700380F00380F00380F00380F00380F00380F003807003807003807807
803807801E1F800FFBF803E3F815207E9F19>I<03F0000FFC001E1E003C0F0038070078070070
0380F00380FFFF80FFFF80F00000F00000F000007000007800003801801C03800F070007FE0001
F80011147F9314>I<003C00FE01CF038F03060700070007000700070007000700FFF0FFF00700
0700070007000700070007000700070007000700070007000700070007007FF07FF01020809F0E
>I<0001E003E3F00FFF701C1C703C1E00380E00780F00780F00780F00780F00380E003C1E001C
1C003FF80033E0003000003000003800003FFE001FFF803FFFC07003E07000F0E00070E00070E0
0070F000F07801E03E07C00FFF0003FC00141F7F9417>I<0E0000FE0000FE00000E00000E0000
0E00000E00000E00000E00000E00000E00000E00000E3E000EFF000FC3800F01C00F01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FC
FFE7FC16207F9F19>I<1C001E003E001E001C000000000000000000000000000E007E007E000E
000E000E000E000E000E000E000E000E000E000E000E000E000E000E00FFC0FFC00A1F809E0C>
I<00E001F001F001F000E0000000000000000000000000007007F007F000F00070007000700070
00700070007000700070007000700070007000700070007000700070007000706070F0E0F1C07F
803E000C28829E0E>I<0E0000FE0000FE00000E00000E00000E00000E00000E00000E00000E00
000E00000E00000E1FF00E1FF00E0F800E0E000E0C000E18000E30000E70000EF0000FF8000F38
000E1C000E1E000E0E000E07000E07800E03800E03C0FFCFF8FFCFF815207F9F18>I<0E00FE00
FE000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E
000E000E000E000E000E000E000E000E00FFE0FFE00B20809F0C>I<0E3F03F000FEFF8FF800FF
C1DC1C000F80F80E000F00F00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E00
0E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E000E00E00E
00FFE7FE7FE0FFE7FE7FE023147F9326>I<0E3E00FEFF00FFC3800F01C00F01C00E01C00E01C0
0E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C0FFE7FCFFE7FC
16147F9319>I<01F80007FE001E07803C03C03801C07000E07000E0F000F0F000F0F000F0F000
F0F000F0F000F07000E07801E03801C03C03C01E078007FE0001F80014147F9317>I<0E3F00FE
FF80FFC3C00F01E00E00F00E00F00E00780E00780E00780E00780E00780E00780E00780E00700E
00F00F00F00F01E00FC3C00EFF800E3E000E00000E00000E00000E00000E00000E00000E0000FF
E000FFE000151D7F9319>I<03E1800FF9801E1F803C0780780780780380700380F00380F00380
F00380F00380F00380F003807003807803807807803C0F801E1F800FF38003E380000380000380
000380000380000380000380000380003FF8003FF8151D7E9318>I<0E78FEFCFF9E0F1E0F0C0F
000E000E000E000E000E000E000E000E000E000E000E000E00FFE0FFE00F147F9312>I<1F903F
F07070E030E030E030F00078007F803FE00FF000F0C078C038C038E038E038F070DFE08F800D14
7E9312>I<06000600060006000E000E001E003E00FFF8FFF80E000E000E000E000E000E000E00
0E000E000E000E180E180E180E180E18073007E003C00D1C7F9B12>I<0E01C0FE1FC0FE1FC00E
01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E01C00E03C00E
03C0070FC007FDFC01F1FC16147F9319>I<FF87F8FF87F81E01E00E01C00E01800E0180070300
070300070300038600038600038E0001CC0001CC0001FC0000F80000F800007000007000007000
15147F9318>I<FF9FF3FCFF9FF3FC1C0780F01C0780601C07C0E00E0FC0C00E0CC0C00E0CE0C0
070CE18007186180071871800398730003B0330003B0330003B03F0001F03E0001E01E0001E01E
0000E01C0000C00C001E147F9321>I<7FC7FC7FC7FC0703E007038003830001C70000EE0000EC
00007800003800003C00007C0000EE0001C7000187000303800701C00F01E0FF87FEFF87FE1714
809318>I<FF87F8FF87F81E01E00E01C00E01800E018007030007030007030003860003860003
8E0001CC0001CC0001FC0000F80000F80000700000700000700000600000600000600000C000F0
C000F18000F380007F00003C0000151D7F9318>I<3FFF3FFF380E301C703C6038607060F001E0
01C0038007830F030E031C073C063806701EFFFEFFFE10147F9314>I<FFFFFCFFFFFC1602808C
17>I E /Fv 45 122 df<70F8FCFC7C0C0C0C0C181830306040060F7C840E>44
D<FFE0FFE0FFE00B037F8B10>I<70F8F8F87005057C840E>I<01F00007FC000E0E001C07003803
803803807803C07001C07001C07001C0F001E0F001E0F001E0F001E0F001E0F001E0F001E0F001
E0F001E0F001E0F001E0F001E0F001E0F001E07001C07001C07001C07803C03803803803801C07
000E0E0007FC0001F00013227EA018>48 D<018003800F80FF80F3800380038003800380038003
800380038003800380038003800380038003800380038003800380038003800380038003800380
0380FFFEFFFE0F217CA018>I<03F0000FFC001C1F003007807007807807C07807C07803C03807
C00007C0000780000780000F00000E00003C0003F80003F800001E00000F000007800007C00003
C00003E02003E07003E0F803E0F803E0F003C06003C06007803007801C1F000FFC0003F0001322
7EA018>51 D<000E00000E00001E00001E00003E00003E00006E0000EE0000CE0001CE00018E00
030E00070E00060E000E0E000C0E00180E00180E00300E00700E00600E00E00E00FFFFF8FFFFF8
000E00000E00000E00000E00000E00000E00000E0001FFF001FFF015217FA018>I<1000801C07
801FFF001FFE001FFC001FF00018000018000018000018000018000018000018F8001BFE001F0F
001C07801803801803C00001C00001E00001E00001E00001E06001E0F001E0F001E0F001C0C003
C06003C06003803007001C1E000FFC0003F00013227EA018>I<007E0001FF0003C1800701C00E
03C01C03C01C0180380000380000780000700000702000F1FC00F3FE00F60700FC0380F803C0F8
01C0F001C0F001E0F001E0F001E0F001E0F001E07001E07001E07001E03801C03803C01803801C
07000E0E0007FC0001F00013227EA018>I<01F00007FC000E0F001807803803803001C07001C0
7001C07001C07801C07803803E03803F07001FCE000FFC0003F80003FC000F7F001C3F80380F80
7007C07003C0E001E0E000E0E000E0E000E0E000E0E000E07001C07001C03803801E0F000FFE00
03F00013227EA018>56 D<01F00007FC000E0E001C0700380380780380700380F001C0F001C0F0
01C0F001E0F001E0F001E0F001E0F001E07001E07003E07803E03807E01C0DE00FF9E007F1E000
81C00001C00003C0000380000380300700780700780E00701C003838001FF0000FC00013227EA0
18>I<000180000003C0000003C0000003C0000007E0000007E0000007E000000FF000000CF000
000CF000001CF800001878000018780000383C0000303C0000303C0000601E0000601E0000601E
0000C00F0000C00F0000C00F0001FFFF8001FFFF8001800780030003C0030003C0030003C00600
01E0060001E0060001E00E0000F01F0001F0FFC00FFFFFC00FFF20237EA225>65
D<000FE010003FF83000F81C7001E0067003C003F0078001F00F0000F01E0000F03E0000703C00
00707C0000707C0000307800003078000030F8000030F8000000F8000000F8000000F8000000F8
000000F8000000F800000078000030780000307C0000307C0000303C0000603E0000601E000060
0F0000C0078000C003C0018001E0030000F80E00003FF800000FE0001C247DA223>67
D<FFFFFFC0FFFFFFC007800FC0078003C0078001C0078000C0078000E0078000E0078000600780
60600780606007806060078060000780E0000781E00007FFE00007FFE0000781E0000780E00007
806000078060000780601807806018078000180780003007800030078000300780003007800070
078000F0078001F0078007E0FFFFFFE0FFFFFFE01D227EA121>69 D<FFFFFFC0FFFFFFC007800F
C0078003C0078001C0078000C0078000E0078000E0078000600780006007806060078060600780
6000078060000780E0000781E00007FFE00007FFE0000781E0000780E000078060000780600007
806000078060000780000007800000078000000780000007800000078000000780000007800000
FFFE0000FFFE00001B227EA120>I<FFFCFFFC0780078007800780078007800780078007800780
078007800780078007800780078007800780078007800780078007800780078007800780078007
80FFFCFFFC0E227EA112>73 D<FFC00003FFFFE00007FF07E00007E007E00007E006F0000DE006
F0000DE006F0000DE006780019E006780019E006780019E0063C0031E0063C0031E0063C0031E0
061E0061E0061E0061E0061E0061E0060F00C1E0060F00C1E006078181E006078181E006078181
E00603C301E00603C301E00603C301E00601E601E00601E601E00601E601E00600FC01E00600FC
01E00600FC01E006007801E01F807801E0FFF0783FFFFFF0303FFF28227EA12D>77
D<FF800FFFFFC00FFF07C001F807E0006007F0006006F000600678006006780060063C0060063E
0060061E0060060F0060060F0060060780600607C0600603C0600601E0600601E0600600F06006
0078600600786006003C6006003C6006001E6006000F6006000F60060007E0060007E0060003E0
060001E0060001E01F8000E0FFF000E0FFF0006020227EA125>I<FFFFF000FFFFFC0007803F00
07800F8007800780078003C0078003C0078003E0078003E0078003E0078003E0078003E0078003
C0078003C00780078007800F8007803F0007FFFC0007FFF0000780000007800000078000000780
0000078000000780000007800000078000000780000007800000078000000780000007800000FF
FC0000FFFC00001B227EA121>80 D<FFFFE00000FFFFF8000007803E000007800F000007800780
00078007C000078003E000078003E000078003E000078003E000078003E000078003E000078007
C000078007800007800F000007803E000007FFF8000007FFF00000078078000007803C00000780
1E000007800E000007800F000007800F000007800F000007800F000007800F800007800F800007
800F800007800F818007800FC180078007C180FFFC03E300FFFC01FE000000007C0021237EA124
>82 D<03F0200FFC601C0EE03803E07001E07001E0E000E0E000E0E00060E00060E00060F00000
F000007800007F00003FF0001FFE000FFF0003FF80003FC00007E00001E00000F00000F0000070
C00070C00070C00070C00070E00060E000E0F000C0F801C0EF0380C7FF0081FC0014247DA21B>
I<7FFFFFF87FFFFFF87C0780F8700780386007801860078018E007801CC007800CC007800CC007
800CC007800CC007800C0007800000078000000780000007800000078000000780000007800000
078000000780000007800000078000000780000007800000078000000780000007800000078000
00078000000780000007800003FFFF0003FFFF001E227EA123>I<0FE0001FF8003C1C003C0E00
180700000700000700000F0003FF000FFF003F07007C0700780700F00700F00718F00718F00F18
780F187C3FB83FF3F00FC3C015157E9418>97 D<0E0000FE0000FE00001E00000E00000E00000E
00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FE00FC0F00F00780E00380E
003C0E001C0E001E0E001E0E001E0E001E0E001E0E001E0E001E0E001C0E003C0F00380F80700F
C1F00C7FC00C1F0017237FA21B>I<01FE0007FF000F07801C0780380300780000700000F00000
F00000F00000F00000F00000F00000F000007800007800C03C00C01E01800F030007FE0001F800
12157E9416>I<0000E0000FE0000FE00001E00000E00000E00000E00000E00000E00000E00000
E00000E00000E00000E003F0E007FEE01F07E03C01E03800E07800E07000E0F000E0F000E0F000
E0F000E0F000E0F000E0F000E07000E07800E03801E03C03E01E0EF00FFCFE03F0FE17237EA21B
>I<01FC0007FF000F07801C03C03801C07801E07000E0FFFFE0FFFFE0F00000F00000F00000F0
0000F000007800007800603C00601E00C00F838007FF0000FC0013157F9416>I<003C00FE01CF
038F038F070007000700070007000700070007000700FFF8FFF807000700070007000700070007
0007000700070007000700070007000700070007007FF87FF8102380A20F>I<0000F001F1F807
FFB80F1F381E0F001C07003C07803C07803C07803C07803C07801C07001E0F000F1E001FFC0019
F0001800001800001C00001FFF000FFFC01FFFE03801F0700070E00038E00038E00038E0003870
00707800F01E03C00FFF8001FC0015217F9518>I<0E0000FE0000FE00001E00000E00000E0000
0E00000E00000E00000E00000E00000E00000E00000E00000E1F800E7FC00FC1E00F80F00F0070
0E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070
0E0070FFE7FFFFE7FF18237FA21B>I<1C001E003E001E001C0000000000000000000000000000
0000000E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E00
0E00FFC0FFC00A227FA10E>I<01C003E003E003E001C000000000000000000000000000000000
01E00FE00FE001E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000
E000E000E000E000E000E000E060C0F1C0F1807F003E000B2C82A10F>I<0E0000FE0000FE0000
1E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E00000E0FFC0E0FFC
0E07E00E03800E07000E0E000E18000E30000E78000EF8000F9C000F1E000E0E000E07000E0780
0E03C00E01C00E01E00E01F0FFE3FEFFE3FE17237FA21A>I<0E00FE00FE001E000E000E000E00
0E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E000E
000E000E000E000E000E000E00FFE0FFE00B237FA20E>I<0E1FC07F00FE7FE1FF80FEC0F303C0
1F807E01E00F003C00E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800
E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E003800E00E0038
00E0FFE3FF8FFEFFE3FF8FFE27157F942A>I<0E1F80FE7FC0FFC1E01F80F00F00700E00700E00
700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E00700E0070FFE7
FFFFE7FF18157F941B>I<01FC0007FF000F07801C01C03800E07800F0700070F00078F00078F0
0078F00078F00078F00078F000787000707800F03800E01C01C00F078007FF0001FC0015157F94
18>I<0E1F80FE7FE0FFC1F00F00780E00780E003C0E003C0E001E0E001E0E001E0E001E0E001E
0E001E0E001E0E003C0E003C0F00780F80700FC1F00E7FC00E1F000E00000E00000E00000E0000
0E00000E00000E00000E0000FFE000FFE000171F7F941B>I<0E3CFEFEFFCF1F8F0F060F000E00
0E000E000E000E000E000E000E000E000E000E000E000E00FFF0FFF010157F9413>114
D<0F883FF87078E038E018E018E018F0007F003FE01FF001F8003CC01CC01CE01CE01CF018F878
DFF08FC00E157E9413>I<060006000600060006000E000E000E001E003E00FFF8FFF80E000E00
0E000E000E000E000E000E000E000E000E0C0E0C0E0C0E0C0E0C0E08071803F001E00E1F7F9E13
>I<0E0070FE07F0FE07F01E00F00E00700E00700E00700E00700E00700E00700E00700E00700E
00700E00700E00700E00700E00F00E01F007037803FE7F01F87F18157F941B>I<FFC3FEFFC3FE
1E00F80E00600E00600700C00700C00700C003818003818003C38001C30001C30000E60000E600
00E600007C00007C00007C0000380000380017157F941A>I<FF8FF9FFFF8FF9FF1E01C07C1C03
C0380E03E0300E03E0300E0660300706706007067060070E7060038C30C0038C38C0039C38C001
D8198001D81D8001D81D8001F00F8000F00F0000F00F0000E007000060060020157F9423>I<FF
C3FEFFC3FE1E00F80E00600E00600700C00700C00700C003818003818003C38001C30001C30000
E60000E60000E600007C00007C00007C00003800003800003000003000007000006000006000F0
C000F1C000F380007F00003E0000171F7F941A>121 D E /Fw 20 118 df<FFFF80FFFF80FFFF
8011037F9016>45 D<FFFFFFE00000FFFFFFFC000007E0007F000003E0000F800003E00007C000
03E00001E00003E00000F00003E00000780003E000007C0003E000003E0003E000003E0003E000
001F0003E000001F0003E000000F8003E000000F8003E000000FC003E000000FC003E0000007C0
03E0000007C003E0000007E003E0000007E003E0000007E003E0000007E003E0000007E003E000
0007E003E0000007E003E0000007E003E0000007E003E0000007E003E0000007E003E0000007C0
03E0000007C003E0000007C003E000000FC003E000000F8003E000000F8003E000000F8003E000
001F0003E000001F0003E000003E0003E000007C0003E00000780003E00000F80003E00001F000
03E00007E00003E0000F800007E0007F0000FFFFFFFC0000FFFFFFE000002B317CB033>68
D<FFFF80FFFF8007F00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E0
0003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E0
0003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E00003E0
0003E00003E00003E00003E00003E00003E00003E00007F000FFFF80FFFF8011317DB017>73
D<FFF00000007FF8FFF8000000FFF807F8000000FF0003F8000000FE00037C000001BE00037C00
0001BE00037C000001BE00033E0000033E00033E0000033E00033E0000033E00031F0000063E00
031F0000063E00031F0000063E00030F80000C3E00030F80000C3E00030F80000C3E000307C000
183E000307C000183E000307C000183E000303E000303E000303E000303E000303E000303E0003
01F000603E000301F000603E000301F000C03E000300F800C03E000300F800C03E0003007C0180
3E0003007C01803E0003007C01803E0003003E03003E0003003E03003E0003003E03003E000300
1F06003E0003001F06003E0003001F06003E0003000F8C003E0003000F8C003E0003000F8C003E
00030007D8003E00030007D8003E00030007D8003E00030003F0003E00030003F0003E00030003
F0003E00078001E0003E000FC001E0007F00FFFC01E00FFFF8FFFC00C00FFFF835317CB03D>77
D<FFFFFFC000FFFFFFF80007E000FE0003E0001F0003E0000F8003E00007C003E00003E003E000
03F003E00001F003E00001F003E00001F803E00001F803E00001F803E00001F803E00001F803E0
0001F803E00001F003E00001F003E00003F003E00003E003E00007C003E0000F8003E0001F0003
E000FE0003FFFFF80003FFFFC00003E000000003E000000003E000000003E000000003E0000000
03E000000003E000000003E000000003E000000003E000000003E000000003E000000003E00000
0003E000000003E000000003E000000003E000000003E000000003E000000003E000000007F000
0000FFFF800000FFFF80000025317CB02D>80 D<007F802001FFE06007C0F8600F001CE01E000F
E03C0007E0380003E0780001E0700001E0700000E0F00000E0F00000E0F0000060F0000060F000
0060F8000060F80000007C0000007E0000003F0000003FC000001FF800000FFF800007FFF80003
FFFC0000FFFF00000FFF800000FFC000001FE0000007E0000003F0000001F0000000F0000000F8
000000F8C0000078C0000078C0000078C0000078C0000078E0000078E0000070F00000F0F00000
E0F80000E0FC0001C0FE000380E7800700C1F01E00C07FFC00800FF0001D337CB125>83
D<01FE00000FFFC0001C03E0003C00F0003E0078003E003C003E001C0008001E0000001E000000
1E0000001E0000001E000001FE00003FFE0001FF1E0007F01E000F801E001F001E003E001E007C
001E007C001E00F8001E0CF8001E0CF8001E0CF8003E0CF8003E0C7C007E0C3E00DF1C1F038FB8
0FFF07F003F803C01E1F7D9E21>97 D<003FC000FFF803E01C07801E0F003E1E003E1E003E3C00
087C00007C0000780000F80000F80000F80000F80000F80000F80000F80000F80000F800007C00
007C00007C00003E00031E00031F00060F800E07C01C03F03800FFE0003F80181F7D9E1D>99
D<000001E000003FE000003FE0000003E0000001E0000001E0000001E0000001E0000001E00000
01E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E0000001E000
3F81E000FFE1E003E079E007800DE00F0007E01E0003E01E0001E03C0001E07C0001E07C0001E0
780001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001E0F80001
E0780001E07C0001E07C0001E03C0001E03E0003E01E0007E00F000FE007801DE003E071F001FF
E1FF003F01FF20327DB125>I<003F800000FFE00003E0F80007803C000F001E001E001E001E00
0F003C000F007C000F007C000F8078000780F8000780FFFFFF80FFFFFF80F8000000F8000000F8
000000F8000000F8000000F8000000780000007C0000003C0000003C0001801E0001800F000300
0F80060007C00E0001F03C0000FFF000001FC000191F7E9E1D>I<0003E0001FF8003C3800707C
00F07C01E07C01E03803C00003C00003C00003C00003C00003C00003C00003C00003C00003C000
03C00003C000FFFFC0FFFFC003C00003C00003C00003C00003C00003C00003C00003C00003C000
03C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C000
03C00003C00003C00003C00007E0007FFF007FFF0016327FB114>I<000000F0007F03F801FFCF
1C03C1FC1C0780F8180F0078001F007C001E003C003E003E003E003E003E003E003E003E003E00
3E003E003E003E003E001E003C001F007C000F0078000780F0000FC1E0000DFFC0001C7F000018
000000180000001C0000001C0000001E0000000FFFF8000FFFFF0007FFFF800FFFFFC01E0007E0
380001F07000007070000078E0000038E0000038E0000038E0000038E000003870000070780000
F03C0001E01E0003C007C01F0001FFFC00003FE0001E2F7E9F21>I<07000F801F801F800F8007
00000000000000000000000000000000000000000000000780FF80FF800F800780078007800780
078007800780078007800780078007800780078007800780078007800780078007800780078007
800FC0FFF8FFF80D307EAF12>105 D<0781FE003FC000FF87FF80FFF000FF8E07C1C0F8000F98
03E3007C0007B001E6003C0007E000FC001E0007C000F8001E0007C000F8001E0007C000F8001E
00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000
F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00
078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0001E00078000F0
001E00078000F0001E00078000F0001E000FC001F8003F00FFFC1FFF83FFF0FFFC1FFF83FFF034
1F7E9E38>109 D<0781FE0000FF87FF8000FF8E07C0000F9803E00007B001E00007E000F00007
C000F00007C000F00007C000F000078000F000078000F000078000F000078000F000078000F000
078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0
00078000F000078000F000078000F000078000F000078000F000078000F0000FC001F800FFFC1F
FF80FFFC1FFF80211F7E9E25>I<001FC00000FFF80001E03C0007800F000F0007801E0003C01E
0003C03C0001E03C0001E0780000F0780000F0780000F0F80000F8F80000F8F80000F8F80000F8
F80000F8F80000F8F80000F8F80000F8780000F07C0001F03C0001E03C0001E01E0003C01E0003
C00F00078007C01F0001F07C0000FFF800001FC0001D1F7E9E21>I<0783E0FF8FF8FF9C7C0FB0
7C07F07C07E03807C00007C00007C0000780000780000780000780000780000780000780000780
000780000780000780000780000780000780000780000780000780000780000780000FC000FFFE
00FFFE00161F7E9E19>114 D<01FC100FFF301E03F03800F0700070E00070E00030E00030E000
30F00030F800007E00003FF0001FFF000FFF8003FFE0003FF00003F8000078C0003CC0003CC000
1CE0001CE0001CF0001CF00018F80038FC0070EF01E0C3FFC080FE00161F7E9E1A>I<00C00000
C00000C00000C00000C00001C00001C00001C00003C00003C00007C0000FC0001FC000FFFFE0FF
FFE003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003C00003
C00003C00003C00003C00003C03003C03003C03003C03003C03003C03003C03003C03001E06001
E06000F0C0007F80001F00142C7FAB19>I<078000F000FF801FF000FF801FF0000F8001F00007
8000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F000
078000F000078000F000078000F000078000F000078000F000078000F000078000F000078000F0
00078000F000078000F000078000F000078001F000078001F000078003F000038007F00003C00E
F00001F03CF80000FFF0FF80003FC0FF80211F7E9E25>I E /Fx 5 85 df<0000000060000000
0000E00000000000E00000000001E00000000001F00000000003F00000000003F00000000007F0
000000000FF0000000000FF0000000001BF0000000001BF00000000033F00000000033F0000000
0063F00000000063F000000000C3F000000001C3F00000000183F00000000303F00000000303F0
0000000603F80000000603F80000000C01F80000000C01F80000001801F80000003801F8000000
3001F80000006001F80000006001F8000000C001F8000000C001F80000018001F8000001FFFFF8
000003FFFFF80000070001F80000060001F800000C0001F800000C0001F80000180001FC000018
0001FC0000300000FC0000300000FC0000600000FC0000E00000FC0000C00000FC0001C00000FC
0003C00000FC000FE00003FC00FFFC003FFFE0FFF8003FFFE02B337CB234>65
D<000FFFFFFC00000FFFFFFF8000007F000FC000007E0003E000007E0001F000007E0000F80000
7E00007C0000FC00007C0000FC00003E0000FC00003E0000FC00003E0001F800001E0001F80000
1F0001F800001F0001F800001F0003F000001F0003F000001F0003F000001F0003F000001F0007
E000003F0007E000003F0007E000003F0007E000003F000FC000003E000FC000007E000FC00000
7E000FC000007E001F8000007C001F800000FC001F800000FC001F800000F8003F000001F8003F
000001F0003F000003F0003F000003E0007E000007E0007E000007C0007E00000F80007E00000F
0000FC00001F0000FC00003E0000FC00007C0000FC0000F80001F80001F00001F80007C00001F8
000F800003F8007E0000FFFFFFF80000FFFFFFC0000030317BB035>68 D<000FFFFFFFF8000FFF
FFFFF800007F0003F800007E0000F800007E00007800007E00007800007E0000300000FC000030
0000FC0000300000FC0000300000FC0000300001F80000300001F80000300001F80000300001F8
0000600003F00180600003F00180000003F00180000003F00180000007E00300000007E0030000
0007E00700000007E01F0000000FFFFE0000000FFFFE0000000FC01E0000000FC00E0000001F80
0C0000001F800C0000001F800C0000001F800C0000003F00180000003F00180000003F00000000
003F00000000007E00000000007E00000000007E00000000007E0000000000FC0000000000FC00
00000000FC0000000000FC0000000001F80000000001F80000000001F80000000003F800000000
FFFFF0000000FFFFF00000002D317BB02E>70 D<000FFFFFE000000FFFFFFC0000007F003F0000
007E000F8000007E0007C000007E0003E000007E0003F00000FC0001F00000FC0001F00000FC00
01F00000FC0001F00001F80003F00001F80003F00001F80003F00001F80003E00003F00007E000
03F00007C00003F0000FC00003F0001F800007E0003F000007E0007C000007E000F8000007E007
E000000FFFFF8000000FFFFF0000000FC00FC000000FC007E000001F8003F000001F8001F00000
1F8001F800001F8001F800003F0001F800003F0001F800003F0001F800003F0001F800007E0003
F000007E0003F000007E0003F000007E0003F00000FC0007E00000FC0007E00000FC0007E00000
FC0007E01801F80007E03001F80007E03001F80007E06003F80003E060FFFFE001F1C0FFFFC000
FF80000000003E002D327BB033>82 D<07FFFFFFFFF007FFFFFFFFF00FE007F007F00F8007E001
F00E0007E000E00C0007E000E01C0007E000E018000FC000E018000FC000E030000FC000C03000
0FC000C030001F8000C060001F8000C060001F8000C060001F8000C0C0003F00018000003F0000
0000003F00000000003F00000000007E00000000007E00000000007E00000000007E0000000000
FC0000000000FC0000000000FC0000000000FC0000000001F80000000001F80000000001F80000
000001F80000000003F00000000003F00000000003F00000000003F00000000007E00000000007
E00000000007E00000000007E0000000000FC0000000000FC0000000000FC0000000000FC00000
00001F80000000001F80000000001F80000000003F800000007FFFFF8000007FFFFF8000002C31
74B032>84 D E end
%%EndProlog
%%BeginSetup
%%Feature: *Resolution 300
TeXDict begin
%%EndSetup
%%Page: 0 1
bop 799 911 a Fx(D)25 b(R)g(A)g(F)g(T)225 1002 y Fw(Do)r(cumen)n(t)20
b(for)i(a)f(Standard)g(Message-P)n(assing)f(In)n(terface)621
1196 y Fv(Message)c(P)o(assing)h(In)o(terface)e(F)l(orum)844
1323 y(Ma)o(y)h(9,)g(1996)77 1381 y(This)h(w)o(ork)f(w)o(as)h(supp)q(orted)g
(in)f(part)h(b)o(y)e(ARP)l(A)h(and)h(NSF)e(under)h(gran)o(t)h(ASC-9310330,)i
(the)192 1439 y(National)d(Science)f(F)l(oundation)i(Science)e(and)i(T)l(ec)o
(hnology)f(Cen)o(ter)f(Co)q(op)q(erativ)o(e)76 1497 y(Agreemen)o(t)e(No.)22
b(CCR-8809615,)d(and)e(b)o(y)e(the)h(Commission)e(of)j(the)f(Europ)q(ean)i
(Comm)o(unit)n(y)654 1555 y(through)f(Esprit)f(pro)s(ject)g(P6643.)p
eop
%%Page: 1 2
bop 166 49 a Fu(This)20 b(is)h(the)f(result)g(of)f(a)h(LaT)l(eX)g(run)g(of)g
(a)f(draft)g(of)h(a)f(single)j(c)o(hapter)d(of)h(the)g(MPIF)f(Final)75
106 y(Rep)q(ort)d(do)q(cumen)o(t.)969 2828 y(i)p eop
%%Page: 1 3
bop 75 361 a Ft(Chapter)31 b(4)75 568 y Fs(One)39 b(Sided)e(Comm)m
(unications)75 809 y Fr(4.1)59 b(Intro)r(duction)75 910 y Fu(Remote)11
b(Memory)f(Access)i(\(RMA\))e(extends)i(the)f(comm)o(unication)g(mec)o
(hanisms)h(of)e(MPI)h(b)o(y)g(allo)o(wing)75 967 y(transfer)i(of)g(data)f
(from)h(the)h(memory)f(of)g(one)g(pro)q(cess)h(to)f(the)g(memory)g(of)g
(another)g(pro)q(cess)h(to)f(o)q(ccur)75 1023 y(with)19 b(the)g(explicit)h
(in)o(v)o(olv)o(emen)o(t)f(of)f(one)h(of)f(the)h(t)o(w)o(o)e(pro)q(cesses)i
(only:)27 b(either)19 b(the)g(source)g(pro)q(cess)75 1080 y(\(put\))c(or)f
(the)i(destination)g(pro)q(cess)f(\(get\).)166 1136 y(This)h(mo)q(de)f(of)g
(comm)o(unication)h(facilitates)g(the)f(co)q(ding)h(of)f(some)g(applications)
i(with)e(dynami-)75 1193 y(cally)f(c)o(hanging)f(data)e(access)i(patterns)f
(but)h(where)f(the)h(data)f(distribution)i(is)f(\014xed)g(or)f(slo)o(wly)h(c)
o(hang-)75 1249 y(ing.)23 b(Eac)o(h)16 b(pro)q(cess)g(can)g(compute)g(what)g
(data)f(it)h(needs)h(to)f(access)g(or)f(up)q(date)i(at)e(other)h(pro)q
(cesses.)75 1306 y(Ho)o(w)o(ev)o(er,)j(pro)q(cesses)g(ma)o(y)g(not)f(kno)o(w)
h(whic)o(h)h(data)e(in)i(their)g(o)o(wn)e(memory)h(need)h(b)q(e)g(accessed)f
(or)75 1362 y(up)q(dated)g(b)o(y)g(remote)f(pro)q(cesses,)h(nor)f(do)g(they)h
(kno)o(w)f(the)g(iden)o(tit)o(y)i(of)e(these)g(pro)q(cesses.)30
b(Regular)75 1418 y(send/receiv)o(e)19 b(comm)o(unication)g(requires)f(matc)o
(hing)g(op)q(erations)g(b)o(y)g(sender)g(and)g(receiv)o(er.)29
b(In)18 b(our)75 1475 y(situation,)g(only)f(one)h(op)q(eration)f(out)g(of)g
(eac)o(h)g(matc)o(hing)g(pair)h(is)g(sp)q(eci\014ed.)28 b(In)17
b(order)g(to)g(issue)h(the)75 1531 y(matc)o(hing)i(op)q(erations,)h(one)g
(needs)f(to)g(compute)g(the)g(in)o(v)o(erse)h(of)f(the)g(mapping)h(sp)q
(eci\014ed)h(b)o(y)e(the)75 1588 y(transfer)g(requests.)37
b(This)21 b(requires)h(all)g(pro)q(cesses)f(to)f(participate)i(in)g(a)e(time)
i(consuming)f(global)75 1644 y(computation.)29 b(The)18 b(use)h(of)e(put)i
(or)e(get)h(calls)h(a)o(v)o(oids)f(the)g(need)i(for)d(the)h(global)h
(computation.)29 b(A)75 1701 y(generic)17 b(example)h(of)e(this)h(nature)f
(is)h(the)g(execution)g(of)f(an)h(assignmen)o(t)f(of)g(the)h(form)f
Fq(A)23 b(=)h(B\(map\))p Fu(,)75 1757 y(where)15 b Fq(map)g
Fu(is)h(a)f(p)q(erm)o(utation)g(v)o(ector,)f(and)h Fq(A,)24
b(B)15 b Fu(and)g Fq(map)g Fu(are)g(distributed)h(in)g(the)f(same)g(manner.)
166 1814 y(RMA)k(will)h(b)q(e)g(supp)q(orted)f(on)g(distributed)h(memory)e
(systems)g(b)o(y)h(an)f(\\RMA)h(agen)o(t")f(at)g(the)75 1870
y(target)e(no)q(de)j(that)e(accepts)h(RMA)g(requests)f(and)h(p)q(erforms)f
(the)h(required)h(read)f(or)f(write)h(accesses)75 1927 y(in)d(the)g(memory)f
(of)g(the)h(target)e(pro)q(cess.)20 b(A)15 b(p)q(ortable)g(implemen)o(tation)
h(ma)o(y)e(use)h(an)f(async)o(hronous)75 1983 y(receiv)o(e)k(handler)g(to)e
(implemen)o(t)i(this)f(RMA)g(agen)o(t.)24 b(Systems)16 b(with)h(dedicated)h
(put/get)f(hardw)o(are)75 2039 y(\(Cra)o(y)h(T3D\))g(should)i(b)q(e)f(to)g
(tak)o(e)f(adv)m(an)o(tage)h(of)f(that)h(hardw)o(are,)g(at)f(least)h(for)g
(simple)h(transfers.)75 2096 y(More)f(complex)i(transfers)e(ma)o(y)g(require)
h(a)g(soft)o(w)o(are)e(agen)o(t)h(at)g(the)g(target)g(no)q(de.)34
b(Systems)20 b(with)75 2152 y(comm)o(unication)h(copro)q(cessors)f(\(P)o
(aragon,)f(Meik)o(o\))h(should)h(b)q(e)g(able)f(to)g(tak)o(e)f(adv)m(an)o
(tage)h(of)g(that)75 2209 y(copro)q(cessor)13 b(in)h(order)f(to)g(run)h(the)f
(RMA)g(agen)o(t)g(without)g(in)o(terfering)h(with)g(the)f(application)i(pro)q
(ces-)75 2265 y(sor)i(at)f(the)i(target)e(no)q(de,)i(at)f(least)g(for)g
(common)g(transfers.)25 b(On)18 b(shared)f(memory)g(systems,)g(if)h(the)75
2322 y(caller)h(can)f(directly)h(access)f(the)g(memory)f(of)h(the)g(target)e
(pro)q(cess,)i(then)h(it)f(should)h(b)q(e)f(p)q(ossible)i(to)75
2378 y(implemen)o(t)f(RMA)g(without)f(an)g(RMA)g(agen)o(t:)25
b(the)18 b(caller)h(pro)q(cess)f(can)g(directly)i(cop)o(y)e(data)f(to)g(or)75
2435 y(from)d(the)i(memory)e(of)h(the)g(target)f(pro)q(cess.)166
2491 y(W)l(e)g(shall)h(denote)f(b)o(y)g Fp(origin)h Fu(the)f(pro)q(cess)g
(that)g(p)q(erforms)f(the)i(call,)g(and)f(b)o(y)g Fp(target)g
Fu(the)g(other)75 2548 y(in)o(v)o(olv)o(ed)i(pro)q(cess.)k(Th)o(us,)14
b(in)i(a)f(put)g(op)q(eration,)g(source=origin)h(and)f(destination=target;)g
(in)h(a)e(get)75 2604 y(op)q(eration,)h(source=target)f(and)i
(destination=origin.)964 2828 y(1)p eop
%%Page: 2 4
bop 75 -100 a Fu(2)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)75 49 y Fr(4.2)59 b(Initialization)75 151 y Fu(In)12
b(some)f(systems,)h(RMA)f(op)q(erations)h(will)h(run)f(faster)e(when)i
(accessing)g(sp)q(ecially)i(allo)q(cated)f(memory)75 207 y(\(e.g.,)19
b(memory)g(that)g(is)h(shared)g(b)o(y)g(the)f(other)h(pro)q(cesses)g(in)g
(the)g(comm)o(unicating)g(group,)g(on)g(an)75 264 y(SMP\).)14
b(MPI)g(pro)o(vides)h(a)f(mec)o(hanism)h(for)f(allo)q(cating)h(and)g(freeing)
g(suc)o(h)g(sp)q(ecial)h(memory)l(.)j(The)c(use)75 320 y(of)g(suc)o(h)g
(memory)g(for)f(RMA)i(is)g(not)e(mandatory)l(.)75 471 y Fn(MPI)p
160 471 14 2 v 16 w(RMA)p 275 471 V 17 w(MALLOC\(size,)g(comm)m(,)e(base\))
117 548 y Fm(IN)155 b Fn(size)519 b Fm(size)11 b(of)e(memory)e(segmen)o(t)i
(in)g(b)o(ytes)h(\(non)g(negativ)o(e)f(in)o(teger\))117 620
y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)13 b(for)i(group)h(where)g(RMA)f
(comm)o(unication)905 677 y(is)f(exp)q(ected)i(to)e(o)q(ccur)h(\(handle\))117
749 y(OUT)108 b Fn(base)505 b Fm(initial)12 b(address)j(of)f(memory)d(allo)q
(cated)i(\(in)o(teger\))75 873 y Fq(int)23 b(MPI)p 245 873
15 2 v 17 w(Rma)p 334 873 V 17 w(malloc\(int)f(size,)i(MPI)p
829 873 V 16 w(Comm)g(comm,)f(void)g(*base\))75 960 y(MPI)p
150 960 V 17 w(RMA)p 239 960 V 17 w(MALLOC\()f(SIZE,)i(COMM,)f(BASE,)g
(IERROR\))170 1016 y(INTEGER)g(SIZE,)h(COMM,)f(IERROR)170 1073
y(MPI)p 245 1073 V 17 w(AINT)h(BASE)75 1254 y Fn(MPI)p 160
1254 14 2 v 16 w(RMA)p 275 1254 V 17 w(FREE\(base\))117 1331
y Fm(IN)155 b Fn(base)505 b Fm(initial)12 b(address)j(of)f(memory)d(segmen)o
(t)i(allo)q(cated)h(b)o(y)905 1387 y Fl(MPI)p 982 1387 13 2
v 15 w(RMA)p 1088 1387 V 14 w(MALLOC)75 1512 y Fq(int)23 b(MPI)p
245 1512 15 2 v 17 w(Rma)p 334 1512 V 17 w(free\(void)g(*base\))75
1598 y(MPI)p 150 1598 V 17 w(RMA)p 239 1598 V 17 w(FREE\()g(BASE,)g(IERROR\))
170 1655 y(MPI)p 245 1655 V 17 w(AINT)h(BASE)166 1741 y Fu(The)13
b(F)l(ortran)f(t)o(yp)q(e)h Fl(MPI)p 597 1741 13 2 v 15 w(AINT)f
Fu(in)i(the)f(F)l(ortran)f(binding)j(is)f(used)g(to)e(indicate)j(a)d(\\p)q
(oin)o(ter")h(t)o(yp)q(e,)75 1797 y(or)20 b(an)h(address)f(sized)i(in)o
(teger.)36 b(This)21 b(is)g(useful)h(only)f(on)g(F)l(ortran)e(systems)h(with)
h(extensions)h(for)75 1854 y(\(Cra)o(y\))14 b(p)q(oin)o(ters.)189
1939 y Fk(R)n(ationale.)60 b Fu(The)19 b(function)h Fn(MPI)p
815 1939 14 2 v 15 w(RMA)p 929 1939 V 17 w(MALLOC)f Fu(pro)o(vides)g(the)f
(opp)q(ortunit)o(y)h(to)f(allo)q(cate)189 1995 y(memory)13
b(that)g(is)h(more)f(easily)h(shared)g(within)h(the)f(sp)q(eci\014ed)h(comm)o
(unicator)f(group.)19 b(E.g.,)12 b(on)189 2051 y(a)g(shared)h(memory)g(mac)o
(hine,)h(it)f(can)g(allo)q(cate)h(the)f(memory)g(out)f(of)h(a)f(segmen)o(t)h
(that)f(is)i(shared)189 2108 y(b)o(y)h(all)h(pro)q(cesses)f(in)i(the)e(group)
g(of)f Fn(comm)n Fu(.)189 2180 y(The)f(comm)o(unicator)h(argumen)o(t)e(is)i
(pro)o(vided)h(for)d(the)i(case)g(that)e(di\013eren)o(t)i(segmen)o(ts)f(are)g
(used)189 2236 y(for)k(comm)o(unication)h(with)g(di\013eren)o(t)g(pro)q
(cesses.)28 b(E.g.,)17 b(in)i(the)f(case)f(of)h(a)f(cluster)i(of)e(SMP's,)189
2293 y(a)h(segmen)o(t)h(shared)g(b)o(y)g(all)h(pro)q(cessors)f(in)h(the)f
(SMP)g(w)o(ould)g(b)q(e)h(used)f(if)h(comm)o(unication)f(is)189
2349 y(in)o(ternal)c(to)f(the)h(SMP)g(no)q(de,)g(but)g(an)f(arbitrary)g
(segmen)o(t)h(migh)o(t)f(b)q(e)i(used)f(if)g(comm)o(unication)189
2406 y(is)g(external.)189 2478 y(A)22 b(system)g(ma)o(y)g(ha)o(v)o(e)g
(compiler)h(\(or)f(prepro)q(cessor\))g(supp)q(orted)h(mec)o(hanisms)g(to)e
(allo)q(cate)189 2534 y(\\go)q(o)q(d")16 b(memory)l(,)g(that)g(w)o(ould)i
(allo)o(w)f(suc)o(h)g(memory)f(to)g(b)q(e)i(used)f(for)f(static)h(v)m
(ariables,)h(e.g.,)189 2591 y(static)12 b(F)l(ortran)f(v)m(ariables,)k
Fl(COMMON)d Fu(areas,)g(or)g Fl(extern)i Fu(C)f(v)m(ariables.)20
b(The)13 b(curren)o(t)f(design)i(can)189 2647 y(tak)o(e)h(adv)m(an)o(tage)g
(of)h(suc)o(h)h(mec)o(hanisms:)22 b(One)17 b(can)f(op)q(en)g(a)g(windo)o(w)g
(in)o(to)g(memory)g(allo)q(cated)189 2704 y(b)o(y)f(suc)o(h)g(mec)o(hanism,)h
(rather)e(than)h(memory)g(allo)q(cated)h(b)o(y)f Fn(MPI)p 1360
2704 V 16 w(RMA)p 1475 2704 V 17 w(MALLOC)p Fu(.)1967 46 y
Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 3 5
bop 75 -100 a Fo(4.2.)34 b(INITIALIZA)l(TION)1302 b Fu(3)189
49 y(The)14 b(function)h Fn(MPI)p 543 49 14 2 v 16 w(RMA)p
658 49 V 17 w(MALLOC)f Fu(is)h(pro)o(vided)g(for)e(p)q(ortabilit)o(y)l(.)21
b(On)15 b(systems)f(that)f(do)h(not)189 106 y(ha)o(v)o(e)k(sp)q(ecial)j
(memory)l(,)f(it)g(will)g(b)q(e)g(a)f(simple)i(wrapp)q(er)e(on)g(top)g(of)g
(the)g(C)g Fn(mallo)q(c)d Fu(function.)189 162 y(Systems)d(that)f(ha)o(v)o(e)
g(a)h(sp)q(ecial)i(mec)o(hanism)f(for)e(dynamic)i(allo)q(cation)g(of)f(\\go)q
(o)q(d")f(memory)h(can)189 219 y(easily)j(use)f(those)g(to)g(supp)q(ort)g
Fn(MPI)p 820 219 V 16 w(RMA)p 935 219 V 17 w(MALLOC)p Fu(.)f(\()p
Fk(End)i(of)g(r)n(ationale.)p Fu(\))189 319 y Fk(A)n(dvic)n(e)j(to)h(users.)
66 b Fu(The)19 b(comm)o(unicator)g(argumen)o(t)g(in)h(the)g(call)g(is)g(only)
g(a)f(\\p)q(erformance)189 375 y(hin)o(t":)k(the)17 b(allo)q(cated)g(memory)f
(ma)o(y)g(b)q(e)i(used)f(for)f(RMA)h(comm)o(unication)g(o)o(v)o(er)f(an)o(y)h
(group,)189 432 y(lik)o(e)f(an)o(y)f(other)g(memory)l(.)k(\()p
Fk(End)d(of)g(advic)n(e)g(to)h(users.)p Fu(\))189 532 y Fk(A)n(dvic)n(e)12
b(to)h(implementors.)38 b Fu(A)12 b(design)g(similar)h(to)e(the)h(design)h
(of)e(C)h(mallo)q(c)g(and)g(free)g(functions)189 588 y(has)h(to)h(b)q(e)g
(used,)g(so)g(as)f(to)h(b)q(e)g(able)h(to)e(\014nd)h(out)g(the)g(size)h(of)e
(a)g(segmen)o(t,)h(when)g(the)g(segmen)o(t)f(is)189 645 y(freed.)21
b(MPI)15 b(has)h(to)f(k)o(eep)h(a)f(list)h(of)f(curren)o(tly)h(allo)q(cated)h
(segmen)o(ts.)j(This)d(list)f(can)g(also)f(hold)189 701 y(information)10
b(on)h(the)f(t)o(yp)q(e)g(of)g(segmen)o(t)g(allo)q(cated,)i(whic)o(h)g(ma)o
(y)d(b)q(e)i(needed)h(b)o(y)e Fn(MPI)p 1645 701 V 16 w(RMA)p
1760 701 V 17 w(INIT)p Fu(.)189 758 y(\()p Fk(End)15 b(of)i(advic)n(e)f(to)g
(implementors.)p Fu(\))166 941 y Fi(Missing:)35 b Fm(Need)15
b(new)f(error)h(co)q(des.)75 1124 y Fp(Example)j(4.1)k Fu(Example)13
b(of)f(use)h(of)f Fn(MPI)p 825 1124 V 16 w(RMA)p 940 1124 V
16 w(MALLOC)p Fu(,)g(in)h(F)l(ortran)e(with)i(p)q(oin)o(ter)g(supp)q(ort.)19
b(W)l(e)75 1180 y(assume)c(4)g(b)o(yte)g Fl(REAL)p Fu(s.)75
1270 y Fq(REAL)23 b(A)75 1326 y(POINTER)g(\(P,)g(A\(100,100\)\))70
b(!)24 b(no)g(memory)f(is)g(allocated)75 1383 y(CALL)g
(MPI_RMA_MALLOC\(4*100*100,)e(COMM,)i(P,)h(IERR\))695 1439
y(!)g(memory)f(is)h(allocated)166 1528 y Fu(Since)16 b(standard)e(F)l(ortran)
f(77)h(do)q(es)g(not)g(supp)q(ort)h(p)q(oin)o(ters,)f(this)h(co)q(de)g(is)g
(not)f(F)l(ortran)f(77)h(co)q(de.)75 1585 y(It)h(is)h(exp)q(ected)g(that)f
(most)f(\(all?\))21 b(F)l(ortran)14 b(compilers)j(o\013er)d(similar)i
(facilities.)166 1685 y(The)g(initialization)j(op)q(eration)d(allo)o(ws)g
(eac)o(h)g(pro)q(cess)h(to)e(sp)q(ecify)l(,)i(in)g(a)f(collectiv)o(e)h(op)q
(eration,)f(a)75 1742 y(\\memory)11 b(windo)o(w")h(that)f(is)h(made)g
(accessible)h(to)f(accesses)g(b)o(y)f(remote)h(pro)q(cesses.)19
b(The)12 b(information)75 1798 y(is)18 b(attac)o(hed)e(to)h(a)g(comm)o
(unicator)g(returned)g(b)o(y)h(the)f(call.)27 b(The)18 b(initialization)i(op)
q(eration)d(sp)q(eci\014es)75 1855 y(the)e(lo)q(cation)h(of)f(the)g(windo)o
(w)h(in)g(memory)l(.)75 2005 y Fn(MPI)p 160 2005 V 16 w(RMA)p
275 2005 V 17 w(INIT)e(\(base,)i(size,)f(disp)p 694 2005 V
17 w(unit,)h(comm)m(,)c(new)o(comm)n(\))117 2083 y Fm(IN)155
b Fn(base)505 b Fm(initial)12 b(address)j(of)f(windo)o(w)f(\(c)o(hoice\))117
2157 y(IN)155 b Fn(size)519 b Fm(size)15 b(of)e(windo)o(w)g(in)h(b)o(ytes)g
(\(non)g(negativ)o(e)g(in)o(teger\))117 2231 y(IN)155 b Fn(disp)p
395 2231 V 18 w(unit)422 b Fm(unit)16 b(size)h(for)e(displacemen)o(ts)h(in)f
(RMA)h(op)q(erations)g(\(non-)905 2288 y(negativ)o(e)e(in)o(teger\))117
2362 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(\(handle\))117
2436 y(OUT)108 b Fn(new)o(comm)393 b Fm(new)15 b(comm)o(uni)o(cator)d(to)h(b)
q(e)i(used)g(for)e(RMA)h(\(handle\))75 2561 y Fq(int)23 b(MPI)p
245 2561 15 2 v 17 w(RMA)p 334 2561 V 17 w(init\(void*)f(base,)i(int)f(size,)
g(int)h(disp)p 1187 2561 V 16 w(unit,)g(MPI)p 1419 2561 V 16
w(Comm)g(comm,)393 2617 y(MPI)p 468 2617 V 17 w(Comm)f(*newcomm\))75
2704 y(MPI)p 150 2704 V 17 w(RMA)p 239 2704 V 17 w(INIT\(BASE,)f(SIZE,)h
(DISP)p 757 2704 V 17 w(UNIT,)g(COMM,)g(NEWCOMM,)g(IERROR\))-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 4 6
bop 75 -100 a Fu(4)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)170 49 y Fq(<type>)23 b(BASE\(*\))170 106 y(INTEGER)g(SIZE,)h(DISP)p
604 106 15 2 v 16 w(UNIT,)f(COMM,)h(NEWCOMM,)e(IERROR)166 192
y Fu(This)e(is)h(a)e(collectiv)o(e)j(call)f(executed)g(b)o(y)e(all)i(pro)q
(cesses)g(in)f(the)g(group)g(of)f Fn(comm)m Fu(.)31 b(It)20
b(returns)75 249 y(a)e(comm)o(unicator)h(that)f(can)h(b)q(e)g(used)g(b)o(y)g
(these)g(pro)q(cesses)g(to)f(p)q(erform)h(RMA)g(op)q(erations.)30
b(Eac)o(h)75 305 y(pro)q(cess)16 b(sp)q(eci\014es)i(a)e(windo)o(w)g(of)g
(existing)h(memory)e(that)g(it)i(exp)q(oses)f(to)g(remote)f(accesses)h(b)o(y)
g(other)75 362 y(pro)q(cesses,)j(for)e(RMA)h(op)q(eration)g(that)f(use)h
(comm)o(unicator)g Fn(new)o(comm)m Fu(.)25 b(The)18 b(windo)o(w)g(consists)g
(of)75 418 y Fn(size)13 b Fu(b)o(ytes,)g(starting)f(at)h(address)g
Fn(base)p Fu(.)20 b(A)13 b(pro)q(cess)g(ma)o(y)g(elect)h(to)e(exp)q(ose)i(no)
f(memory)f(b)o(y)h(sp)q(ecifying)75 474 y Fn(size)i(=)h(0)p
Fu(.)166 531 y(The)g(displacemen)o(t)h(unit)g(argumen)o(t)e(is)h(pro)o(vided)
g(to)f(facilitate)i(address)f(arithmetics)g(in)h(RMA)75 587
y(op)q(erations:)j(the)15 b(target)e(displacemen)o(t)j(argumen)o(t)e(of)h(an)
g(RMA)f(op)q(eration)h(is)h(scaled)f(b)o(y)g(the)g(factor)75
644 y Fn(disp)p 152 644 14 2 v 18 w(unit)h Fu(sp)q(eci\014ed)h(b)o(y)e(the)h
(target)e(pro)q(cess,)h(at)f(windo)o(w)i(creation.)166 700
y(Eac)o(h)21 b(windo)o(w)h(is)f(asso)q(ciated)h(with)f(a)g(coun)o(ter)h(that)
e(can)i(b)q(e)g(used)f(to)g(sync)o(hronize)i(remote)-170 687
y Fh(>)p Fu(\()p Fp(June)p Fu(\))75 746 y(memory)15 b(access)g(op)q
(erations.)20 b(This)c(is)f(describ)q(ed)j(in)e(more)e(detail)j(in)f(Section)
g(4.6.)-170 732 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 791 y(A)i(comm)o
(unicator)g(can)g(b)q(e)h(used)g(for)e(RMA)h(commands)g(only)h(after)e(the)i
(RMA)f(initialization)75 848 y(call.)j(Suc)o(h)15 b(comm)o(unicator)f(can)h
(also)f(b)q(e)h(used)g(for)f(other)g(MPI)h(calls,)g(as)f(an)o(y)h(other)f
(comm)o(unicator.)166 904 y(The)21 b(newly)h(created)f(comm)o(unicator)f
Fn(new)o(comm)c Fu(carries)21 b(all)h(the)f(information)g(cac)o(hed)h(with)75
961 y Fn(comm)m Fu(,)12 b(including)18 b(top)q(ology)d(information,)g(as)g
(if)g Fn(new)o(comm)10 b Fu(w)o(as)k(created)i(b)o(y)f(duplicating)i
Fn(comm)m Fu(.)166 1017 y(The)i(v)m(arious)h(pro)q(cesses)f(in)h(the)f(group)
g(of)f Fn(comm)13 b Fu(ma)o(y)18 b(sp)q(ecify)j(completely)f(di\013eren)o(t)f
(target)75 1073 y(windo)o(ws,)g(in)h(lo)q(cation,)g(size)f(and)g(displacemen)
o(t)h(units.)31 b(As)18 b(long)h(as)f(all)i(the)f(gets)f(and)h(puts)f(to)g(a)
75 1130 y(particular)e(pro)q(cess)f(\014t)g(its)h(sp)q(eci\014c)h(target)d
(windo)o(w)h(this)h(should)g(p)q(ose)g(no)f(problem.)166 1186
y(The)k(information)h(ab)q(out)f(the)g(set)h(of)e(windo)o(ws)i(asso)q(ciated)
f(with)h Fn(new)o(comm)14 b Fu(is)20 b(cac)o(hed)g(with)75
1243 y(the)13 b(newly)i(created)e(comm)o(unicator)g(\(in)h(the)g(same)f
(manner)g(top)q(ology)g(information)h(is)g(cac)o(hed)f(when)75
1299 y(a)k(top)q(ology)g(is)h(created\).)25 b(Th)o(us,)18 b(this)g
(information)f(is)h(transfered)f(to)f(the)i(newly)g(created)f(comm)o(u-)75
1356 y(nicator)i(if)g Fn(new)o(comm)14 b Fu(is)19 b(duplicated)i(b)o(y)e(a)f
(call)i(to)e Fn(MPI)p 1100 1356 V 16 w(COMM)p 1258 1356 V 17
w(DUP)p Fu(,)g(but)h(is)h(not)e(transfered)h(to)75 1412 y(comm)o(unicators)c
(created)g(b)o(y)g(other)g(calls.)166 1469 y(The)k(same)g(area)g(in)h(memory)
f(ma)o(y)f(app)q(ear)i(in)g(m)o(ultiple)h(windo)o(ws,)f(eac)o(h)f(asso)q
(ciated)h(with)f(a)75 1525 y(di\013eren)o(t)c(comm)o(unicator.)189
1631 y Fk(A)n(dvic)n(e)k(to)i(users.)70 b Fu(The)20 b(displacemen)o(t)h(unit)
g(argumen)o(t)f(facilitates)g(computation)g(of)g(dis-)189 1688
y(placemen)o(ts)d(in)h(a)e(heterogeneous)h(en)o(vironmen)o(t.)25
b(Supp)q(ose)17 b(that)f(eac)o(h)h(windo)o(w)g(con)o(tains)g(an)189
1744 y(arra)o(y)f(of)h(\015oats.)25 b(The)17 b(size)i(of)e
Fl(\015oat)g Fu(ma)o(y)f(di\013er)i(on)f(di\013eren)o(t)h(mac)o(hines.)27
b(Eac)o(h)17 b(pro)q(cess)g(will)189 1801 y(pro)o(vide)h(a)g(displacemen)o(t)
h(unit)g(whic)o(h)g(is)g(the)f(exten)o(t)f(of)h Fl(MPI)p 1303
1801 13 2 v 14 w(FLO)o(A)m(T)p Fu(.)f(Then,)i(en)o(tries)f(of)g(the)189
1857 y(arra)o(y)d(in)h(a)g(target)f(pro)q(cess)i(can)f(b)q(e)h(sp)q
(eci\014ed)h(b)o(y)e(their)h(index,)g(with)f(the)g(righ)o(t)g(scaling)i(done)
189 1914 y(automatically)d(b)o(y)g(MPI.)g(If)h Fn(disp)p 778
1914 14 2 v 17 w(unit)h(=)e(1)p Fu(,)g(then)g(b)o(yte)g(displacemen)o(ts)i
(are)e(used.)189 1989 y(A)d(\\go)q(o)q(d")g(windo)o(w)g(for)g(RMA)h(comm)o
(unication)g(is)g(created)f(b)o(y)h(\014rst)f(allo)q(cating)h(\\go)q(o)q(d")f
(mem-)189 2045 y(ory)g(with)i Fn(MPI)p 453 2045 V 16 w(RMA)p
568 2045 V 16 w(MALLOC)f Fu(next)h(creating)f(an)g(RMA)h(windo)o(w)f(in)h
(the)f(memory)g(allo)q(cated)189 2102 y(with)i Fn(MPI)p 377
2102 V 16 w(RMA)p 492 2102 V 17 w(INIT)p Fu(.)f(\()p Fk(End)h(of)i(advic)n(e)
f(to)h(users.)p Fu(\))189 2208 y Fk(A)n(dvic)n(e)f(to)i(implementors.)50
b Fu(In)17 b(cases)g(where)g(RMA)f(op)q(erations)h(use)g(di\013eren)o(t)g
(mec)o(hanisms)189 2264 y(in)d(di\013eren)o(t)f(memory)g(areas)f(\(e.g.,)g
(load/store)h(in)h(a)f(shared)g(memory)g(segmen)o(t,)f(and)i(an)f(asyn-)189
2321 y(c)o(hronous)21 b(handler)h(in)g(priv)m(ate)g(memory\),)g(the)f
Fn(MPI)p 1157 2321 V 16 w(RMA)p 1272 2321 V 17 w(INIT)g Fu(call)h(needs)g(to)
f(\014gure)g(out)189 2377 y(whic)o(h)c(t)o(yp)q(e)g(of)f(memory)g(is)h(used)h
(for)e(the)g(windo)o(w.)25 b(T)l(o)16 b(do)h(so,)f(MPI)h(main)o(tains,)g(in)o
(ternally)l(,)189 2434 y(the)j(list)h(of)f(memory)g(segmen)o(ts)g(allo)q
(cated)h(b)o(y)f Fn(MPI)p 1141 2434 V 16 w(RMA)p 1256 2434
V 17 w(MALLOC)p Fu(,)f(or)h(b)o(y)g(other,)h(imple-)189 2490
y(men)o(tation)e(sp)q(eci\014c,)k(mec)o(hanisms,)e(together)f(with)g
(information)g(on)g(the)g(t)o(yp)q(e)g(of)g(memory)189 2547
y(segmen)o(t)f(allo)q(cated.)33 b(When)20 b(a)g(call)g(to)f
Fn(MPI)p 1004 2547 V 16 w(RMA)p 1119 2547 V 17 w(INIT)g Fu(o)q(ccurs,)h(then)
g(MPI)f(c)o(hec)o(ks)h(whic)o(h)189 2603 y(segmen)o(t)15 b(con)o(tains)h(eac)
o(h)g(windo)o(w,)g(and)h(decide,)g(accordingly)l(,)g(whic)o(h)g(mec)o(hanism)
g(to)e(use)i(for)189 2660 y(RMA)e(op)q(erations.)20 b(\()p
Fk(End)c(of)g(advic)n(e)g(to)h(implementors.)p Fu(\))1967 46
y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 5 7
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(5)166 49 y(It)10 b(is)h(con)o(v)o(enien)o(t)g(to)f(b)q(e)h(able)g(to)f
(access)g(information)h(on)f(windo)o(w)h(parameters)e(from)h(the)g(comm)o(u-)
75 106 y(nicator)k(that)f(represen)o(ts)h(the)g(windo)o(w.)20
b(T)l(o)13 b(do)h(so,)g(three)g(prede\014ned)h(attribute)f(k)o(eys)g(are)g
(pro)o(vided.)75 212 y Fp(MPI)p 184 212 16 2 v 18 w(WINDO)o(W)p
449 212 V 19 w(BASE:)22 b Fu(Initial)17 b(address)e(of)g(lo)q(cal)h(address)
75 306 y Fp(MPI)p 184 306 V 18 w(WINDO)o(W)p 449 306 V 19 w(SIZE:)22
b Fu(size)16 b(of)f(lo)q(cal)h(windo)o(w,)f(in)h(b)o(ytes)75
400 y Fp(MPI)p 184 400 V 18 w(WINDO)o(W)p 449 400 V 19 w(DISP)p
593 400 V 18 w(UNIT:)23 b Fu(displacemen)o(t)16 b(unit)g(used)g(for)f(lo)q
(cal)h(windo)o(w)166 506 y(If)g(a)f(comm)o(unicator)h Fn(comm)10
b Fu(w)o(as)15 b(created)g(b)o(y)h(a)g(call)h(to)e Fn(MPI)p
1254 506 14 2 v 15 w(RMA)p 1368 506 V 17 w(INIT\(base,)h(size,)g(disp)p
1774 506 V 17 w(unit,)75 562 y Fg(:)8 b(:)g(:)e Fn(\))p Fu(,)20
b(then)g Fn(MPI)p 380 562 V 16 w(A)l(TTR)p 513 562 V 17 w(GET\(comm)m(,)d
(MPI)p 866 562 V 16 w(WINDO)o(W)p 1078 562 V 16 w(BASE,)j(val,)g(\015ag\))g
Fu(will)h(return)f Fn(val)f(=)h(base)75 619 y Fu(\(after)14
b(t)o(yp)q(e)h(con)o(v)o(ersion,)g(in)h(C\);)f(and)g(similarly)l(,)i(for)d
(the)i(other)f(t)o(w)o(o.)189 725 y Fk(A)n(dvic)n(e)d(to)i(users.)38
b Fu(If)12 b Fn(comm)6 b Fu(is)13 b(not)e(asso)q(ciated)h(with)h(RMA)f(windo)
o(ws,)g(then)h Fn(MPI)p 1640 725 V 16 w(A)l(TTR)p 1773 725
V 17 w(GET)189 782 y Fu(will)18 b(return)f Fn(\015ag)g(=)g(false)p
Fu(.)25 b(Th)o(us,)17 b(these)g(attributes)f(also)h(pro)o(vide)g(a)g(mec)o
(hanism)g(for)g(\014nding)189 838 y(out)f(whether)g(a)g(comm)o(unicator)g(is)
h(asso)q(ciated)g(with)g(an)f(RMA)h(windo)o(w.)23 b(\()p Fk(End)17
b(of)g(advic)n(e)h(to)189 894 y(users.)p Fu(\))75 1038 y Fr(4.3)59
b(Put)20 b(and)g(Get)f(Calls)75 1139 y Fu(The)c(execution)g(of)f(a)g(put)h
(op)q(eration)g(is)g(similar)g(to)f(the)h(execution)g(of)f(a)g(send)i(b)o(y)e
(the)h(origin)g(pro)q(cess)75 1196 y(and)f(a)g(matc)o(hing)g(receiv)o(e)h(b)o
(y)g(the)f(target)f(pro)q(cess.)19 b(The)c(ob)o(vious)f(di\013erence)i(is)e
(that)g(all)h(parameters)75 1252 y(are)g(pro)o(vided)h(b)o(y)f(one)g(call)i
({)e(the)g(call)h(executed)g(b)o(y)f(the)h(origin)g(pro)q(cess.)75
1374 y Ff(4.3.1)49 b(Non-blo)q(cking)19 b(and)d(Blo)q(cking)i(Put)111
1460 y Fn(MPI)p 196 1460 V 16 w(PUT\()g(o)o(rigin)p 445 1460
V 15 w(addr,)h(o)o(rigin)p 682 1460 V 16 w(count,)g(o)o(rigin)p
941 1460 V 16 w(datat)o(yp)q(e,)h(ta)o(rget)p 1265 1460 V 16
w(rank,)f(ta)o(rget)p 1508 1460 V 16 w(disp,)h(ta)o(rget)p
1743 1460 V 16 w(count,)-170 1449 y Fh(>)p Fu(\()p Fp(June)p
Fu(\))75 1599 y Fn(ta)o(rget)p 190 1599 V 16 w(datat)o(yp)q(e,)d(increment,)c
(comm)m(\))117 1677 y Fm(IN)155 b Fn(o)o(rigin)p 427 1677 V
16 w(addr)381 b Fm(address)15 b(of)f(origin)e(bu\013er)j(\(c)o(hoice\))117
1752 y(IN)155 b Fn(o)o(rigin)p 427 1752 V 16 w(count)360 b
Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117
1827 y(IN)155 b Fn(o)o(rigin)p 427 1827 V 16 w(datat)o(yp)q(e)302
b Fm(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(sen)o(t)i(\(handle\))117
1902 y(IN)155 b Fn(ta)o(rget)p 433 1902 V 17 w(rank)375 b Fm(rank)14
b(of)f(target)i(\(in)o(teger\))117 1977 y(IN)155 b Fn(ta)o(rget)p
433 1977 V 17 w(disp)384 b Fm(displacemen)o(t)20 b(from)f(start)i(of)f(windo)
o(w)g(to)h(target)g(bu\013er)905 2034 y(\(in)o(teger\))117
2109 y(IN)155 b Fn(ta)o(rget)p 433 2109 V 17 w(count)353 b
Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
2184 y(IN)155 b Fn(ta)o(rget)p 433 2184 V 17 w(datat)o(yp)q(e)295
b Fm(datat)o(yp)q(e)14 b(used)h(at)f(the)g(target)h(\(handle\))117
2259 y(IN)155 b Fn(increment)399 b Fm(incremen)o(t)16 b(added)g(to)f(the)h
(target)h(windo)o(w)d(coun)o(ter)j(\(in)o(te-)905 2316 y(ger\))117
2391 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(used)k(for)f(comm)o
(unicati)o(on)d(\(handle\))75 2515 y Fq(int)47 b(MPI)p 269
2515 15 2 v 17 w(Put\()23 b(void)h(*origin)p 693 2515 V 16
w(addr,)f(int)h(origin)p 1092 2515 V 16 w(count,)f(MPI)p 1347
2515 V 17 w(Datatype)393 2572 y(origin)p 540 2572 V 17 w(datatype,)f(int)i
(target)p 1035 2572 V 16 w(rank,)f(int)h(target)p 1434 2572
V 16 w(disp,)f(int)393 2628 y(target)p 540 2628 V 17 w(count,)g(MPI)p
796 2628 V 16 w(Datatype)g(target)p 1171 2628 V 16 w(datatype,)g(int)h
(increment,)393 2685 y(MPI)p 468 2685 V 17 w(Comm)f(comm\))-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 6 8
bop 75 -100 a Fu(6)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)75 49 y Fq(MPI)p 150 49 15 2 v 17 w(PUT\()23 b(ORIGIN)p
430 49 V 16 w(ADDR,)h(ORIGIN)p 734 49 V 16 w(COUNT,)f(ORIGIN)p
1061 49 V 16 w(DATATYPE,)g(TARGET)p 1460 49 V 16 w(RANK,)393
106 y(TARGET)p 540 106 V 17 w(DISP,)g(TARGET)p 844 106 V 16
w(COUNT,)g(TARGET)p 1171 106 V 16 w(DATATYPE,)g(INCREMENT,)g(COMM,)393
162 y(IERROR\))170 219 y(<type>)g(ORIGIN)p 484 219 V 17 w(ADDR\(*\))170
275 y(INTEGER)g(ORIGIN)p 508 275 V 17 w(COUNT,)g(ORIGIN)p 836
275 V 16 w(DATATYPE,)g(TARGET)p 1235 275 V 16 w(RANK,)g(TARGET)p
1538 275 V 17 w(DISP,)170 332 y(TARGET)p 317 332 V 17 w(COUNT,)g(TARGET)p
645 332 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(IERROR)-170
396 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 463 y(T)l(ransfers)c
Fn(o)o(rigin)p 478 463 14 2 v 16 w(count)i Fu(successiv)o(e)g(en)o(tries)f
(of)g(the)g(t)o(yp)q(e)g(sp)q(eci\014ed)i(b)o(y)d(the)h Fn(o)o(rigin)p
1685 463 V 16 w(datat)o(yp)q(e)p Fu(,)75 520 y(starting)13
b(at)h(address)g Fn(o)o(rigin)p 568 520 V 15 w(addr)g Fu(on)g(the)g(origin)h
(no)q(de)f(to)g(the)g(target)f(no)q(de)h(sp)q(eci\014ed)i(b)o(y)e(the)g
Fn(comm)m Fu(,)75 576 y Fn(ta)o(rget)p 190 576 V 16 w(rank)19
b Fu(pair.)31 b(The)19 b(data)f(are)h(written)g(in)h(the)e(target)g(bu\013er)
h(at)f(address)h Fn(ta)o(rget)p 1624 576 V 16 w(addr)g(=)g(win-)75
633 y(do)o(w)p 154 633 V 16 w(base)c(+)e(ta)o(rget)p 427 633
V 16 w(disp)p Fh(\002)p Fn(disp)p 626 633 V 20 w(unit)p Fu(,)h(where)f
Fn(windo)o(w)p 1016 633 V 17 w(base)h Fu(and)g Fn(disp)p 1290
633 V 17 w(unit)h Fu(are)e(the)g(base)g(address)h(and)75 689
y(windo)o(w)h(displacemen)o(t)i(unit)f(sp)q(eci\014ed)h(at)e(windo)o(w)g
(initialization,)j(b)o(y)d(the)h(target)e(pro)q(cess.)166 746
y(The)h(target)f(bu\013er)h(is)h(sp)q(eci\014ed)i(b)o(y)d(the)g(argumen)o(ts)
f Fn(ta)o(rget)p 1230 746 V 17 w(count)i Fu(and)f Fn(ta)o(rget)p
1568 746 V 17 w(datat)o(yp)q(e)p Fu(.)166 802 y(The)25 b(outcome)f(is)h(as)f
(if)i(the)e(origin)i(pro)q(cess)f(executed)g(a)g(blo)q(c)o(king)h(send)f(op)q
(eration)g(with)75 859 y(argumen)o(ts)c Fn(o)o(rigin)p 412
859 V 15 w(addr,)i(o)o(rigin)p 653 859 V 15 w(count,)h(o)o(rigin)p
916 859 V 15 w(datat)o(yp)q(e,)g(ta)o(rget)p 1243 859 V 17
w(rank,)e(tag,)h(comm)m Fu(,)d(and)h(the)h(tar-)75 915 y(get)i(pro)q(cess)g
(executed)h(a)e(receiv)o(e)i(op)q(eration)f(with)h(argumen)o(ts)e
Fn(ta)o(rget)p 1398 915 V 16 w(addr,)j(ta)o(rget)p 1649 915
V 16 w(count,)h(ta)o(r-)75 972 y(get)p 137 972 V 17 w(datat)o(yp)q(e,)21
b(source,)g(tag,)f(comm)m Fu(,)d(where)i Fn(ta)o(rget)p 993
972 V 16 w(addr)h Fu(is)g(the)f(target)f(bu\013er)h(address)h(computed)75
1028 y(as)15 b(explained)i(ab)q(o)o(v)o(e.)j(The)15 b(message)g(sen)o(t)g(m)o
(ust)f(\014t,)h(without)g(truncation,)g(in)h(the)g(target)d(bu\013er.)166
1084 y(The)19 b Fn(ta)o(rget)p 378 1084 V 16 w(datat)o(yp)q(e)i
Fu(argumen)o(t)d(is)h(a)g(handle)h(to)e(a)h(datat)o(yp)q(e)f(ob)s(ject)h
(de\014ned)h(at)e(the)h(origin)75 1141 y(pro)q(cess.)j(Ho)o(w)o(ev)o(er,)15
b(this)h(ob)s(ject)f(is)i(in)o(terpreted)f(at)f(the)h(target)f(pro)q(cess:)21
b(the)16 b(outcome)g(is)g(as)g(if)g(the)75 1197 y(target)c(datat)o(yp)q(e)h
(ob)s(ject)f(w)o(as)g(de\014ned)j(at)d(the)i(target)e(pro)q(cess,)h(b)o(y)g
(the)g(same)g(sequence)h(of)f(calls)h(used)75 1254 y(to)h(de\014ne)h(it)f(at)
g(the)g(origin)h(pro)q(cess.)166 1310 y(The)f(comm)o(unication)g(op)q
(eration)g(completes)h(at)e(the)h(origin)g(after)f(the)h(data)f(ha)o(v)o(e)h
(b)q(een)h(copied)75 1367 y(out)g(of)g(the)h(origin)g(bu\013er,)g(p)q
(ossibly)h(b)q(efore)f(the)f(data)g(has)h(b)q(een)g(copied)h(in)o(to)f(the)f
(target)g(memory)l(.)75 1423 y(The)d(coun)o(ter)f(asso)q(ciated)h(with)g(the)
g(target)f(windo)o(w)h(is)g(incremen)o(ted)h(b)o(y)f(the)f(v)m(alue)i
Fn(ta)o(rget)p 1675 1423 V 17 w(increment)-170 1410 y Fh(>)p
Fu(\()p Fp(June)p Fu(\))75 1469 y(after)k(data)h(ha)o(v)o(e)f(b)q(een)j
(copied)f(in)o(to)f(the)g(target)f(memory)l(.)31 b(The)19 b(seman)o(tics)h
(of)e(these)i(coun)o(ters)e(is)75 1525 y(explained)f(in)f(Section)g(4.6.)-170
1512 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 1571 y(The)e Fn(MPI)p
343 1571 V 16 w(Put)h Fu(call)g(is)f(\\lo)q(cal":)20 b(the)14
b(completion)h(of)e(the)h(call)h(do)q(es)g(not)e(dep)q(end)j(on)d(the)h
(state)f(of)75 1627 y(other)i(pro)q(cesses,)g(or)g(on)g(the)g(state)f(of)h
(other)g(ongoing)g(comm)o(unications.)166 1683 y(The)h(target)f(pro)q(cess)i
(ma)o(y)e(b)q(e)i(iden)o(tical)h(with)f(the)f(origin)h(pro)q(cess;)g(i.e.,)f
(a)g(pro)q(cess)g(ma)o(y)g(use)g(a)75 1740 y(put)f(call)i(to)d(mo)o(v)o(e)h
(data)f(in)i(its)g(memory)l(.)j(The)d(same)e(holds)i(true)f(for)g(all)h
(other)f(RMA)g(calls.)166 1796 y(The)i(function)h(returns)e(an)h(error)f(co)q
(de)i(as)e(in)i(MPI.)e(The)h(put)g(call)h(is)f(erroneous)g(if)g(the)g(target)
75 1853 y(bu\013er)h(is)g(not)f(wholly)i(con)o(tained)f(in)g(the)g(target)e
(windo)o(w.)28 b(The)18 b(target)e(datat)o(yp)q(e)h(should)i(con)o(tain)75
1909 y(only)d(relativ)o(e)f(displacemen)o(ts,)i(not)e(absolute)g(addresses.)
189 2016 y Fk(R)n(ationale.)54 b Fu(A)17 b(target)g(datat)o(yp)q(e)g(is)h
(pro)o(vided)g(in)g(the)g(put)g(call)g(so)g(as)f(to)g(allo)o(w)g(data)g(to)g
(b)q(e)189 2072 y(scattered)f(\(put\))g(or)g(gathered)g(\(get\))f(in)i(the)g
(remote)f(memory)l(,)g(without)g(in)o(v)o(olv)o(emen)o(t)h(of)f(the)189
2128 y(remote)e(pro)q(cess;)h(see)h(example)g(in)g(Section)g(4.3.3.)189
2204 y(The)d(use)g(of)g(the)g(windo)o(w)h(coun)o(ter)f(allo)o(ws)g(the)g
(target)f(pro)q(cess)h(to)g(b)q(e)h(informed)f(that)g(a)g(remote)-170
2190 y Fh(>)p Fu(\()p Fp(June)p Fu(\))189 2249 y(access)g(o)q(ccurred)i(in)f
(its)g(memory)l(.)19 b(Use)13 b(of)h(this)f(mec)o(hanism)i(is)f(facultativ)o
(e:)19 b(unlik)o(e)c(comm)o(uni-)189 2305 y(cation)e(via)g(send/receiv)o(e,)h
(there)f(is)h(no)e(need)i(for)e(the)h(target)f(pro)q(cess)h(to)g(p)q(erform)f
(an)h(action)g(in)189 2362 y(order)h(that)f(the)i(comm)o(unication)g(o)q
(ccur;)f(nor)g(is)h(it)g(necessary)f(that)g(the)g(target)f(pro)q(cess)i(c)o
(hec)o(k)189 2418 y(separately)k(for)f(eac)o(h)h(comm)o(unication.)33
b(Instead,)20 b(one)f(could)h(ha)o(v)o(e)f(a)g(sequence)h(of)e(put/get)189
2475 y(calls,)j(follo)o(w)o(ed)e(b)o(y)g(a)g(barrier)g(sync)o(hronization)i
(that)d(informs)h(all)h(pro)q(cesses)g(that)e(all)j(data)189
2531 y(transfers)15 b(completed.)22 b(Ho)o(w)o(ev)o(er,)15
b(it)h(is)h(imp)q(ortan)o(t)e(to)g(ha)o(v)o(e)h(an)f(explicit)j(MPI)e(call)h
(to)e(ensure)189 2588 y(that)c(access)i(to)f(the)g(target)g(bu\013er)g(b)o(y)
g(the)h(target)e(pro)q(cess)i(o)q(ccurs)g(only)g(after)f(the)g(put)h
(transfer)189 2644 y(has)i(completed.)1967 46 y Fj(1)1967 103
y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967
385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959
667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893
y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119
y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345
y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570
y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796
y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022
y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248
y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474
y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699
y(48)p eop
%%Page: 7 9
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(7)-170 49 y Fh(?)p Fu(\()p Fp(June)p Fu(\))189 127 y(RMA)16
b(calls)g(targeted)f(to)h(the)g(origin)g(o)o(wn)f(memory)h(ma)o(y)f(b)q(e)h
(b)q(ene\014cial)i(when)f(they)f(allo)o(w)g(to)189 183 y(use)g(a)f(separate)h
(engine)h(for)e(async)o(hronous)h(memory)f(to)g(memory)h(transfers.)21
b(They)16 b(are)f(v)o(ery)189 240 y(useful)j(with)f(accum)o(ulate)h(op)q
(erations,)f(to)f(allo)o(w)i(atomic)f(up)q(dates)g(of)g(lo)q(cal)h(v)m
(ariables.)27 b(\()p Fk(End)189 296 y(of)16 b(r)n(ationale.)p
Fu(\))189 403 y Fk(A)n(dvic)n(e)11 b(to)i(users.)37 b Fu(The)12
b Fn(ta)o(rget)p 739 403 14 2 v 16 w(datat)o(yp)q(e)h Fu(argumen)o(t)d(is)i
(a)e(handle)j(to)d(a)h(datat)o(yp)q(e)g(ob)s(ject)f(that)g(is)189
459 y(de\014ned)j(at)f(the)g(origin)h(pro)q(cess,)g(ev)o(en)f(though)g(it)h
(de\014nes)g(a)f(data)g(la)o(y)o(out)f(in)i(the)g(target)e(pro)q(cess)189
515 y(memory)l(.)18 b(This)13 b(causes)g(no)f(problems)h(in)g(a)f(homogenous)
g(en)o(vironmen)o(t,)h(or)f(in)h(a)f(heterogenous)189 572 y(en)o(vironmen)o
(t,)i(if)g(only)h(p)q(ortable)f(datat)o(yp)q(es)g(are)f(used.)20
b(A)14 b(datat)o(yp)q(e)g(is)g Fk(p)n(ortable)p Fu(,)g(if)h(it)f(is)g(basic)
189 628 y(datat)o(yp)q(e,)f(or)g(it)h(is)g(deriv)o(ed)h(using)f(only)h(the)e
(t)o(yp)q(e)h(constructors)f Fn(MPI)p 1426 628 V 16 w(TYPE)p
1559 628 V 17 w(CONTIGUOUS,)189 685 y(MPI)p 274 685 V 15 w(TYPE)p
406 685 V 17 w(VECTOR)g Fu(and)e Fn(MPI)p 781 685 V 16 w(TYPE)p
914 685 V 17 w(INDEXED)g Fu(\()p Fn(MPI)p 1237 685 V 15 w(TYPE)p
1369 685 V 17 w(SIMPLE)p 1545 685 V 16 w(STRUCT)h Fu(should)189
741 y(b)q(e)18 b(added,)i(if)e(it)h(mak)o(es)e(it)h(to)g(MPI2\).)28
b(Suc)o(h)19 b(datat)o(yp)q(e)e(is)i(p)q(ortable,)g(b)q(ecause)g(all)g
(displace-)189 798 y(men)o(ts)i(in)i(suc)o(h)g(datat)o(yp)q(e)e(are)h(in)h
(terms)f(of)g(exten)o(ts)f(of)h(basic)h(datat)o(yp)q(es.)40
b(Therefore,)23 b(if)189 854 y(suc)o(h)d(datat)o(yp)q(e)g(\014ts)f(a)h(data)f
(la)o(y)o(out)h(in)h(one)f(memory)l(,)h(it)f(will)i(\014t)d(the)i(corresp)q
(onding)g(data)189 911 y(la)o(y)o(out)i(in)h(another)g(memory)l(,)h(if)f(the)
g(same)g(declarations)g(where)g(used,)i(ev)o(en)e(if)g(the)g(t)o(w)o(o)189
967 y(systems)e(ha)o(v)o(e)g(di\013eren)o(t)h(arc)o(hitectures.)42
b(On)24 b(the)e(other)h(hand,)h(if)f(a)g(datat)o(yp)q(e)f(w)o(as)g(con-)189
1024 y(structed)11 b(using)h Fn(MPI)p 562 1024 V 16 w(TYPE)p
695 1024 V 16 w(HINDEXED,)g(MPI)p 1045 1024 V 15 w(TYPE)p 1177
1024 V 17 w(HVECTOR)h Fu(or)e Fn(MPI)p 1552 1024 V 15 w(TYPE)p
1684 1024 V 17 w(STRUCT)189 1080 y Fu(\(and)g Fn(MPI)p 376
1080 V 16 w(CONTIGUOUS)p 680 1080 V 18 w(STRUCT)p Fu(\),)g(then)h(the)g
(datat)o(yp)q(e)f(con)o(tains)g(explicit)j(b)o(yte)d(displace-)189
1136 y(men)o(ts.)19 b(These)c(displacemen)o(ts)g(are)f(unlik)o(ely)j(to)c(b)q
(e)i(c)o(hosen)g(correctly)g(if)f(they)h(\014t)f(data)g(la)o(y)o(out)189
1193 y(on)d(one)i(memory)l(,)e(but)h(are)g(used)g(for)g(data)f(la)o(y)o(outs)
g(on)h(another)g(pro)q(cess,)g(running)h(on)f(a)f(pro)q(ces-)189
1249 y(sor)18 b(with)h(a)f(di\013eren)o(t)h(arc)o(hitecture.)30
b(In)19 b(the)g(later)f(case,)h(one)g(should)h(use)f(the)f(mec)o(hanisms)189
1306 y(de\014ned)d(in)f(MPI2)g(for)f(transferring)g(datat)o(yp)q(es)h(across)
f(pro)q(cesses.)19 b(\()p Fk(End)c(of)g(advic)n(e)f(to)i(users.)p
Fu(\))189 1412 y Fk(A)n(dvic)n(e)e(to)i(implementors.)39 b
Fu(Put)14 b(can)g(b)q(e)h(implemen)o(ted)h(in)f(a)e(homogeneous)h(shared)h
(memory)189 1469 y(en)o(vironmen)o(t)h(b)o(y)g(co)q(de)h(executed)g(b)o(y)f
(the)g(calling)i(pro)q(cess.)23 b(Supp)q(ose)17 b(one)g(has,)e(in)i(this)g
(en)o(vi-)189 1525 y(ronmen)o(t,)h(and)g(implemen)o(tation)i(of)d
(send/receiv)o(e,)j(using)f(memory-to-memory)e(cop)o(y)l(.)29
b(The)189 1581 y(put)17 b(op)q(eration)g(is)g(implemen)o(ted,)h(in)g
(general,)f(as)g(a)f(\\send")h(from)f(origin)i(memory)e(to)g(an)h(in-)189
1638 y(termediate)f(bu\013er,)h(implemen)o(ted)h(as)f(memory-to-memory)e(cop)
o(y)l(,)i(follo)o(w)o(ed)g(b)o(y)f(a)h(\\receiv)o(e")189 1694
y(from)11 b(the)i(in)o(termediate)g(bu\013er)f(to)g(the)g(target)g(memory)l
(,)g(implemen)o(ted)i(again)e(using)h(memory-)189 1751 y(to-memory)g(cop)o(y)
l(.)20 b(The)15 b(in)o(termediate)h(cop)o(y)e(can)h(b)q(e)h(a)o(v)o(oided)f
(\(or)e(pip)q(elin)q(ed\))k(if)f(the)e(message)189 1807 y(passing)f(implemen)
o(tation)i(has)e(the)g(abilit)o(y)i(to)d(do)i(on-the-\015y)g(datat)o(yp)q(e)e
(in)o(terpretation)i([)p Fp(?)p Fu(].)k(It)189 1864 y(can)c(also)g(b)q(e)h
(optimized)h(a)o(w)o(a)o(y)c(in)k(the)e(prev)m(alen)o(t)h(case)f(where)h
(either)g(the)f(origin)h(bu\013er)f(or)g(the)189 1920 y(target)g(bu\013er)h
(are)g(con)o(tiguous.)189 1995 y(Consider)20 b(a)f(homogenous)g(distributed)j
(memory)d(mac)o(hine,)i(where)f(a)f(DMA)g(engine)i(allo)o(ws)189
2052 y(transfer)12 b(of)g(con)o(tiguous)h(memory)g(blo)q(c)o(ks.)20
b(If)13 b(b)q(oth)g(origin)h(and)f(target)f(bu\013er)h(are)f(con)o(tiguous,)
189 2108 y(then)i(DMA)e(can)i(b)q(e)g(directly)h(used)f(for)f(the)h(put.)19
b(If)14 b(the)g(target)e(bu\013er)i(is)g(con)o(tiguous,)f(but)h(the)189
2165 y(origin)j(bu\013er)g(is)g(not,)f(then)h(a)g(memory-to-memory)e(cop)o(y)
h(can)h(b)q(e)h(used)f(to)f(gather)g(the)h(data)189 2221 y(in)o(to)g(a)g(con)
o(tiguous)h(bu\013er,)f(and)h(the)f(DMA)g(engine)i(can)e(b)q(e)h(used)g(next)
g(for)e(the)i(transfer.)26 b(If)189 2278 y(the)14 b(origin)g(bu\013er)g(is)h
(con)o(tiguous,)f(but)g(the)g(target)f(bu\013er)h(is)g(not,)g(then)g(the)g
(DMA)g(engine)h(can)189 2334 y(b)q(e)f(used)h(to)e(transfer)g(the)h(data,)f
(and)h(an)g(agen)o(t)f(on)h(the)g(target)e(no)q(de)j(is)f(needed)i(to)d
(scatter)g(the)189 2391 y(data)i(\(the)i(target)e(datat)o(yp)q(e)h(has)g(to)g
(b)q(e)h(sen)o(t)f(to)g(that)f(agen)o(t\).)23 b(In)17 b(the)f(general)h
(case,)g(data)e(is)189 2447 y(gathered)g(on)g(the)g(origin)h(no)q(de,)f(sen)o
(t)g(to)g(the)g(target)f(no)q(de,)i(and)f(scattered)g(there.)189
2522 y(A)i(generic)h(implemen)o(tation)g(of)f(put)g(on)g(top)g(of)g(comm)o
(unication)h(handlers)g(can)f(b)q(e)h(used)g(for)189 2579 y(p)q(ortabilit)o
(y)l(,)h(in)h(net)o(w)o(ork)o(ed)d(heterogenous)h(en)o(vironmen)o(ts.)30
b(A)18 b(p)q(ersisten)o(t)h(RMA)f(handler)h(is)189 2635 y(p)q(osted)c(at)g
(eac)o(h)h(pro)q(cess.)21 b(The)15 b(origin)i(pro)q(cess)e(sends)h(the)g(put)
g(request,)f(whic)o(h)h(includes)i(the)189 2691 y(target)e(displacemen)o(t,)k
(coun)o(t)d(and)h(datat)o(yp)q(e)f(argumen)o(ts,)g(to)g(the)h(target)f(pro)q
(cess,)h(where)g(it)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32
215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32
498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40
780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40
1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288
y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40
1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796
y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40
2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304
y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40
2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 8 10
bop 75 -100 a Fu(8)746 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)l
(TIONS)189 49 y Fu(is)j(receiv)o(ed)h(b)o(y)f(the)h(RMA)f(handler.)29
b(The)19 b(origin)f(next)g(sends)h(the)f(data,)g(using)h(a)e(standard)189
106 y Fn(MPI)p 274 106 14 2 v 15 w(SEND)g Fu(call,)h(with)e(the)h(origin)g
(address,)f(coun)o(t,)g(and)h(datat)o(yp)q(e)f(argumen)o(ts.)22
b(The)17 b(RMA)189 162 y(handler)23 b(at)f(the)h(target)e(pro)q(cess)i
(receiv)o(es)g(the)g(put)f(request)h(and)g(p)q(osts)f(a)g(corresp)q(onding)
189 219 y(receiv)o(e,)15 b(using)h(the)g(target)e(displacemen)o(t,)i(coun)o
(t)f(and)g(datat)o(yp)q(e)g(argumen)o(ts.)189 294 y(This)e(generic)i
(implemen)o(tation)f(can)g(b)q(e)g(optimized)g(in)g(v)m(arious)g(w)o(a)o(ys:)
k(the)13 b(RMA)h(handler)g(can)189 350 y(b)q(e)j(p)q(olling)h(co)q(de)f
(executing)h(on)e(a)g(separate)g(comm)o(unication)h(pro)q(cessor,)f(th)o(us)g
(a)o(v)o(oiding)h(the)189 407 y(o)o(v)o(erhead)f(of)g(in)o(terrupt)h(receiv)o
(es;)h(if)g(m)o(ultiple)g(pro)q(cesses)f(run)g(on)g(the)g(same)f(no)q(de,)i
(then)f(one)189 463 y(\(k)o(ernel?\))28 b(handler,)19 b(with)e(access)h(to)f
(their)h(memories,)g(can)g(serv)o(e)f(as)h(RMA)f(handler)i(for)e(all)189
520 y(pro)q(cesses)e(on)g(the)h(no)q(de;)f(and)h(so)e(on.)189
595 y(In)d(the)f(general)h(case,)f(a)g(put)h(call)g(requires)g(the)f
(transfer)g(of)g(a)g(descriptor)g(of)g(the)g(target)f(datat)o(yp)q(e)189
651 y(to)15 b(the)g(target)g(pro)q(cess,)h(and)f(its)h(in)o(terpretation)g(b)
o(y)g(an)f(agen)o(t)g(at)g(the)h(target)f(no)q(de.)21 b(The)16
b(de-)189 708 y(scriptor)11 b(format)f(should)j(allo)o(w)e(correct)g(in)o
(terpretation,)h(so)f(as)g(to)g(recreate)g(the)h(lo)q(cal)g(datat)o(yp)q(e)
189 764 y(that)k(w)o(ould)h(b)q(e)g(created)g(b)o(y)g(the)g(same)f(sequence)i
(of)f(calls.)26 b(One)17 b(example)h(of)e(suc)o(h)h(p)q(ortable)189
821 y(represen)o(tation)h(is)h(a)f(parse)g(tree)g(of)g(the)h(datat)o(yp)q(e)f
(de\014nition,)i(with)f(the)f(argumen)o(ts)g(of)g(the)189 877
y(t)o(yp)q(e)13 b(constructor)g(calls)h(at)f(the)h(tree)f(no)q(des.)20
b(An)14 b(example)g(of)f(a)g(non)h(p)q(ortable)g(represen)o(tation)189
933 y(is)h(a)f(t)o(yp)q(emap,)g(i.e.)20 b(a)14 b(list)i(of)e(\(basic)g(datat)
o(yp)q(e,)g(displacemen)o(t\))i(pairs.)k(In)15 b(this)g(\015attened)g(rep-)
189 990 y(resen)o(tation,)k(the)g(distinction)h(b)q(et)o(w)o(een)f(b)o(yte)g
(displacemen)o(ts)h(and)f(displacemen)o(ts)h(that)e(are)189
1046 y(m)o(ultiples)f(of)d(basic)i(datat)o(yp)q(e)f(exten)o(t)g(has)g(b)q
(een)h(lost.)189 1121 y(Implemen)o(tors)j(will)i(w)o(an)o(t)c(to)i(a)o(v)o
(oid)f(an)h(access)g(to)f(the)h(target)f(windo)o(w)h(coun)o(ter)g(when)g
Fn(ta)o(r-)189 1178 y(get)p 251 1178 V 16 w(increment)14 b(=)h(0)p
Fu(.)189 1253 y(A)j(high)h(qualit)o(y)g(implemen)o(tation)h(will)g(prev)o(en)
o(t)e(remote)g(accesses)g(to)g(memory)g(outside)h(the)189 1310
y(windo)o(w)14 b(that)g(w)o(as)g(exp)q(osed)h(b)o(y)f(the)h(pro)q(cess.)20
b(This,)14 b(b)q(oth)h(for)f(debugging)h(purp)q(oses,)g(and)g(for)189
1366 y(protection)e(with)h(clien)o(t-serv)o(er)h(co)q(des)f(that)f(use)h
(RMA.)f(I.e.,)g(a)h(high-qualit)o(y)h(implemen)o(tation)189
1422 y(will)h(c)o(hec)o(k)g(windo)o(w)f(b)q(ounds)h(on)g(eac)o(h)f(RMA)g
(call.)21 b(\()p Fk(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p
Fu(\))186 1529 y Fn(MPI)p 271 1529 V 16 w(IPUT\()10 b(o)o(rigin)p
525 1529 V 15 w(addr,)h(o)o(rigin)p 754 1529 V 15 w(count,)h(o)o(rigin)p
1005 1529 V 16 w(datat)o(yp)q(e,)g(ta)o(rget)p 1321 1529 V
16 w(rank,)f(ta)o(rget)p 1556 1529 V 16 w(disp,)h(ta)o(rget)p
1783 1529 V 16 w(count,)-170 1518 y Fh(>)p Fu(\()p Fp(June)p
Fu(\))75 1669 y Fn(ta)o(rget)p 190 1669 V 16 w(datat)o(yp)q(e,)17
b(increment,)c(comm)m(,)f(o)o(rigin)p 862 1669 V 15 w(request\))117
1746 y Fm(IN)155 b Fn(o)o(rigin)p 427 1746 V 16 w(addr)381
b Fm(address)15 b(of)f(origin)e(bu\013er)j(\(c)o(hoice\))117
1821 y(IN)155 b Fn(o)o(rigin)p 427 1821 V 16 w(count)360 b
Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117
1896 y(IN)155 b Fn(o)o(rigin)p 427 1896 V 16 w(datat)o(yp)q(e)302
b Fm(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(sen)o(t)i(\(handle\))117
1971 y(IN)155 b Fn(ta)o(rget)p 433 1971 V 17 w(rank)375 b Fm(rank)14
b(of)f(target)i(\(in)o(teger\))117 2046 y(IN)155 b Fn(ta)o(rget)p
433 2046 V 17 w(disp)384 b Fm(displacemen)o(t)20 b(from)f(start)i(of)f(windo)
o(w)g(to)h(target)g(bu\013er)905 2103 y(\(in)o(teger\))117
2178 y(IN)155 b Fn(ta)o(rget)p 433 2178 V 17 w(count)353 b
Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
2253 y(IN)155 b Fn(ta)o(rget)p 433 2253 V 17 w(datat)o(yp)q(e)295
b Fm(datat)o(yp)q(e)14 b(used)h(at)f(the)g(target)h(\(handle\))117
2328 y(IN)155 b Fn(increment)399 b Fm(incremen)o(t)14 b(for)f(target)i(windo)
o(w)e(coun)o(ter)i(\(in)o(teger\))117 2403 y(IN)155 b Fn(comm)466
b Fm(comm)o(unicator)11 b(used)k(for)f(comm)o(unicati)o(on)d(\(handle\))117
2479 y(OUT)108 b Fn(o)o(rigin)p 427 2479 V 16 w(request)330
b Fm(request)16 b(at)d(origin)g(pro)q(cess)j(\(handle\))75
2603 y Fq(int)47 b(MPI)p 269 2603 15 2 v 17 w(Iput\()23 b(void)h(*origin)p
717 2603 V 16 w(addr,)f(int)g(origin)p 1115 2603 V 17 w(count,)g(MPI)p
1371 2603 V 17 w(Datatype)393 2659 y(origin)p 540 2659 V 17
w(datatype,)f(int)i(target)p 1035 2659 V 16 w(rank,)f(int)h(target)p
1434 2659 V 16 w(disp,)f(int)1967 46 y Fj(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 9 11
bop 75 -100 a Fo(4.3.)34 b(PUT)15 b(AND)g(GET)g(CALLS)1177
b Fu(9)393 49 y Fq(target)p 540 49 15 2 v 17 w(count,)23 b(MPI)p
796 49 V 16 w(Datatype)g(target)p 1171 49 V 16 w(datatype,)g(int)h
(increment,)393 106 y(MPI)p 468 106 V 17 w(Comm)f(comm,)g(MPI)p
819 106 V 17 w(Request)g(*origin)p 1195 106 V 16 w(request\))75
192 y(MPI)p 150 192 V 17 w(IPUT\()g(ORIGIN)p 454 192 V 16 w(ADDR,)g(ORIGIN)p
757 192 V 17 w(COUNT,)g(ORIGIN)p 1085 192 V 16 w(DATATYPE,)g(TARGET)p
1484 192 V 16 w(RANK,)393 249 y(TARGET)p 540 249 V 17 w(DISP,)g(TARGET)p
844 249 V 16 w(COUNT,)g(TARGET)p 1171 249 V 16 w(DATATYPE,)g(INREMENT,)g
(COMM,)393 305 y(ORIGIN)p 540 305 V 17 w(REQUEST,)f(IERROR\))170
362 y(<type>)h(ORIGIN)p 484 362 V 17 w(ADDR\(*\))170 418 y(INTEGER)g(ORIGIN)p
508 418 V 17 w(COUNT,)g(ORIGIN)p 836 418 V 16 w(DATATYPE,)g(TARGET)p
1235 418 V 16 w(RANK,)g(TARGET)p 1538 418 V 17 w(DISP,)170
474 y(TARGET)p 317 474 V 17 w(COUNT,)g(TARGET)p 645 474 V 16
w(DATATYPE,)g(INCREMENT,)f(COMM,)i(ORIGIN)p 1450 474 V 16 w(REQUEST,)f
(IERROR)-170 539 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 606 y(This)17
b(is)g(the)f(non)o(blo)q(c)o(king)i(v)o(ersion)e(of)g(put.)23
b(It)17 b(is)f(iden)o(tical)j(to)c Fn(MPI)p 1386 606 14 2 v
16 w(PUT)p Fu(,)h(except)h(that)f(it)g(has)75 663 y(an)h Fn(o)o(rigin)p
249 663 V 15 w(request)h Fu(argumen)o(t.)24 b(The)17 b(origin)h(request)f
(completes)g(after)f(data)g(ha)o(v)o(e)g(b)q(een)i(copied)g(out)75
719 y(of)d(the)h(origin)h(memory)l(,)e(p)q(ossibly)j(b)q(efore)e(the)g(op)q
(eration)g(has)g(completed)h(in)f(target)f(memory)l(.)22 b(The)75
776 y(origin)16 b(pro)q(cess)g(ma)o(y)f(test)g(or)h(w)o(ait)f(for)g(the)h
(comm)o(unication)g(op)q(eration)g(to)f(complete)h(at)f(the)h(origin)75
832 y(using)g(MPI)f(test)g(and)g(w)o(ait)g(constructs.)75 954
y Ff(4.3.2)49 b(Non-blo)q(cking)19 b(and)d(Blo)q(cking)i(Get)112
1040 y Fn(MPI)p 197 1040 V 16 w(GET\()g(o)o(rigin)p 443 1040
V 16 w(addr,)h(o)o(rigin)p 681 1040 V 15 w(count,)h(o)o(rigin)p
940 1040 V 16 w(datat)o(yp)q(e,)g(ta)o(rget)p 1264 1040 V 16
w(rank,)f(ta)o(rget)p 1507 1040 V 17 w(disp,)h(ta)o(rget)p
1743 1040 V 16 w(count,)-170 1029 y Fh(>)p Fu(\()p Fp(June)p
Fu(\))75 1180 y Fn(ta)o(rget)p 190 1180 V 16 w(datat)o(yp)q(e,)d(increment,)c
(comm)m(\))117 1257 y Fm(OUT)108 b Fn(o)o(rigin)p 427 1257
V 16 w(addr)381 b Fm(address)15 b(of)f(origin)e(bu\013er)j(\(c)o(hoice\))117
1332 y(IN)155 b Fn(o)o(rigin)p 427 1332 V 16 w(count)360 b
Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
1407 y(IN)155 b Fn(o)o(rigin)p 427 1407 V 16 w(datat)o(yp)q(e)302
b Fm(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(receiv)o(ed)j(\(handle\))117
1482 y(IN)155 b Fn(ta)o(rget)p 433 1482 V 17 w(rank)375 b Fm(rank)14
b(of)f(target)i(\(in)o(teger\))117 1557 y(IN)155 b Fn(ta)o(rget)p
433 1557 V 17 w(disp)384 b Fm(displacemen)o(t)16 b(from)e(windo)o(w)i(start)g
(to)h(target)f(bu\013er)h(\(in-)905 1614 y(teger\))117 1689
y(IN)155 b Fn(ta)o(rget)p 433 1689 V 17 w(count)353 b Fm(n)o(um)o(b)q(er)13
b(of)h(elemen)o(ts)f(sen)o(t)i(\(in)o(teger\))117 1764 y(IN)155
b Fn(ta)o(rget)p 433 1764 V 17 w(datat)o(yp)q(e)295 b Fm(datat)o(yp)q(e)14
b(of)g(elemen)o(ts)f(sen)o(t)i(\(handle\))117 1839 y(IN)155
b Fn(increment)399 b Fm(incremen)o(t)14 b(for)f(target)i(windo)o(w)e(coun)o
(ter)i(\(in)o(teger\))117 1914 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11
b(used)k(for)f(comm)o(unicati)o(on)d(\(handle\))75 2039 y Fq(int)47
b(MPI)p 269 2039 15 2 v 17 w(get\()23 b(void)h(*origin)p 693
2039 V 16 w(addr,)f(int)h(origin)p 1092 2039 V 16 w(count,)f(MPI)p
1347 2039 V 17 w(Datatype)393 2095 y(origin)p 540 2095 V 17
w(datatype,)f(int)i(target,)f(int)g(target)p 1321 2095 V 16
w(disp,)h(int)393 2152 y(target)p 540 2152 V 17 w(count,)f(MPI)p
796 2152 V 16 w(Datatype)g(target)p 1171 2152 V 16 w(datatype,)g(int)h
(increment,)393 2208 y(MPI)p 468 2208 V 17 w(Comm)f(comm\))75
2295 y(MPI)p 150 2295 V 17 w(GET\()g(ORIGIN)p 430 2295 V 16
w(ADDR,)h(ORIGIN)p 734 2295 V 16 w(COUNT,)f(ORIGIN)p 1061 2295
V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p 1651 2295 V 16 w(DISP,)393
2351 y(TARGET)p 540 2351 V 17 w(COUNT,)g(TARGET)p 868 2351
V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)h(IERROR\))170 2408 y(<type>)g(ORIGIN)p
484 2408 V 17 w(ADDR\(*\))170 2464 y(INTEGER)g(ORIGIN)p 508
2464 V 17 w(COUNT,)g(ORIGIN)p 836 2464 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p
1426 2464 V 16 w(DISP,)170 2520 y(TARGET)p 317 2520 V 17 w(COUNT,)g(TARGET)p
645 2520 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(IERROR)-170
2585 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 2652 y(Similar)19
b(to)e Fn(MPI)p 468 2652 14 2 v 15 w(PUT)p Fu(,)h(except)f(that)g(the)h
(direction)g(of)f(data)g(transfer)g(is)h(rev)o(ersed.)26 b(Data)16
b(are)75 2709 y(copied)g(from)e(the)g(target)f(memory)h(to)g(the)h(origin.)40
b(The)15 b(target)e(windo)o(w)i(coun)o(ter)f(is)h(up)q(dated)h(after)-170
2695 y Fh(>)p Fu(\()p Fp(June)p Fu(\))-32 46 y Fj(1)-32 103
y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385
y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667
y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 10 12
bop 75 -100 a Fu(10)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fu(the)k(data)g(ha)o(v)o(e)f(b)q(een)j(copied)f(out)f(of)f
(the)i(memory)e(of)h(the)g(target)f(pro)q(cess.)32 b(The)19
b(call)h(completes)75 106 y(when)c(the)f(data)g(ha)o(v)o(e)f(arriv)o(ed)i(at)
e(the)h(origin,)h(and)f(after)g(the)g(target)f(coun)o(ter)h(has)g(b)q(een)h
(up)q(dated.)75 257 y Fn(MPI)p 160 257 14 2 v 16 w(IGET\()22
b(o)o(rigin)p 423 257 V 16 w(addr,)i(o)o(rigin)p 666 257 V
16 w(count,)h(o)o(rigin)p 931 257 V 15 w(datat)o(yp)q(e,)h(ta)o(rget,)f(ta)o
(rget)p 1410 257 V 16 w(disp,)g(ta)o(rget)p 1650 257 V 17 w(count,)g(ta)o(r-)
75 313 y(get)p 137 313 V 17 w(datat)o(yp)q(e,)16 b(increment,)d(comm)n(,)e(o)
o(rigin)p 809 313 V 16 w(request\))117 390 y Fm(OUT)108 b Fn(o)o(rigin)p
427 390 V 16 w(addr)381 b Fm(address)15 b(of)f(origin)e(bu\013er)j(\(c)o
(hoice\))117 465 y(IN)155 b Fn(o)o(rigin)p 427 465 V 16 w(count)360
b Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(receiv)o(ed)j(\(in)o(teger\))117
541 y(IN)155 b Fn(o)o(rigin)p 427 541 V 16 w(datat)o(yp)q(e)302
b Fm(datat)o(yp)q(e)14 b(of)g(elemen)o(ts)f(receiv)o(ed)j(\(handle\))117
616 y(IN)155 b Fn(ta)o(rget)475 b Fm(rank)14 b(of)f(target)i(\(in)o(teger\))
117 691 y(IN)155 b Fn(ta)o(rget)p 433 691 V 17 w(disp)384 b
Fm(displacemen)o(t)16 b(from)e(windo)o(w)i(start)g(to)h(target)f(bu\013er)h
(\(in-)905 747 y(teger\))117 822 y(IN)155 b Fn(ta)o(rget)p
433 822 V 17 w(count)353 b Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(sen)o
(t)i(\(in)o(teger\))117 898 y(IN)155 b Fn(ta)o(rget)p 433 898
V 17 w(datat)o(yp)q(e)295 b Fm(datat)o(yp)q(e)14 b(used)h(at)f(the)g(target)h
(\(handle\))117 973 y(IN)155 b Fn(increment)399 b Fm(incremen)o(t)14
b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))117
1048 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(used)k(for)f(comm)o
(unicati)o(on)d(\(handle\))117 1123 y(OUT)108 b Fn(o)o(rigin)p
427 1123 V 16 w(request)330 b Fm(request)16 b(at)d(origin)g(pro)q(cess)j
(\(handle\))75 1247 y Fq(int)47 b(MPI)p 269 1247 15 2 v 17
w(Iget\()23 b(void)h(*origin)p 717 1247 V 16 w(addr,)f(int)g(origin)p
1115 1247 V 17 w(count,)g(MPI)p 1371 1247 V 17 w(Datatype)393
1304 y(origin)p 540 1304 V 17 w(datatype,)f(int)i(target,)f(integer)f(target)
p 1416 1304 V 17 w(disp,)h(int)393 1360 y(target)p 540 1360
V 17 w(count,)g(MPI)p 796 1360 V 16 w(Datatype)g(target)p 1171
1360 V 16 w(datatype,)g(int)h(increment,)393 1417 y(MPI)p 468
1417 V 17 w(Comm)f(comm,)g(MPI)p 819 1417 V 17 w(Request)g(*origin)p
1195 1417 V 16 w(request\))75 1503 y(MPI)p 150 1503 V 17 w(Iget\()g(ORIGIN)p
454 1503 V 16 w(ADDR,)g(ORIGIN)p 757 1503 V 17 w(COUNT,)g(ORIGIN)p
1085 1503 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p 1675 1503 V
16 w(DISP,)393 1560 y(TARGET)p 540 1560 V 17 w(COUNT,)g(TARGET)p
868 1560 V 16 w(DATATYPE,)g(INCREMENT)f(COMM,)h(ORIGIN)p 1648
1560 V 17 w(REQUEST,)393 1616 y(IERROR\))170 1673 y(<type>)g(ORIGIN)p
484 1673 V 17 w(ADDR\(*\))170 1729 y(INTEGER)g(ORIGIN)p 508
1729 V 17 w(COUNT,)g(ORIGIN)p 836 1729 V 16 w(DATATYPE,)g(TARGET,)g(TARGET)p
1426 1729 V 16 w(DISP,)170 1786 y(TARGET)p 317 1786 V 17 w(COUNT,)g(TARGET)p
645 1786 V 16 w(DATATYPE,)g(INCREMENT,)f(COMM,)i(ORIGIN)p 1450
1786 V 16 w(REQUEST,)f(IERROR)-170 1850 y Fh(?)p Fu(\()p Fp(June)p
Fu(\))166 1917 y(This)16 b(is)f(the)h(non)o(blo)q(c)o(king)g(v)o(ersion)g(of)
e Fn(MPI)p 937 1917 14 2 v 16 w(GET)p Fu(.)75 2039 y Ff(4.3.3)49
b(Examples)75 2125 y Fp(Example)18 b(4.2)k Fu(W)l(e)f(sho)o(w)e(ho)o(w)h(to)g
(implemen)o(t)h(the)g(generic)g(indirect)h(assignmen)o(t)e
Fq(A)j(=)h(B\(map\))p Fu(,)75 2181 y(where)15 b Fq(A,)23 b(B)15
b Fu(and)f Fq(map)g Fu(ha)o(v)o(e)g(the)h(same)f(distribution,)i(and)e
Fq(map)g Fu(is)h(a)f(p)q(erm)o(utation.)20 b(T)l(o)14 b(simplify)l(,)i(w)o(e)
75 2238 y(assume)f(a)g(blo)q(c)o(k)h(distribution)h(with)e(equal)h(size)g
(blo)q(c)o(ks.)75 2344 y Fq(SUBROUTINE)22 b(MAP\()i(A,)f(B,)h(map,)f(m,)h
(comm,)f(p\))75 2401 y(INTEGER)g(m,)g(map\(m\),)g(comm,)h(p)75
2457 y(REAL)f(A\(m\),)g(B\(m\))75 2570 y(INTEGER)g(otypes\(p\),)f
(oindex\(m\),)94 b(!)24 b(used)g(to)f(construct)g(origin)g(datatypes)266
2626 y(tytpes\(p\),)f(tindex\(m\),)94 b(!)24 b(used)g(to)f(construct)g
(target)g(datatypes)266 2683 y(count\(p\),)g(blen\(m\),)f(total\(p\),)1967
46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 11 13
bop 75 -100 a Fo(4.3.)29 b(PUT)15 b(AND)g(GET)g(CALLS)1160
b Fu(11)266 49 y Fq(sizeofreal,)22 b(ierr)75 162 y(!)i(This)f(part)g(does)h
(the)f(work)g(that)h(depends)f(on)g(the)h(locations)e(of)i(B.)75
219 y(!)g(Can)f(be)h(reused)f(while)g(this)g(does)h(not)f(change)75
332 y(CALL)g(MPI_EXTENT\(MPI_REAL,)e(sizeofreal,)i(ierr\))75
444 y(DO)h(i=1,m)147 501 y(blen\(i\))e(=)i(1)75 557 y(END)f(DO)75
670 y(DO)h(i=1,p)147 727 y(count\(i\))e(=)i(1)75 783 y(END)f(DO)75
896 y(CALL)g(MPI_RMA_INIT\(B,)f(m*sizeofreal,)g(sizeofreal,)h(comm,)g
(getcomm,)g(ierr\))75 1009 y(!)h(This)f(part)g(does)h(the)f(work)g(that)h
(depends)f(on)g(the)h(value)f(of)g(map)h(and)75 1065 y(!)g(the)f(locations)g
(of)g(the)h(arrays.)75 1122 y(!)g(Can)f(be)h(reused)f(while)g(these)g(do)h
(not)f(change)75 1235 y(!)h(Compute)f(number)g(of)g(elements)g(to)h(be)f
(received)g(from)g(each)h(process)75 1348 y(DO)g(i=1,m)147
1404 y(j)f(=)h(map\(i\)/m)147 1461 y(count\(j\))e(=)i(count\(j\)+1)75
1517 y(END)f(DO)75 1630 y(total\(1\))g(=)g(0)75 1686 y(DO)h(i=2,p)147
1743 y(total\(i\))e(=)i(total\(i-1\))f(+)g(count\(i-1\))75
1856 y(DO)h(i=1,p)147 1912 y(count\(i\))e(=)i(0)75 1969 y(END)f(DO)75
2082 y(!)h(compute)f(origin)g(and)g(target)g(indices)g(of)h(elements)75
2138 y(!)g(received)e(from)i(each)f(process)75 2195 y(DO)h(i=1,m)147
2251 y(j)f(=)h(map\(i\)/m)147 2307 y(k)f(=)h(MOD\(map\(i\),m\))147
2364 y(count\(j\))e(=)i(count\(j\)+1)147 2420 y(oindex\(total\(j\))d(+)j
(count\(j\)\))f(=)g(i)147 2477 y(tindex\(total\(j\))e(+)j(count\(j\)\))f(=)g
(k)75 2533 y(END)g(DO)75 2646 y(!)h(create)f(origin)g(and)g(target)g
(datatypes)g(for)g(each)h(get)f(operation)75 2703 y(DO)h(i=1,p)-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 12 14
bop 75 -100 a Fu(12)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)147 49 y Fq(CALL)23 b(MPI_TYPE_INDEXED\(count\(i\))o(,)e(blen,)i
(oindex\(total\(i\)\),)743 106 y(MPI_REAL,)g(otype\(i\),)f(ierr\))147
162 y(CALL)h(MPI_TYPE_COMMIT\(otype\(i\),)d(ierr\))147 219
y(CALL)j(MPI_TYPE_INDEXED\(count\(i\))o(,)e(blen,)i(tindex\(total\(i\)\),)743
275 y(MPI_REAL,)g(ttype\(i\),)f(ierr\))147 332 y(CALL)h
(MPI_TYPE_COMMIT\(ttype\(i\),)d(ierr\))75 388 y(END)j(DO)75
501 y(!)h(this)f(part)g(does)h(the)f(assignment)g(itself)75
614 y(DO)h(i=1,p)147 670 y(CALL)f(MPI_GET)g(\(A,)g(1,)h(otype\(i\),)e(i,)i
(0,)g(ttype\(i\),)e(0,)i(getcomm,)f(ierr\))75 727 y(END)g(DO)75
783 y(RETURN)75 889 y Fp(Example)18 b(4.3)k Fu(A)16 b(simpler)g(v)o(ersion)g
(can)f(b)q(e)h(written,)g(that)e(do)q(es)i(not)f(require)h(to)f(built)h(a)f
(datat)o(yp)q(e)75 946 y(for)e(the)g(target)f(bu\013er.)19
b(But,)13 b(then)h(one)f(needs)h(a)f(separate)f(get)h(call)h(for)f(eac)o(h)g
(elemen)o(t,)h(as)f(illustrated)75 1002 y(b)q(elo)o(w.)28 b(This)19
b(co)q(de)f(is)g(m)o(uc)o(h)g(simpler,)h(but)f(m)o(uc)o(h)g(less)g(e\016cien)
o(t,)h(for)e(large)h(arra)o(ys,)f(unless)i(h/w)e(is)75 1059
y(fast)d(enough)i(to)e(transfer)h(e\016cien)o(tly)h(single)h(w)o(ords.)75
1165 y Fq(SUBROUTINE)22 b(MAP\()i(A,)f(B,)h(map,)f(m,)h(comm,)f(p\))75
1222 y(INTEGER)g(m,)g(map\(m\),)g(comm,)h(p)75 1278 y(REAL)f(A\(m\),)g
(B\(m\))75 1334 y(INTEGER)g(sizeofreal,)f(ierr)75 1447 y(CALL)h
(MPI_TYPE_EXTENT\(MPI_REAL,)e(sizeofreal,)h(ierr\))75 1504
y(CALL)h(MPI_RMA_INIT\(B,)f(m*sizeofreal,)g(sizeofreal,)h(comm,)g(getcomm,)g
(ierr\))75 1617 y(DO)h(i=1,m)147 1673 y(j)f(=)h(map\(i\)/m)147
1730 y(k)f(=)h(MOD\(map\(i\),m\))147 1786 y(CALL)f(MPI_GET\(A\(i\),)f(1,)i
(MPI_REAL,)e(j,)i(k,)f(1,)h(MPI_REAL,)f(0,)1101 1843 y(getcomm,)g(ierr\))75
1899 y(END)g(DO)75 1955 y(RETURN)75 2099 y Fr(4.4)59 b(Accumulate)18
b(F)n(unctions)75 2200 y Fu(It)e(is)g(often)g(useful)g(in)h(a)e(put)h(op)q
(eration)g(to)f(com)o(bine)i(the)f(data)f(mo)o(v)o(ed)g(to)g(the)h(target)f
(pro)q(cess)h(with)75 2257 y(the)f(data)f(that)g(resides)h(at)f(that)g(pro)q
(cess,)h(rather)f(then)h(replacing)h(the)f(data)f(there.)20
b(This)15 b(will)h(allo)o(w,)75 2313 y(for)f(example,)i(to)e(accum)o(ulate)i
(a)e(sum,)h(b)o(y)g(ha)o(ving)g(all)h(in)o(v)o(olv)o(ed)g(pro)q(cesses)g(add)
f(their)g(con)o(tribution)75 2369 y(to)f(the)g(sum)g(v)m(ariable)i(in)f(the)f
(memory)g(of)f(one)i(pro)q(cess.)198 2426 y Fn(MPI)p 283 2426
14 2 v 16 w(A)o(CCUMULA)l(TE\(o)o(rigin)p 721 2426 V 16 w(addr,)g(o)o(rigin)p
956 2426 V 15 w(count,)i(o)o(rigin)p 1213 2426 V 15 w(datat)o(yp)q(e,)g(ta)o
(rget)p 1534 2426 V 16 w(rank,)e(ta)o(rget)p 1774 2426 V 16
w(disp,)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967
272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959
554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780
y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959
1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959
1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959
1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959
1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959
1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959
2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959
2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959
2643 y(47)1959 2699 y(48)p eop
%%Page: 13 15
bop 75 -100 a Fo(4.4.)34 b(A)o(CCUMULA)l(TE)15 b(FUNCTIONS)1029
b Fu(13)-170 49 y Fh(>)p Fu(\()p Fp(June)p Fu(\))75 200 y Fn(ta)o(rget)p
190 200 14 2 v 16 w(count,)16 b(ta)o(rget)p 452 200 V 17 w(datat)o(yp)q(e,)g
(op,)f(comm)m(\))117 277 y Fm(IN)155 b Fn(o)o(rigin)p 427 277
V 16 w(addr)381 b Fm(initial)12 b(address)j(of)f(bu\013er)g(\(c)o(hoice\))117
353 y(IN)155 b Fn(o)o(rigin)p 427 353 V 16 w(count)360 b Fm(n)o(um)o(b)q(er)
13 b(of)h(elemen)o(ts)f(in)h(bu\013er)h(\(non)f(negativ)o(e)f(in)o(teger\))
117 428 y(IN)155 b Fn(o)o(rigin)p 427 428 V 16 w(datat)o(yp)q(e)302
b Fm(data)14 b(t)o(yp)q(e)g(of)f(eac)o(h)i(bu\013er)g(elemen)o(t)e
(\(handle\))117 503 y(IN)155 b Fn(ta)o(rget)p 433 503 V 17
w(rank)375 b Fm(rank)14 b(of)f(destination)h(pro)q(cess)i(in)d(comm)e(\(in)o
(teger\))117 578 y(IN)155 b Fn(ta)o(rget)p 433 578 V 17 w(disp)384
b Fm(displacemen)o(t)20 b(from)f(start)i(of)f(windo)o(w)g(to)h(target)g
(bu\013er)905 634 y(\(in)o(teger\))117 710 y(IN)155 b Fn(ta)o(rget)p
433 710 V 17 w(count)353 b Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(target)g(bu\013er)h(\(in)o(teger\))117 785 y(IN)155 b Fn(ta)o(rget)p
433 785 V 17 w(datat)o(yp)q(e)295 b Fm(datat)o(yp)q(e)14 b(used)h(at)f
(target)g(\(handle\))117 860 y(IN)155 b Fn(increment)399 b
Fm(incremen)o(t)14 b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))
117 935 y(IN)155 b Fn(op)541 b Fm(reduce)16 b(op)q(eration)e(\(handle\))117
1010 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(\(handle\))75
1135 y Fq(int)23 b(MPI)p 245 1135 15 2 v 17 w(Accumulate\(void*)f(origin)p
812 1135 V 16 w(addr,)h(int)h(origin)p 1211 1135 V 16 w(count,)393
1191 y(MPI)p 468 1191 V 17 w(Datatype)f(origin)p 844 1191 V
16 w(datatype,)g(int)g(target)p 1338 1191 V 17 w(rank,)g(int)g(target)p
1737 1191 V 17 w(disp,)393 1247 y(int)h(target)p 636 1247 V
16 w(count,)f(MPI)p 891 1247 V 17 w(Datatype)g(target)p 1267
1247 V 16 w(datatype,)g(int)g(increment,)393 1304 y(MPI)p 468
1304 V 17 w(Op)h(op,)f(MPI)p 724 1304 V 17 w(Comm)g(comm\))75
1390 y(MPI)p 150 1390 V 17 w(ACCUMULATE\(ORIGIN)p 575 1390
V 15 w(ADDR,)g(ORIGIN)p 877 1390 V 16 w(COUNT,)g(ORIGIN)p 1204
1390 V 17 w(DATATYPE,)f(TARGET)p 1603 1390 V 17 w(RANK,)393
1447 y(TARGET)p 540 1447 V 17 w(DISP,)h(TARGET)p 844 1447 V
16 w(COUNT,)g(TARGET)p 1171 1447 V 16 w(DATATYPE,)g(INCREMENT,)g(OP,)393
1503 y(COMM,)g(IERROR\))170 1560 y(<type>)g(ORIGIN)p 484 1560
V 17 w(ADDR\(*\))170 1616 y(INTEGER)g(ORIGIN)p 508 1616 V 17
w(COUNT,)g(ORIGIN)p 836 1616 V 16 w(DATATYPE,)g(TARGET)p 1235
1616 V 16 w(RANK,)g(TARGET)p 1538 1616 V 17 w(DISP,)170 1673
y(TARGET)p 317 1673 V 17 w(DATATYPE,)f(INCREMENT,)h(OP,)g(COMM,)h(IERROR)-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 14 16
bop 75 -100 a Fu(14)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fn(MPI)p 160 49 14 2 v 16 w(IA)o(CCUMULA)l(TE\(o)o(rigin)p
611 49 V 15 w(addr,)22 b(o)o(rigin)p 851 49 V 15 w(count,)h(o)o(rigin)p
1113 49 V 15 w(datat)o(yp)q(e,)g(ta)o(rget)p 1439 49 V 16 w(rank,)f(ta)o
(rget)p 1685 49 V 16 w(disp,)g(ta)o(r-)75 106 y(get)p 137 106
V 17 w(count,)16 b(ta)o(rget)p 400 106 V 16 w(datat)o(yp)q(e,)g(op,)f(comm)n
(,)c(o)o(rigin)p 931 106 V 16 w(request\))117 183 y Fm(IN)155
b Fn(o)o(rigin)p 427 183 V 16 w(addr)381 b Fm(initial)12 b(address)j(of)f
(bu\013er)g(\(c)o(hoice\))117 258 y(IN)155 b Fn(o)o(rigin)p
427 258 V 16 w(count)360 b Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(bu\013er)h(\(non)f(negativ)o(e)f(in)o(teger\))117 333 y(IN)155
b Fn(o)o(rigin)p 427 333 V 16 w(datat)o(yp)q(e)302 b Fm(data)14
b(t)o(yp)q(e)g(of)f(eac)o(h)i(bu\013er)g(elemen)o(t)e(\(handle\))117
408 y(IN)155 b Fn(ta)o(rget)p 433 408 V 17 w(rank)375 b Fm(rank)14
b(of)f(destination)h(pro)q(cess)i(in)d(comm)e(\(in)o(teger\))117
483 y(IN)155 b Fn(ta)o(rget)p 433 483 V 17 w(disp)384 b Fm(displacemen)o(t)20
b(from)f(start)i(of)f(windo)o(w)g(to)h(target)g(bu\013er)905
540 y(\(in)o(teger\))117 615 y(IN)155 b Fn(ta)o(rget)p 433
615 V 17 w(count)353 b Fm(n)o(um)o(b)q(er)13 b(of)h(elemen)o(ts)f(in)h
(target)g(bu\013er)h(\(in)o(teger\))117 690 y(IN)155 b Fn(ta)o(rget)p
433 690 V 17 w(datat)o(yp)q(e)295 b Fm(datat)o(yp)q(e)14 b(used)h(at)f
(target)g(\(handle\))117 765 y(IN)155 b Fn(increment)399 b
Fm(incremen)o(t)14 b(for)f(target)i(windo)o(w)e(coun)o(ter)i(\(in)o(teger\))
117 840 y(IN)155 b Fn(op)541 b Fm(reduce)16 b(op)q(eration)e(\(handle\))117
916 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(\(handle\))117
991 y(OUT)108 b Fn(o)o(rigin)p 427 991 V 16 w(request)330 b
Fm(request)16 b(at)d(origin)g(pro)q(cess)j(\(handle\))75 1115
y Fq(int)23 b(MPI)p 245 1115 15 2 v 17 w(Iaccumulate\(void*)f(origin)p
836 1115 V 16 w(addr,)h(int)h(origin)p 1235 1115 V 16 w(count,)393
1172 y(MPI)p 468 1172 V 17 w(Datatype)f(origin)p 844 1172 V
16 w(datatype,)g(int)g(target)p 1338 1172 V 17 w(rank,)g(int)g(target)p
1737 1172 V 17 w(disp,)393 1228 y(int)h(target)p 636 1228 V
16 w(count,)f(MPI)p 891 1228 V 17 w(Datatype)g(target)p 1267
1228 V 16 w(datatype,)g(int)g(increment,)393 1285 y(MPI)p 468
1285 V 17 w(Op)h(op,)f(MPI)p 724 1285 V 17 w(Comm)g(comm,)g(MPI)p
1075 1285 V 17 w(Request)g(*origin)p 1451 1285 V 16 w(request\))75
1371 y(MPI)p 150 1371 V 17 w(IACCUMULATE\(ORIGIN)p 599 1371
V 14 w(ADDR,)h(ORIGIN)p 901 1371 V 16 w(COUNT,)f(ORIGIN)p 1228
1371 V 16 w(DATATYPE,)g(TARGET)p 1627 1371 V 16 w(RANK,)393
1427 y(TARGET)p 540 1427 V 17 w(DISP,)g(TARGET)p 844 1427 V
16 w(COUNT,)g(TARGET)p 1171 1427 V 16 w(DATATYPE,)g(INCREMENT,)g(OP,)393
1484 y(COMM,)g(ORIGIN)p 683 1484 V 17 w(REQUEST,)f(IERROR)170
1540 y(<type>)h(ORIGIN)p 484 1540 V 17 w(ADDR\(*\))170 1597
y(INTEGER)g(ORIGIN)p 508 1597 V 17 w(COUNT,)g(ORIGIN)p 836
1597 V 16 w(DATATYPE,)g(TARGET)p 1235 1597 V 16 w(RANK,)g(TARGET)p
1538 1597 V 17 w(DISP,)170 1653 y(TARGET)p 317 1653 V 17 w(COUNT,)g(TARGET)p
645 1653 V 16 w(DATATYPE,)g(INCREMENT,)f(OP,)i(COMM,)f(ORIGIN)p
1545 1653 V 16 w(REQUEST,)170 1710 y(IERROR)-170 1774 y Fh(?)p
Fu(\()p Fp(June)p Fu(\))166 1842 y(Accum)o(ulate)e(the)f(con)o(ten)o(ts)g(of)
g(the)h(origin)g(bu\013er)f(\(as)g(de\014ned)h(b)o(y)g Fn(o)o(rigin)p
1517 1842 14 2 v 15 w(addr)p Fu(,)g Fn(o)o(rigin)p 1756 1842
V 16 w(count)75 1898 y Fu(and)c Fn(o)o(rigin)p 274 1898 V 15
w(datat)o(yp)q(e)p Fu(\))i(to)d(the)h(bu\013er)g(sp)q(eci\014ed)i(b)o(y)e
(argumen)o(ts)f Fn(ta)o(rget)p 1343 1898 V 16 w(count)i Fu(and)f
Fn(ta)o(rget)p 1684 1898 V 17 w(datat)o(yp)q(e)p Fu(,)75 1955
y(at)j(o\013set)g Fn(ta)o(rget)p 377 1955 V 16 w(disp)p Fu(,)j(in)f(the)f
(target)e(windo)o(w)j(sp)q(eci\014ed)g(b)o(y)f Fn(ta)o(rget)p
1326 1955 V 16 w(rank)g Fu(and)g Fn(comm)m Fu(,)e(using)j(the)75
2011 y(op)q(eration)g Fn(op)p Fu(.)38 b(This)22 b(is)g(lik)o(e)h
Fn(MPI)p 719 2011 V 15 w(PUT)f Fu(except)g(that)e(data)h(is)h(com)o(bined)h
(in)o(to)e(the)g(target)g(area)75 2067 y(instead)15 b(of)g(o)o(v)o(erwriting)
f(it.)40 b(The)15 b(call)h(completes)f(at)f(the)h(origin)g(after)f(the)h
(data)f(ha)o(v)o(e)g(b)q(een)i(copied)-170 2054 y Fh(>)p Fu(\()p
Fp(June)p Fu(\))75 2113 y(out)i(of)g(the)h(origin)g(bu\013er;)h(the)f(target)
e(coun)o(ter)i(is)g(incremen)o(ted)h(after)d(the)i(up)q(date)g(of)g(the)f
(target)75 2169 y(bu\013er)d(is)h(complete.)-170 2156 y Fh(?)p
Fu(\()p Fp(June)p Fu(\))166 2215 y(An)o(y)c(of)h(the)f(prede\014ned)i(op)q
(erations)f(for)f Fn(MPI)p 973 2215 V 16 w(REDUCE)h Fu(can)f(b)q(e)i(used.)19
b(User-de\014ned)14 b(functions)75 2271 y(cannot)h(b)q(e)h(used.)21
b(F)l(or)14 b(example,)i(if)g Fn(op)f Fu(is)h Fn(MPI)p 914
2271 V 16 w(SUM)p Fu(,)f(eac)o(h)g(elemen)o(t)h(of)f(the)g(origin)h(bu\013er)
g(is)f(added)75 2328 y(to)g(the)g(corresp)q(onding)h(elemen)o(t)g(in)g(the)f
(target,)f(replacing)i(the)g(former)e(v)m(alue)j(in)f(the)f(target.)166
2384 y(The)e Fn(ta)o(rget)p 372 2384 V 17 w(datat)o(yp)q(e)h
Fu(should)g(b)q(e)g(a)f(a)g(basic)h(datat)o(yp)q(e,)e(or)h(a)g(deriv)o(ed)h
(datat)o(yp)q(e)f(where)g(all)h(basic)75 2441 y(comp)q(onen)o(ts)j(are)f(of)g
(the)g(same)h(basic)g(datat)o(yp)q(e.)23 b(The)17 b(op)q(eration)g
Fn(op)f Fu(applies)j(to)c(elemen)o(ts)j(of)e(that)75 2497 y(basic)g(t)o(yp)q
(e.)166 2554 y(A)i(new)g(prede\014ned)i(op)q(eration,)e Fl(MPI)p
833 2554 13 2 v 14 w(SW)m(AP)p Fu(,)f(is)h(de\014ned.)30 b(It)18
b(corresp)q(onds)g(to)g(the)g(asso)q(ciativ)o(e)75 2610 y(function)g
Fg(f)5 b Fu(\()p Fg(a;)j(b)p Fu(\))15 b(=)h Fg(b)p Fu(;)i(i.e.,)f(the)g
(curren)o(t)h(v)m(alue)g(in)g(the)g(target)e(memory)h(is)h(replaced)g(b)o(y)f
(the)h(v)m(alue)75 2666 y(supplied)g(b)o(y)d(the)g(origin.)1967
46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 15 17
bop 75 -100 a Fo(4.4.)34 b(A)o(CCUMULA)l(TE)15 b(FUNCTIONS)1029
b Fu(15)189 49 y Fk(A)n(dvic)n(e)16 b(to)h(users.)46 b Fn(MPI)p
637 49 14 2 v 16 w(PUT)16 b Fu(is)h(a)e(sp)q(ecial)j(case)e(of)g
Fn(MPI)p 1226 49 V 16 w(A)o(CCUMULA)l(TE)p Fu(,)g(with)h(the)f(op)q(era-)189
106 y(tion)h Fl(MPI)p 361 106 13 2 v 14 w(SW)m(AP)f Fu(Note,)g(ho)o(w)o(ev)o
(er,)g(that)g Fn(MPI)p 1001 106 14 2 v 16 w(PUT)h Fu(and)f
Fn(MPI)p 1296 106 V 16 w(A)o(CCUMULA)l(TE)h Fu(ha)o(v)o(e)g(sligh)o(tly)189
162 y(di\013eren)o(t)e(atomicit)o(y)g(seman)o(tics)g({)g(see)h(Section)g
(4.7.)j(\()p Fk(End)c(of)i(advic)n(e)f(to)g(users.)p Fu(\))166
351 y Fi(Alternativ)o(es)o(:)42 b Fm(W)m(e)16 b(could)f(allo)o(w)g(user)h
(de\014ned)h(functions.)24 b(If)16 b(w)o(e)g(do)f(so,)h(w)o(e)g(w)o(an)o(t)f
(to)h(lea)o(v)o(e)g(to)f(the)75 408 y(implemen)o(tor)c(the)j(c)o(hoice)h(of)e
(ev)n(aluating)f(the)j(function)e(either)i(at)e(the)i(origin)d(or)i(at)g(the)
g(target.)k(Ho)o(w)o(ev)o(er,)c(to)75 464 y(do)c(so,)h(w)o(e)g(m)o(ust)e
(pair)h(the)i(op)q(eration)e(de\014ned)i(at)e(the)h(origin)e(with)i(an)f(op)q
(eration)g(de\014ned)i(at)e(the)h(target.)18 b(In)10 b(the)75
520 y(case)k(of)f(a)g(collectiv)o(e)g(reduce)i(op)q(eration)e(this)g(pairing)
g(is)g(implicit)d(as)k(eac)o(h)f(pro)q(cess)i(calls)e Fl(MPI)p
1607 520 13 2 v 15 w(REDUCE)e Fm(with)75 577 y(its)k(o)o(wn)g(op)q(eration)f
(argumen)o(t.)21 b(But)15 b(accum)o(ulate)f(is)h(single-sided,)g(so)g(that)g
(another)h(pairing)d(mec)o(hanism)g(is)75 633 y(needed.)19
b(W)m(e)11 b(w)o(ould)g(need)h(an)f(additional)f(collectiv)o(e)h(call,)g(to)h
(create)h(the)f(\\equiv)n(alence)f(classes")i(of)e(op)q(erators.)75
879 y Fp(Example)18 b(4.4)k Fu(W)l(e)d(w)o(an)o(t)e(to)g(compute)i
Fq(B)p Fu(\()p Fq(j)p Fu(\))e(=)976 847 y Fe(P)1020 890 y Fd(map)p
Fc(\()p Fd(i)p Fc(\)=)p Fd(j)1158 879 y Fq(A)p Fu(\()p Fq(i)p
Fu(\).)28 b(The)18 b(arra)o(ys)f Fq(A,)24 b(B)18 b Fu(and)g
Fq(map)g Fu(are)75 935 y(distributed)f(in)f(the)f(same)g(manner.)20
b(W)l(e)15 b(write)g(the)h(simple)g(v)o(ersion.)75 1042 y Fq(SUBROUTINE)22
b(SUM\()i(A,)f(B,)h(map,)f(m,)h(comm,)f(p\))75 1098 y(INTEGER)g(m,)g
(map\(m\),)g(comm,)h(p)75 1154 y(REAL)f(A\(m\),)g(B\(m\))75
1267 y(CALL)g(MPI_RMA_INIT\(B,)f(m,)i(MPI_REAL,)e(comm,)i(sumcomm,)e(ierr\))
75 1380 y(DO)i(i=1,m)147 1437 y(j)f(=)h(map\(i\)/m)147 1493
y(k)f(=)h(MOD\(map\(i\),m\))147 1550 y(CALL)f(MPI_ACCUMULATE\(A\(i\),)e(1,)j
(MPI_REAL,)e(j,)i(k,)f(1,)h(0,)g(MPI_SUM,)1149 1606 y(sumcomm,)f(ierr\))75
1663 y(END)g(DO)75 1719 y(RETURN)166 1825 y Fu(This)18 b(co)q(de)h(is)f(iden)
o(tical)i(to)d(the)h(co)q(de)h(in)g(Example)f(4.3,)f(page)h(12,)g(except)g
(that)f(a)h(call)h(to)e(get)75 1882 y(has)f(b)q(een)h(replaced)h(b)o(y)e(a)f
(call)j(to)d(accum)o(ulate.)23 b(\(Note)16 b(that,)f(if)h Fq(map)g
Fu(is)h(one-to-one,)f(then)g(the)g(co)q(de)75 1938 y(computes)c
Fq(B)h Fu(=)g Fq(A)p Fu(\()p Fq(map)473 1922 y Fb(\000)p Fd(1)515
1938 y Fu(\),)f(whic)o(h)h(the)f(rev)o(erse)g(assignmen)o(t)g(to)f(he)i(one)f
(computed)h(in)f(that)g(example.\))75 1995 y(In)j(a)g(similar)g(manner,)g(w)o
(e)f(can)h(replace)h(the)f(call)g(to)f(get)h(b)o(y)f(a)h(call)g(to)f(accum)o
(ulate)i(in)f(Example)g(4.2,)75 2051 y(page)g(10,)g(th)o(us)g(p)q(erforming)g
(the)h(computation)f(with)h(only)g(one)f(comm)o(unication)h(b)q(et)o(w)o(een)
g(an)o(y)f(pair)75 2108 y(of)g(pro)q(cesses.)75 2229 y Ff(4.4.1)49
b(RMW)17 b(op)q(erations)75 2315 y Fu(Remote)12 b(Read)p 343
2315 14 2 v 17 w(Mo)q(dify)p 501 2315 V 16 w(W)l(rite)g(op)q(erations)g(pro)o
(vide)g(atomic)f(test)g(and)h(up)q(date)g(of)f(a)g(remote)g(v)m(ariable.)75
2371 y(They)17 b(can)h(b)q(e)g(used)f(to)g(implemen)o(t)i(sync)o(hronization)
f(op)q(erations)f(suc)o(h)h(as)e(semaphores,)i(coun)o(ting)75
2428 y(semaphores,)13 b(etc.)20 b(T)l(o)13 b(simplify)i(implemen)o(tation,)g
(a)e(Read)p 1131 2428 V 17 w(Mo)q(dify)p 1289 2428 V 17 w(W)l(rite)h(op)q
(erations)g(op)q(erates)f(on)75 2484 y(a)i(unique)i(in)o(teger)e(v)m
(ariable.)196 2541 y Fn(MPI)p 281 2541 V 16 w(RMW\()g(inva)o(r,)f(outva)o(r,)
h(ta)o(rget,)f(disp,)i(op,)f(comm)10 b(\))-32 46 y Fj(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 16 18
bop 75 -100 a Fu(16)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)-170 49 y Fh(>)p Fu(\()p Fp(June)p Fu(\))117 221 y
Fm(IN)155 b Fn(inva)o(r)495 b Fm(input)14 b(bu\013er)h(to)f(RMW)f(op)q
(eration)g(\(c)o(hoice\))117 296 y(OUT)108 b Fn(outva)o(r)467
b Fm(output)14 b(bu\013er)h(of)e(RMW)h(op)q(eration)f(\(c)o(hoice\))117
371 y(IN)155 b Fn(ta)o(rget)475 b Fm(rank)14 b(of)f(target)i(pro)q(cess)g
(\(in)o(teger\))117 446 y(IN)155 b Fn(disp)513 b Fm(displacemen)o(t)13
b(to)h(target)g(bu\013er)h(\(in)o(teger\))117 521 y(IN)155
b Fn(op)541 b Fm(read-mo)q(dify-write)12 b(op)q(eration)i(\(handle\))117
597 y(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(\(handle\))75
721 y Fq(int)23 b(MPI)p 245 721 15 2 v 17 w(Rmw\()h(int)f(invar,)g(int)g
(*outvar,)g(int)h(target,)f(int)g(disp,)g(MPI)p 1551 721 V
17 w(Op)h(op,)393 778 y(MPI)p 468 778 V 17 w(Comm)f(comm)h(\))75
864 y(MPI)p 150 864 V 17 w(RMW\()f(INVAR,)g(OUTVAR,)g(TARGET,)g(DISP,)g(OP,)h
(COMM,)f(IERROR\))170 920 y(INTEGER)g(INVAR,)g(OUTVAR,)g(TARGET,)g(DISP,)g
(OP,)h(COMM,)f(IERROR)166 1007 y Fu(The)15 b(call)i(has)e(the)g(same)g
(e\013ect)g(as)g(an)g(atomic)g(execution)h(of)f(the)g(co)q(de)75
1101 y Fq(outvar[i])94 b(=)24 b(targetvar[i];)75 1157 y(targetvar[i])e(=)i
(targetvar[i])e(op)i(invar[i];)75 1251 y Fu(I.e.,)14 b(the)g(call)i(returns)e
(the)g(old)h(v)m(alue)h(of)d(the)i(target)e(v)m(ariable)i(and)g(up)q(date)g
(it,)f(according)h(the)f(to)g(the)75 1307 y(op)q(eration)k
Fn(op)p Fu(.)29 b(The)19 b(call)g(completes)g(after)e(the)h(target)f(v)m
(ariable)j(has)e(b)q(een)h(up)q(dated)g(and)f(the)g(old)75
1364 y(v)m(alue)e(returned)g(at)f(the)g(origin.)166 1420 y(The)g(follo)o
(wing)h(op)q(erations)g(are)e(supp)q(orted:)21 b Fl(MPI)p 1057
1420 13 2 v 14 w(SW)m(AP)m(,)12 b(MPI)p 1281 1420 V 14 w(SUM)k
Fu(and)f Fl(MPI)p 1562 1420 V 14 w(BAND)p Fu(.)166 1477 y(This)h(call)h(can)e
(b)q(e)i(used)f(to)e(implemen)o(t)j(a)e(test)g(and)h(set)f(op)q(eration.)21
b(Assume)16 b(that)f Fn(ta)o(rgetva)o(r)f Fu(is)75 1533 y(a)j(Bo)q(olean)i
(lo)q(c)o(k.)27 b(A)18 b(call)h(to)e Fn(MPI)p 693 1533 14 2
v 16 w(RMW\(v,)g Fg(:)8 b(:)g(:)e Fn(,)18 b(MPI)p 1064 1533
V 15 w(SW)l(AP)l(,)h Fg(:)8 b(:)g(:)d Fn(\))17 b Fu(with)h
Fn(v=1)g Fu(sets)f(the)h(lo)q(c)o(k)g(to)f(1)75 1590 y(and)e(returns)f(the)g
(previous)i(lo)q(c)o(k)f(v)m(alue.)20 b(The)15 b(lo)q(c)o(k)g(call)h
(succeeded)g(if)f(the)f(previous)h(v)m(alue)h(w)o(as)e(0.)19
b(A)75 1646 y(call)g(to)e Fn(MPI)p 304 1646 V 15 w(RMW\(v,)h
Fg(:)8 b(:)g(:)d Fn(,)18 b(MPI)p 674 1646 V 16 w(SW)l(AP)l(,)g
Fg(:)8 b(:)g(:)e Fn(,)18 b(MPI)p 1014 1646 V 15 w(SW)l(AP)l(,)g
Fg(:)8 b(:)g(:)e Fn(\))17 b Fu(with)h Fn(v=0)g Fu(resets)f(the)h(lo)q(c)o(k)g
(to)f(0.)75 1703 y(The)g(call)i(can)e(also)g(b)q(e)g(used)h(to)f(implemen)o
(t)h(a)f(F)l(etc)o(h)p 1037 1703 V 16 w(Add)h(op)q(eration)f(\(with)g
Fl(MPI)p 1557 1703 13 2 v 14 w(SUM)p Fu(\),)g(coun)o(ting)75
1759 y(lo)q(c)o(ks,)e(etc.)189 1865 y Fk(R)n(ationale.)58 b
Fu(E\016cien)o(t)18 b(sync)o(hronization)i(requires)e(the)h(use)f(of)g(RMW)g
(op)q(erations,)h(suc)o(h)f(as)189 1922 y(de\014ned)d(here;)g(it)f(requires,)
h(in)f(a)g(preemptiv)o(e)h(execution)g(en)o(vironmen)o(t,)f(in)o(tegration)g
(with)h(the)189 1978 y(sc)o(heduler.)29 b(W)l(e)18 b(ha)o(v)o(e)g(not)g
(attempted)f(to)g(handle)j(the)e(later,)g(as)g(MPI)f(has)h(so)g(far)f
(refrained)189 2035 y(from)h(pro)o(viding)j(sc)o(heduling)g(services,)g(or)e
(from)f(making)i(assumptions)f(on)h(the)f(underlying)189 2091
y(sc)o(heduling)e(p)q(olicies.)22 b(\()p Fk(End)16 b(of)g(r)n(ationale.)p
Fu(\))166 2273 y Fi(Alternativ)o(es)o(:)47 b Fm(W)m(e)17 b(ma)o(y)e(prefer)j
(to)f(use)h Fl(long)f(int)g Fm(argumen)o(ts,)g(rather)h(than)f
Fl(int)p Fm(.)27 b(Or)18 b(should)f(it)f(b)q(e)75 2323 y Fl(MPI)p
152 2323 V 14 w(Aint)p Fm(?)166 2380 y(W)m(e)f(could)g(ha)o(v)o(e)h(a)f(blo)q
(c)o(king)f(RMW)h(call,)g(where)h(b)q(oth)g Fl(inva)o(r)g Fm(and)f
Fl(outva)o(r)h Fm(are)g(inputs:)21 b(the)16 b(call)f(blo)q(c)o(ks)75
2436 y(un)o(til)10 b(the)i(target)f(v)n(ariable)f(equals)h(to)g
Fl(outva)o(r)p Fm(.)18 b(E.g.,)10 b(a)g(call)h(to)g Fl(MPI)p
1146 2436 V 14 w(W)o(ait)p 1243 2436 V 15 w(RMW\(1,)g(b,)h
Fa(:)7 b(:)g(:)e Fl(,)11 b(MPI)p 1618 2436 V 14 w(SUM\))g Fm(returns)75
2493 y(when)i(the)h(v)n(alue)e(of)g(the)h(target)g(v)n(ariable)f(is)h(b,)f
(and)h(then)g(incremen)o(ts)g(it)f(b)o(y)h(one.)18 b(Tw)o(o)12
b(suc)o(h)h(coun)o(ters)h(can)f(b)q(e)75 2549 y(used)i(to)f(implem)o(en)o(t)e
(a)h(fair)g(semaphore,)g(using)h(the)h(co)q(de)f(sho)o(wn)g(b)q(elo)o(w.)1967
46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 17 19
bop 75 -100 a Fo(4.5.)34 b(COMPLETION)16 b(OF)f(RMA)h(OPERA)l(TIONS)807
b Fu(17)75 49 y Fq(void)23 b(lock\(\))75 106 y({)75 162 y(/*)h(get)f(ticket)g
(in)h(the)f(queue)g(*/)75 219 y(MPI_RMW\(1,)f(&myticket,)h(target,)g
(waitcount,)f(MPI_SUM,)h(comm\);)75 275 y(/*)h(Wait)f(for)g(turn)h(*/)75
332 y(MPI_Wait_RMW\(1,)e(myticket,)g(target,)h(servicedcount,)f(MPI_SUM,)h
(comm\);)75 388 y(})75 501 y(void)g(unlock\(\))75 557 y({)75
614 y(MPI_RMW\(1,)f(&dontcare,)h(target,)g(servicedcount,)f(MPI_SUM,)h
(comm\);)75 670 y(})-170 754 y Fh(?)p Fu(\()p Fp(June)p Fu(\))75
909 y Fr(4.5)59 b(Completion)19 b(of)g(RMA)g(Op)r(erations)75
1010 y Fu(The)d(de\014nition)i(of)e(the)g(put)h(and)f(accum)o(ulate)h(op)q
(erations)f(implies)i(that)e(the)g(comm)o(unication)h(ma)o(y)75
1067 y(con)o(tin)o(ue)d(after)e(the)i(call)g(completed)g(at)f(the)g(origin.)
20 b(Also,)14 b(MPI)f(do)q(es)g(not)g(guaran)o(tee)g(the)g(execution)75
1123 y(order,)e(at)g(the)g(target)f(memory)l(,)i(or)e(t)o(w)o(o)g(subsequen)o
(tly)j(issued)f(puts.)19 b(This)11 b(is)h(similar)h(to)d(the)i(situation)75
1180 y(in)19 b(mo)q(dern)f(shared)g(memory)g(m)o(ultipro)q(cessors,)g(where)h
(the)f(completion)h(of)e(store)g(op)q(eration)i(do)q(es)75
1236 y(not)c(imply)h(that)e(the)h(stored)g(v)m(alue)h(is)g(visible)h(to)d
(other)h(pro)q(cessors,)f(and)h(where)g(stores)g(ma)o(y)f(go)g(out)75
1292 y(of)i(order.)22 b(On)17 b(suc)o(h)g(systems,)e(a)h(separate)g(sync)g
(call)h(is)g(needed)h(to)d(ensure)i(that)e(a)h(store)g(op)q(eration)75
1349 y(is)i(\\committed",)e(and)i(will)h(precede)f(subsequen)o(t)f(stores)g
(in)h(global)g(memory)l(.)25 b(A)17 b(useful)h(construct)75
1405 y(is)g(a)g(collectiv)o(e)i(call)f(that)e(ensures)i(that)e(all)i
(previously)g(issued)g(RMA)f(calls)h(ha)o(v)o(e)f(completed)h(in)f(a)75
1462 y(group.)i(This)c(will)h(allo)o(w)f(to)f(implemen)o(t)h(algorithms)g
(that)e(consists)i(of)f(successiv)o(e)h(phases,)g(where)f(a)75
1518 y(new)e(phase)g(starts)e(when)i(all)h(comm)o(unications)f(p)q(ertaining)
h(to)f(the)f(previous)i(phase)f(ha)o(v)o(e)f(completed)75 1575
y(\(in)k(the)f(st)o(yle)g(of)g(the)g(Bulk)i(Sync)o(hronous)e(P)o(arallel)h
(mo)q(del\).)166 1631 y(The)f(MPI)h(collectiv)o(e)h(function)f
Fn(MPI)p 824 1631 14 2 v 16 w(BARRIER\(comm)n(\))p Fu(,)c(when)k(called)h
(with)e(a)g(comm)o(unicator)75 1688 y Fn(comm)10 b Fu(used)17
b(for)f(RMA)g(comm)o(unication,)h(has)f(the)g(additional)i(seman)o(tics)e(of)
g(a)g(fence)h(with)f(resp)q(ect)75 1744 y(to)f(RMA)g(calls)i(that)d(use)i
(this)g(comm)o(unicator.)k(All)d(the)e(RMA)h(calls)g(issued)h(on)e(the)g
(comm)o(unicator)75 1801 y(b)q(efore)d(the)f(call)i(to)d Fn(MPI)p
498 1801 V 16 w(BARRIER)j Fu(will)g(complete)f(at)f(the)g(target)f(b)q(efore)
i Fn(MPI)p 1455 1801 V 16 w(BARRIER)h Fu(completes.)75 1857
y(This)i(includes)i(non)o(blo)q(c)o(king)f(calls)g(started)d(b)q(efore)i(the)
g(call)h(to)e(barrier,)g(ev)o(en)h(if)g(the)g(matc)o(hing)f(w)o(ait)-170
1844 y Fh(>)p Fu(\()p Fp(June)p Fu(\))75 1902 y(or)h(test)f(call)j(has)e(not)
g(y)o(et)f(b)q(een)j(issued.)-170 1889 y Fh(?)p Fu(\()p Fp(June)p
Fu(\))166 2031 y Fi(Missing:)37 b Fm(If)15 b(MPI2)f(appro)o(v)o(es)h(a)f(non)
o(blo)q(c)o(king)f(barrier)i Fl(MPI)p 1190 2031 13 2 v 15 w(IBARRIER\(com)n
(m)m(,)c(request\))p Fm(,)16 b(then,)f(this)75 2087 y(will)d(apply)i(to)f(it)
h(as)g(w)o(ell.)166 2226 y Fu(It)f(is)h(also)f(useful)h(to)e(ha)o(v)o(e)h(a)g
(\\fence")g(mec)o(hanism)h(that)e(guaran)o(tees)h(completion)h(of)f(puts)g
(issued)75 2283 y(b)o(y)g(one)g(pro)q(cess.)19 b(This)14 b(is)f(useful)h(if)g
(a)f(pro)q(cess)g(has)g(to)f(mo)q(dify)h(atomically)h(a)f(structure)g(stored)
f(in)i(the)75 2339 y(memory)g(of)h(other)f(pro)q(cess\(es\).)19
b(The)d(op)q(eration)f(ma)o(y)f(consists)h(of)f(a)g(lo)q(c)o(k,)h(follo)o(w)o
(ed)h(b)o(y)e(a)h(sequence)75 2396 y(of)i(gets)g(and)h(puts,)f(follo)o(w)o
(ed)h(b)o(y)f(an)h(unlo)q(c)o(k)g(op)q(eration.)27 b(One)18
b(has)g(to)e(ensure)i(that)f(all)h(up)q(dates)g(of)75 2452
y(remote)d(data)f(ha)o(v)o(e)h(o)q(ccurred)h(b)q(efore)f(the)h(unlo)q(c)o(k)g
(op)q(eration)f(is)h(issued.)-32 46 y Fj(1)-32 103 y(2)-32
159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32
441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40
724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40
1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232
y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40
1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740
y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40
2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248
y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40
2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 18 20
bop 75 -100 a Fu(18)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fn(MPI)p 160 49 14 2 v 16 w(FENCE\(comm)m(,)d(rank\))117
126 y Fm(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11 b(used)k(in)f(RMA)f
(\(handle\))117 202 y(IN)155 b Fn(rank)504 b Fm(rank)14 b(of)f(target)i(pro)q
(cess)g(\(in)o(teger\))75 326 y Fq(int)23 b(MPI)p 245 326 15
2 v 17 w(Fence\(MPI)p 478 326 V 16 w(Comm)g(comm,)h(int)f(rank\))75
413 y(MPI)p 150 413 V 17 w(FENCE\()g(COMM,)g(RANK,)g(IERROR\))170
469 y(INTEGER)g(COMM,)h(RANK,)f(IERROR)166 555 y Fu(The)18
b(call)i(returns)e(when)h(all)g(previous)g(RMA)f(calls)h(issued)h(b)o(y)e
(the)g(caller)h(on)g(this)f(comm)o(uni-)75 612 y(cator)e(to)g(the)g(target)g
(pro)q(cess)h Fn(rank)f Fu(ha)o(v)o(e)g(completed)i(in)f(the)g(memory)f(of)g
(pro)q(cess)h Fn(rank)p Fu(.)24 b(If)17 b Fn(rank)g(=)75 668
y(MPI)p 160 668 14 2 v 16 w(ANY)p 268 668 V 17 w(SOURCE)f Fu(then)g(the)f
(call)h(returns)f(when)g(all)h(RMA)g(calls)g(issued)g(b)o(y)f(the)g(caller)h
(on)f Fn(comm)75 725 y Fu(ha)o(v)o(e)g(completed.)166 864 y
Fi(Alternativ)o(es)o(:)34 b Fm(Require)14 b(put)g(and)f(accum)o(ulate)g(to)h
(b)q(e)g(complete)f(at)g(the)i(target)f(memory)m(,)c(b)q(efore)k(the)75
920 y(op)q(eration)k(completes)g(lo)q(cally)m(.)29 b(W)m(e)18
b(should)g(discuss)i(this)e(alternativ)o(e)g(seriously:)27
b(It)19 b(do)q(es)g(not)f(a\013ect)h(get,)75 977 y(whic)o(h)12
b(is)h(lik)o(ely)e(to)h(b)q(e)h(the)g(most)e(frequen)o(tly)i(used)g(RMA)g(op)
q(eration)f(\(assuming)f(an)h(o)o(wner)h(compute)f(mo)q(del\);)75
1033 y(users)19 b(can)f(still)f(o)o(v)o(erlap)h(comm)o(unicati)o(on)d(with)j
(computation,)e(using)i(non)o(blo)q(c)o(king)f(put)h(and)f(accum)o(ulate;)75
1090 y(fence)12 b(and)e(mo)q(di\014ed)f(barrier)i(are)g(not)g(needed,)h(an)o
(ymore;)e(the)h(programming)c(mo)q(del)i(is)i(easier)g(to)g(explain;)f(and)75
1146 y(shared)i(memory)d(implem)o(en)o(tations)g(\(or,)i(more)f(generally)m
(,)h(implem)o(en)o(tations)e(that)i(do)g(not)g(ha)o(v)o(e)g(in)o(termediate)
75 1203 y(bu\013ering\))j(are)f(not)g(a\013ected.)20 b(On)13
b(the)h(other)g(hand,)e(distributed)i(memory)d(implemen)o(tati)o(ons)g(with)i
(bu\013ering)75 1259 y(will)f(need)j(an)f(additional)e(end-to-end)j(ac)o(kno)
o(wledgmen)o(t.)189 1448 y Fk(R)n(ationale.)39 b Fu(W)l(e)16
b(do)e(not)h(require)h(that)e(the)h(calls)h(complete)g(also)f(at)f(the)h
(origin)h(at)e(a)h(fence)g(or)189 1505 y(barrier)i({)g(the)g(caller)h(can)f
(ensure)h(this)f(b)o(y)g(using)h(a)e(W)l(aitall)i(call.)27
b(This)17 b(c)o(hoice)h(signi\014can)o(tly)189 1561 y(simpli\014es)f
(implemen)o(tation.)k(\()p Fk(End)16 b(of)g(r)n(ationale.)p
Fu(\))189 1667 y Fk(A)n(dvic)n(e)k(to)i(users.)76 b Fu(A)21
b(fence)h(op)q(eration)f(ma)o(y)g(require)h(comm)o(unication)g(with)f(the)g
(target)189 1724 y(pro)q(cess,)15 b(or)f(with)i(all)g(pro)q(cesses)g(in)g
(the)f(comm)o(unication)h(group,)e(if)i(no)f(target)f(w)o(as)g(sp)q
(eci\014ed.)189 1780 y(\()p Fk(End)h(of)i(advic)n(e)f(to)g(users.)p
Fu(\))189 1886 y Fk(A)n(dvic)n(e)22 b(to)h(implementors.)83
b Fu(Implemen)o(tation)23 b(of)f(these)h(t)o(w)o(o)e(constructs)h(are)g(easy)
g(in)h(a)189 1943 y(shared)14 b(memory)f(implemen)o(tation:)21
b(one)14 b(merely)h(needs)g(to)e(sync)h(up)q(dates)h(to)e(memory)h(b)q(efore)
189 1999 y(the)g(barrier)h(sync)o(hronization)g(is)g(executed.)20
b(\(In)15 b(practice,)g(no)f(extra)f(co)q(de)i(will)h(b)q(e)f(needed)h(on)189
2056 y(most)e(systems.\))189 2131 y(On)g(a)f(distributed)h(memory)l(,)f
(remote)g(agen)o(t)g(implemen)o(tation,)h(on)g(can)f(use)h(a)f(coun)o(ting)h
(mec)o(h-)189 2187 y(anism.)24 b(Eac)o(h)17 b(pro)q(cess)f(k)o(eeps)h(trac)o
(k,)f(for)g(eac)o(h)h(RMA)f(comm)o(unicator,)h(of)f(the)h(total)f(n)o(um)o(b)
q(er)189 2244 y(of)f(RMA)h(op)q(erations)g(issued)h(to)f(eac)o(h)g(target,)e
(and)i(the)g(total)g(n)o(um)o(b)q(er)g(of)f(requests)h(receiv)o(ed,)189
2300 y(from)e(eac)o(h)h(origin.)189 2375 y(A)j(barrier)g(is)h(implemen)o(ted)
g(using)g(a)f(reduce-scatter)g(comm)o(unication)h(where)g(eac)o(h)f(pro)q
(cess)189 2432 y(sends)d(the)f(n)o(um)o(b)q(er)h(of)f(RMA)g(op)q(erations)h
(it)g(issued)g(for)f(eac)o(h)g(destination,)i(and)e(eac)o(h)h(pro)q(cess)189
2488 y(receiv)o(es)i(the)h(total)e(n)o(um)o(b)q(er)h(of)g(of)f(RMA)i
(requests)f(issued)h(to)e(it.)26 b(Next,)17 b(eac)o(h)g(pro)q(cess)g(initi-)
189 2545 y(ates)d(a)h(barrier)h(sync)o(hronization)g(call)h(when)e(it)h(has)f
(completed)h(handling)h(all)g(p)q(ending)g(RMA)189 2601 y(requests.)i(This)14
b(will)h(double)f(the)g(time)f(needed)i(for)e(a)g(barrier)g(sync)o
(hronization)h(proto)q(col)g(that)189 2658 y(uses)h(p)q(oin)o(t)h(to)e(p)q
(oin)o(t)i(comm)o(unication.)1967 46 y Fj(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 19 21
bop 75 -100 a Fo(4.6.)34 b(WINDO)o(W)15 b(COUNTERS)1163 b Fu(19)189
49 y(If)16 b(a)f(fence)h(call)h(sp)q(eci\014es)h(a)d(sp)q(eci\014c)j(target,)
c(then)i(the)g(calling)h(pro)q(cess)f(sends)g(to)f(that)g(target)189
106 y(the)j(coun)o(t)h(of)f(RMA)g(op)q(erations)h(it)g(issued)h(to)e(that)f
(target.)29 b(This)19 b(message)f(is)h(receiv)o(ed)h(b)o(y)189
162 y(the)14 b(RMA)g(agen)o(t)f(at)g(the)h(target)e(who)i(issues)g(a)g(reply)
h(when)f(its)g(coun)o(t)f(of)h(serviced)h(op)q(erations)189
219 y(matc)o(hes)i(the)g(coun)o(t)g(of)h(issued)g(op)q(erations.)27
b(If)18 b(the)f(fence)h(do)q(es)g(not)f(sp)q(ecify)i(a)e(target,)g(then)189
275 y(a)k(scatter)f(needs)i(to)f(b)q(e)h(used,)h(rather)e(than)g(a)g(send,)i
(follo)o(w)o(ed)f(b)o(y)f(a)g(reduce)h(sum)g(on)f(the)189 332
y(ac)o(kno)o(wledgmen)o(ts.)189 405 y(This)g(sc)o(heme)f(in)o(tro)q(duces)h
(a)f(small)h(o)o(v)o(erhead)f(on)h(eac)o(h)f(RMA)h(op)q(erations)f(\(coun)o
(ter)g(incre-)189 462 y(men)o(t\).)189 536 y(Another)d(p)q(ossible)i
(implemen)o(tation,)g(is)e(for)g(eac)o(h)g(pro)q(cess)h(to)e(send)i(to)f(eac)
o(h)g(other)g(in)o(v)o(olv)o(ed)189 592 y(pro)q(cess)k(a)g(\\\015ush")g
(message.)37 b(Since)23 b(messages)e(are)f(ordered,)j(the)e(\015ush)h
(message)e(will)j(b)q(e)189 649 y(receiv)o(ed)12 b(after)f(all)h(previous)g
(RMA)f(requests)g(where)h(receiv)o(ed.)20 b(This)11 b(will)i(require)f
Fg(n)1667 632 y Fc(2)1697 649 y Fu(messages)189 705 y(for)17
b(a)h(barrier,)h Fg(n)f Fu(messages)g(for)f(a)h(fence)h(with)g(no)f(sp)q
(eci\014c)i(target,)d(and)i(one)f(message)g(for)f(a)189 761
y(fence)f(with)f(a)g(sp)q(eci\014c)i(target.)189 835 y(This)e(sc)o(heme)h(in)
o(tro)q(duces)g(no)f(o)o(v)o(erhead)g(on)g(RMA)g(comm)o(unication.)189
909 y(Rather)f(than)f(k)o(eeping)i(trac)o(k)e(of)g(all)i(RMA)f(op)q
(erations,)g(one)g(can)g(k)o(eep)g(trac)o(k)f(only)h(of)g(put)g(and)189
966 y(accum)o(ulate)h(op)q(erations,)g(since)i(get)d(and)i(RMW)f(implicitly)j
(return)d(an)g(ac)o(kno)o(wledgmen)o(t.)189 1039 y(Alternativ)o(ely)l(,)e
(one)f(could)h(implemen)o(t)f(put)g(and)g(accum)o(ulate)g(op)q(erations)g(so)
f(that)g(they)h(return)189 1096 y(an)j(ac)o(kno)o(wledgmen)o(t,)g(once)h
(completed.)22 b(In)16 b(suc)o(h)f(case,)h(the)f(barrier)h(or)f(fence)h(co)q
(de)g(requires)189 1152 y(no)f(additional)h(comm)o(unication.)21
b(\()p Fk(End)16 b(of)g(advic)n(e)g(to)h(implementors.)p Fu(\))75
1294 y Fr(4.6)59 b(Windo)n(w)20 b(Counters)75 1396 y Fu(This)d(section)h(sp)q
(eci\014es)g(the)f(seman)o(tics)g(of)f(the)h(windo)o(w)g(coun)o(ters)f(and)h
(the)g(op)q(erations)g(asso)q(ciated)75 1452 y(with)f(them.)166
1509 y(Eac)o(h)g(windo)o(w)h(is)h(asso)q(ciated)f(with)g(a)f(coun)o(ter)h
(and)g(a)f(threshold.)26 b(The)17 b(coun)o(ter)f(is)i(initialized)75
1565 y(to)d(zero)g(b)o(y)g(the)g(call)h(to)f Fn(MPI)p 592 1565
14 2 v 16 w(RMA)p 707 1565 V 17 w(INIT)p Fu(.)166 1622 y(Eac)o(h)c(put,)h
(get)e(and)i(accum)o(ulate)f(op)q(eration)h(incremen)o(ts)f(the)h(coun)o(ter)
f(asso)q(ciated)g(with)g(the)h(tar-)75 1678 y(get)i(windo)o(w)g(it)g
(accesses)h(b)o(y)f(the)g(v)m(alue)i(of)d(the)h Fn(increment)f
Fu(argumen)o(t.)19 b(The)14 b(coun)o(ter)g(is)h(incremen)o(ted)75
1735 y(after)j(the)i(access)f(to)f(the)h(target)f(windo)o(w)i(is)f(completed)
h(\(p)q(ossibly)l(,)i(b)q(efore)d(the)g(en)o(tire)h(op)q(eration)75
1791 y(completed\).)166 1848 y(The)14 b(target)e(pro)q(cess)i(can)g(c)o(hec)o
(k)g(the)g(v)m(alue)h(of)e(the)g(coun)o(ter)h(for)f(a)g(windo)o(w)h(in)h(its)
f(o)o(wn)f(memory)l(,)75 1904 y(b)o(y)f(calling)i Fn(MPI)p
361 1904 V 16 w(GET)p 465 1904 V 17 w(COUNTER)p Fu(.)f(In)f(addition,)i(a)e
(mec)o(hanism)g(is)h(pro)o(vided)g(for)f(the)g(target)f(windo)o(w)75
1960 y(to)17 b(w)o(ait)g(un)o(til)h(the)g(coun)o(ter)f(reac)o(hes)h(a)f
(threshold)h(v)m(alue.)28 b(The)18 b(threshold)g(is)g(set)f(or)g(reset)g(b)o
(y)h(calls)75 2017 y(to)d Fn(MPI)p 216 2017 V 16 w(SET)p 315
2017 V 17 w(THRESHOLD)p Fu(.)h(This)h(call)g(also)f(allo)q(cates)g(a)g
(request,)g(and)g(returns)g(a)f(handle)i(to)f(that)75 2073
y(request.)k(The)15 b(request)g(completes)h(when)f(the)g(coun)o(ter)g(reac)o
(hes)g(\(or)f(exceeds\))h(the)g(threshold)h(v)m(alue,)75 2130
y(at)d(whic)o(h)i(p)q(oin)o(t)g(the)f(coun)o(ter)f(is)i(decremen)o(ted)g(b)o
(y)f(the)g(threshold)g(v)m(alue.)41 b(The)14 b(threshold)h(can)f(then)-170
2116 y Fh(>)p Fu(\()p Fp(June)p Fu(\))-170 2105 y Fh(?)p Fu(\()p
Fp(June)p Fu(\))75 2164 y(b)q(e)j(reset)e(b)o(y)h(a)g(new)g(call)h(to)e
Fn(MPI)p 669 2164 V 16 w(SET)p 768 2164 V 17 w(THRESHOLD)p
Fu(.)h(The)g(threshold)h(should)g(not)e(b)q(e)i(reset)e(while)75
2221 y(the)g(asso)q(ciated)h(request)f(is)h(activ)o(e.)75 2372
y Fn(MPI)p 160 2372 V 16 w(GET)p 264 2372 V 17 w(COUNTER)g(\(comm)m(,)c
(count\))117 2449 y Fm(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)11
b(iden)o(tifying)i(RMA)g(windo)o(w)g(\(handle\))117 2523 y(OUT)108
b Fn(count)482 b Fm(curren)o(t)18 b(v)n(alue)d(of)h(asso)q(ciated)h(coun)o
(ter)g(\(non)f(negativ)o(e)g(in-)905 2579 y(teger\))75 2704
y Fq(int)23 b(MPI)p 245 2704 15 2 v 17 w(Get)p 334 2704 V 17
w(counter\(MPI)p 615 2704 V 16 w(Comm)g(comm,)g(int)h(count\))-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 20 22
bop 75 -100 a Fu(20)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fq(MPI)p 150 49 15 2 v 17 w(GET)p 239 49 V
17 w(COUNTER\(COMM,)22 b(COUNT,)h(IERROR\))170 106 y(INTEGER)g(COMM,)h
(COUNT,)f(IERROR)166 192 y Fu(Returns)14 b(the)g(curren)o(t)g(v)m(alue)h
(from)e(the)h(coun)o(ter)g(asso)q(ciated)g(with)h(the)f(lo)q(cal)h(windo)o(w)
f(of)f Fn(comm)m Fu(.)75 343 y Fn(MPI)p 160 343 14 2 v 16 w(SET)p
259 343 V 17 w(COUNTER)p 488 343 V 17 w(THRESHOLD)j(\(comm)m(,)c(count,)k
(request\))117 420 y Fm(IN)155 b Fn(comm)466 b Fm(comm)o(unicator)9
b(asso)q(ciated)k(with)e(RMA)h(windo)o(w)f(\(handle\))117 495
y(IN)155 b Fn(count)482 b Fm(n)o(um)o(b)q(er)13 b(of)h(units)g(\(non-negativ)
o(e)f(in)o(teger\))117 570 y(OUT)108 b Fn(request)452 b Fm(request)16
b(asso)q(ciated)e(with)g(coun)o(ter)h(\(handle\))75 695 y Fq(int)23
b(MPI)p 245 695 15 2 v 17 w(Set)p 334 695 V 17 w(counter)p
519 695 V 16 w(threshold\(MPI)p 847 695 V 16 w(Comm)g(comm,)g(int)h(count,)
393 751 y(MPI)p 468 751 V 17 w(Request)f(*request\))75 837
y(MPI)p 150 837 V 17 w(SET)p 239 837 V 17 w(COUNTER)p 424 837
V 16 w(THRESHOLD\(COMM,)f(COUNT,)h(REQUEST,)f(IERROR\))170
894 y(INTEGER)h(COMM,)h(COUNT,)f(REQUEST,)f(IERROR)166 980
y Fu(Set)17 b(the)g(threshold)h(for)f(the)g(coun)o(ter)g(of)f(the)i(lo)q(cal)
g(windo)o(w)f(asso)q(ciated)h(with)f Fn(comm)11 b Fu(to)17
b Fn(count)75 1037 y Fu(and)e(asso)q(ciates)g(a)g(request)h(with)f(the)g(ev)o
(en)o(t)g(\\coun)o(ter)g(reac)o(hes)g(threshold".)189 1141
y Fk(R)n(ationale.)46 b Fu(The)16 b(use)h(of)f(windo)o(w)g(coun)o(ters)g
(could)h(b)q(e)g(a)o(v)o(oided)f(and)h(replaced)g(b)o(y)f(messages)189
1198 y(sen)o(t)21 b(b)o(y)h(the)g(origin)h(pro)q(cess)f(to)f(the)i(target)d
(pro)q(cess.)41 b(But)22 b(this)g(will)i(require)f(additional)189
1254 y(comm)o(unication,)f(in)f(a)f(distributed)h(memory)f(system.)35
b(In)21 b(the)f(case)h(of)e(a)h(get,)h(one)g(w)o(ould)189 1311
y(need)c(an)f(additional)i(ac)o(kno)o(wledgmen)o(t)e(message;)g(in)i(a)e
(case)g(of)g(a)g(put,)h(one)f(w)o(ould)h(ha)o(v)o(e)f(the)189
1367 y(o)o(v)o(erhead)e(of)h(a)g(fence)h(and)f(an)g(additional)i(ac)o(kno)o
(wledgmen)o(t)e(message.)189 1442 y(Note)k(that)f(coun)o(ter)h(seman)o(tics)h
(are)f(carefully)h(designed)h(to)e(a)o(v)o(oid)g(race)g(conditions.)33
b(Since)189 1499 y(coun)o(ters)18 b(are)g(only)h(incremen)o(ted)h(and)f
(decremen)o(ted)g(and)g(nev)o(er)g(set,)g(incremen)o(ts)g(from)f(in-)189
1555 y(coming)13 b(puts)g(and)h(gets)e(are)h(nev)o(er)g(lost.)20
b(The)13 b(only)h(thing)f(to)g(w)o(atc)o(h)f(out)h(for)f(is)i(that)e(a)h
(coun)o(ter)189 1612 y(ma)o(y)h(exceed)i(its)g(threshold)g(v)m(alue.)21
b(\()p Fk(End)15 b(of)i(r)n(ationale.)p Fu(\))189 1716 y Fk(A)n(dvic)n(e)d
(to)h(implementors.)39 b Fu(The)14 b(coun)o(ter)f(need)i(not)e(b)q(e)i(up)q
(dated)f(\(th)o(us)f(reducing)i(comm)o(uni-)189 1773 y(cation)f(o)o(v)o
(erhead\))f(when)h(the)g(incremen)o(t)h(is)f(zero.)20 b(On)14
b(a)g(distributed)h(memory)e(system)h(where)189 1829 y(RMA)19
b(op)q(erations)g(are)g(implemen)o(ted)i(b)o(y)e(a)f(remote)h(agen)o(t,)g
(the)g(agen)o(t)f(will)j(incremen)o(t)f(the)189 1886 y(target)14
b(coun)o(ter.)21 b(In)16 b(a)f(shared)h(memory)f(implemen)o(tation,)h(the)g
(coun)o(ter)f(will)i(b)q(e)g(up)q(dated)f(b)o(y)189 1942 y(the)f(origin)h
(pro)q(cess;)f(lo)q(c)o(king)h(is)g(required.)21 b(\()p Fk(End)15
b(of)i(advic)n(e)f(to)h(implementors.)p Fu(\))75 2085 y Fr(4.7)59
b(Semantics)18 b(of)i(RMA)e(Op)r(erations)75 2188 y Ff(4.7.1)49
b(Order)75 2274 y Fu(When)13 b(a)g(put)g(or)g(get)f(completes)i(then)f(the)h
(op)q(eration)f(has)g(completed)h(at)e(the)h(origin,)h(and)f(the)g(origin)75
2330 y(bu\013er)i(can)h(b)q(e)g(accessed.)21 b(In)16 b(the)f(case)g(of)g(a)g
(put,)g(the)h(data)f(has)g(b)q(een)h(copied)h(out;)d(in)j(the)e(case)g(of)g
(a)75 2387 y(get,)f(the)i(data)e(has)h(b)q(een)i(copied)f(in.)166
2443 y(When)j(a)f(put)g(or)g(get)g(up)q(dates)h(the)g(target)e(request,)i
(then)g(the)f(op)q(eration)h(has)f(completed)i(at)-170 2430
y Fh(>)p Fu(\()p Fp(June)p Fu(\))75 2489 y(the)d(target,)f(and)h(the)g
(target)f(bu\013er)h(can)g(b)q(e)h(accessed.)26 b(In)17 b(the)h(case)f(of)f
(a)h(get,)f(the)i(data)e(has)h(b)q(een)75 2545 y(copied)f(out;)f(in)h(the)f
(case)g(of)g(a)g(put,)g(the)g(data)g(has)g(b)q(een)h(copied)h(in.)-170
2532 y Fh(?)p Fu(\()p Fp(June)p Fu(\))166 2591 y(Calling)k
Fn(MPI)p 412 2591 14 2 v 16 w(BARRIER)h Fu(with)e(a)g(comm)o(unicator)g
(returned)h(b)o(y)f(one)g(of)g(the)g Fn(MPI)p 1658 2591 V 16
w(RMA)p 1773 2591 V 17 w(INIT)75 2647 y Fu(calls)j(has)f(the)g(additional)h
(seman)o(tics)f(of)g(a)f(fence)i(with)f(resp)q(ect)h(to)e(puts)h(and)g(gets.)
40 b(All)23 b(RMA)75 2704 y(comm)o(unications)12 b(started)e(on)h(the)g(comm)
o(unicator)g(b)q(efore)g(the)h(call)g(to)e Fn(MPI)p 1404 2704
V 16 w(BARRIER)i Fu(will)h(complete)1967 46 y Fj(1)1967 103
y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967
385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959
667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893
y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119
y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345
y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570
y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796
y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022
y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248
y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474
y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699
y(48)p eop
%%Page: 21 23
bop 75 -100 a Fo(4.7.)34 b(SEMANTICS)16 b(OF)f(RMA)g(OPERA)l(TIONS)853
b Fu(21)75 49 y(at)12 b(the)h(target)e(b)q(efore)i Fn(MPI)p
554 49 14 2 v 16 w(BARRIER)h Fu(completes.)20 b(Similarly)l(,)15
b(a)d(call)i(to)e Fn(MPI)p 1449 49 V 16 w(FENCE)h Fu(has)f(the)h(e\013ect)75
106 y(that)i(all)h(RMA)g(comm)o(unications)g(started)f(b)o(y)g(the)h(caller)g
(on)g(the)g(comm)o(unicator)f(\(to)f(the)i(sp)q(eci\014ed)75
162 y(target\))e(b)q(efore)h(the)g(call)i(to)d(fence)i(will)h(complete)f(at)f
(the)g(target\(s\))e(b)q(efore)j(the)f(fence)h(call)g(returns.)166
219 y(No)g(other)g(order)g(requiremen)o(ts)h(are)f(imp)q(osed)h(on)g(RMA)f
(op)q(erations.)24 b(Tw)o(o)15 b(RMA)i(op)q(erations)75 275
y(issued)d(consecutiv)o(ely)h(b)o(y)e(the)g(same)g(pro)q(cess)g(for)g(the)g
(same)f(target)g(can)i(execute)f(in)h(arbitrary)f(order,)75
332 y(unless)19 b(the)f(user)h(ensures)f(that)g(the)g(\014rst)f(op)q(eration)
i(has)f(completed)h(b)q(efore)f(it)g(issues)h(the)f(second)75
388 y(one.)h(Similarly)l(,)d(there)e(is)g(no)f(guaran)o(teed)g(order)h(of)f
(execution)h(b)q(et)o(w)o(een)g(message)f(passing)h(calls)h(and)75
444 y(RMA)k(calls,)h(ev)o(en)f(when)g(issued)g(b)o(y)g(the)f(same)h(pro)q
(cess)f(with)h(the)g(same)f(comm)o(unicator)g(and)h(the)75
501 y(same)c(destination.)166 557 y(Note)e(that)f(\\b)q(efore")h(or)g
(\\after",)f(as)g(used)i(in)g(the)f(previous)h(paragraphs,)e(mean)i(b)q
(efore)f(or)g(after)75 614 y(in)e(program)e(order)h(at)f(eac)o(h)i(pro)q
(cess.)18 b(If)11 b(a)e(pro)q(cess)i(is)f(m)o(ultithreaded,)i(then)f(t)o(w)o
(o)e(op)q(erations)h(executed)75 670 y(b)o(y)g(the)h(same)f(pro)q(cess)h(ma)o
(y)e(not)h(b)q(e)h(ordered,)h(in)f(whic)o(h)g(case)g(there)f(is)h(no)f(order)
g(requiremen)o(t.)19 b(Th)o(us,)11 b(a)75 727 y(barrier)h(executed)i(b)o(y)e
(a)g(thread)g(only)h(enforces)f(completion)i(of)e(RMA)g(calls)h(that)f
(preceed)h(the)g(barrier)75 783 y(call)19 b(in)g(program)e(order;)j(an)e(RMA)
g(call)h(executed)h(b)o(y)e(another)g(concurren)o(t)g(thread)g(ma)o(y)l(,)g
(or)g(ma)o(y)75 840 y(not)d(not)g(b)q(e)g(a\013ected.)189 933
y Fk(R)n(ationale.)39 b Fu(The)14 b(order)f(b)q(eha)o(vior)i(describ)q(ed)g
(corresp)q(onds)g(to)e(an)h(op)q(erational)g(mo)q(del)h(where)189
989 y(RMA)k(requests)g(are)f(handled)j(b)o(y)e(an)g(RMA)g(agen)o(t,)g(suc)o
(h)g(as)g(a)g(p)q(ersisten)o(t)g(comm)o(unication)189 1046
y(handler;)h(the)e(agen)o(t)f(p)q(osts)h(a)g(non)o(blo)q(c)o(king)h(receiv)o
(e)g(to)e(handle)i(a)f(put)g(request)g(and)h(p)q(osts)e(a)189
1102 y(non)o(blo)q(c)o(king)d(send)f(to)f(handle)i(a)f(get)f(request.)19
b(Th)o(us,)13 b(ev)o(en)g(though)g(incoming)h(RMA)f(requests)189
1159 y(are)21 b(handled)i(in)f(order,)h(the)e(accesses)h(to)f(the)g(target)g
(memory)g(ma)o(y)f(o)q(ccur)i(out)g(of)f(order.)189 1215 y(Consider)14
b(the)h(case)f(of)g(a)g(blo)q(c)o(king)h(put,)f(immediately)i(follo)o(w)o(ed)
f(b)o(y)f(another)g(blo)q(c)o(king)h(put)g(to)189 1272 y(the)j(same)h
(destination.)31 b(Eac)o(h)18 b(\\send")h(matc)o(hes)f(its)h(o)o(wn)f
(\\receiv)o(e",)h(but)g(the)g(\014rst)f(\\send")189 1328 y(\(put\))10
b(ma)o(y)g(return)h(b)q(efore)g(data)g(has)f(b)q(een)j(copied)f(in)f(the)g
(remote)g(memory)l(.)18 b(Th)o(us,)11 b(the)g(up)q(date)189
1385 y(of)k(the)h(target)f(memory)h(b)q(e)g(the)h(t)o(w)o(o)d(put)j(calls)g
(ma)o(y)e(o)o(v)o(erlap.)22 b(Consider)17 b(a)f(put,)g(follo)o(w)o(ed)g(b)o
(y)189 1441 y(a)h(get,)g(with)g(the)g(same)g(comm)o(unicator)g(and)h(same)f
(target)f(bu\013er.)26 b(The)17 b(put)h(ma)o(y)e(complete)189
1497 y(and)f(the)f(get)h(ma)o(y)f(start)f(b)q(efore)i(data)f(w)o(as)g
(written)h(in)h(the)f(target)e(bu\013er;)i(the)f(t)o(w)o(o)g(calls)i(ma)o(y)
189 1554 y(o)o(v)o(erlap,)e(and)i(the)f(get)g(need)h(not)f(return)g(the)g
(data)f(written)i(b)o(y)f(the)g(put.)189 1627 y(An)k(alternativ)o(e)h(b)q
(eha)o(vior)g(is)g(that)f(ac)o(hiev)o(ed)i(b)o(y)e(an)h(op)q(erational)g(mo)q
(del)g(where)g(the)g(RMA)189 1684 y(agen)o(t)13 b(p)q(osts)g(blo)q(c)o(king)i
(sends)f(to)f(satisfy)g(get)h(requests,)f(and)h(blo)q(c)o(king)h(receiv)o(es)
f(to)f(satisfy)h(put)189 1740 y(requests.)19 b(Th)o(us,)c(the)h(handling)g
(of)f(RMA)h(requests)f(is)g(serialized.)22 b(In)16 b(this)g(mo)q(del)231
1830 y(1.)22 b(Eac)o(h)15 b(RMA)g(op)q(eration)g(is)h(executed)g(atomically)
231 1900 y(2.)22 b(RMA)13 b(op)q(erations)h(issued)g(successiv)o(ely)h(b)q(e)
f(the)g(same)f(origin)h(to)f(the)g(same)g(target,)f(using)289
1956 y(the)j(same)g(comm)o(unicator,)f(are)h(executed)h(in)g(the)g(order)f
(they)g(w)o(ere)g(issued.)189 2046 y(This)e(alternativ)o(e)f(mo)q(del)h(pro)o
(vides)g(stronger)f(order)g(and)g(atomicit)o(y)g(guaran)o(tees,)g(and)h(is)g
(easier)189 2103 y(to)h(program)g(with;)h(ho)o(w)o(ev)o(er,)f(it)i(will)g
(lead)g(to)f(lo)o(w)o(er)g(p)q(erformance,)g(as)f(it)i(reduces)g(the)f(opp)q
(or-)189 2159 y(tunities)h(for)e(concurrency)l(.)21 b(In)15
b(a)g(shared)g(memory)g(implemen)o(tation,)g(one)h(will)g(lik)o(ely)h(need)f
(to)189 2216 y(lo)q(c)o(k)i(the)f(windo)o(w)h(accessed)g(and)g(serialize)h
(all)g(accesses.)27 b(In)18 b(a)f(distributed)i(memory)l(,)f(RMA)189
2272 y(agen)o(t)g(implemen)o(tation,)j(the)d(agen)o(t)h(will)h(ha)o(v)o(e)f
(to)f(use)h(blo)q(c)o(king)h(sends)g(and)f(receiv)o(es,)h(th)o(us)189
2328 y(reducing)c(concurrency)l(.)21 b(\()p Fk(End)16 b(of)g(r)n(ationale.)p
Fu(\))75 2448 y Ff(4.7.2)49 b(Progress)75 2534 y Fu(Once)18
b(a)f(RMA)g(op)q(eration)h(is)g(p)q(osted,)f(it)h(should)g(complete,)g
(irresp)q(ectiv)o(e)h(of)d(other)h(op)q(erations)h(o)q(c-)75
2591 y(curring)f(in)h(the)f(system.)24 b(The)17 b(completion)h(of)e(a)h(put,)
g(accum)o(ulate)g(or)f(RMW)h(op)q(eration)g(in)g(target)75
2647 y(memory)e(means)h(only)h(that)e(data)g(is)i(stored)e(logically)j(in)f
(the)f(target)e(windo)o(w)i(at)g(is)g(visible)i(to)d(sub-)75
2704 y(sequen)o(t)j(RMA)g(op)q(erations)f(originated)h(at)f(an)o(y)h(pro)q
(cess.)27 b(Before)18 b(using)g(loads)g(and)g(stores)e(on)i(the)-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 22 24
bop 75 -100 a Fu(22)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fu(target)g(pro)q(cess)i(to)e(access)i(data)e(up)q(dated)i(b)
o(y)f(a)g(prior)h(RMA)f(op)q(eration,)h(the)f(target)f(pro)q(cess)i(m)o(ust)
75 106 y(\014rst)h(issue)h(a)e(call)j(to)d Fn(MPI)p 554 106
14 2 v 16 w(WINDO)o(W)p 766 106 V 16 w(IN)p Fu(;)g(similarly)j(a)e(call)h(to)
f Fn(MPI)p 1314 106 V 15 w(WINDO)o(W)p 1525 106 V 16 w(OUT)h
Fu(m)o(ust)f(o)q(ccur)75 162 y(after)c(data)f(w)o(as)h(stored)g(b)o(y)g(a)g
(pro)q(cess)h(in)g(its)g(memory)f(and)g(b)q(efore)h(this)g(data)e(can)i(b)q
(e)g(accessed)g(b)o(y)f(a)75 219 y(get)h(\(see)g(b)q(elo)o(w\).)75
362 y Fr(4.8)59 b(A)n(tomicit)n(y)18 b(and)i(Memo)n(ry)e(Coherence)i(Op)r
(erations)75 463 y Fu(First,)14 b(some)h(terminology)l(.)-170
450 y Fh(>)p Fu(\()p Fp(June)p Fu(\))166 509 y(The)i(execution)i(of)d(a)h
(message-passing)g(program)f(imp)q(oses)i(a)f(partial)h(order)f(on)g(the)g
(program)75 565 y(op)q(erations:)i(this)c(is)f(the)g(transitiv)o(e)g(closure)
h(of)e(the)h(order)g(relation)g(at)g(eac)o(h)g(pro)q(cess)g(\(if)g(the)g(pro)
q(cess)75 622 y(is)h(single)g(threaded,)f(then)h(all)g(the)f(op)q(erations)g
(executed)h(b)o(y)f(this)h(pro)q(cess)f(are)g(totally)g(ordered\))g(and)75
678 y(the)g(order)g(implied)j(b)o(y)d(comm)o(unications)h(\(a)f(receiv)o(e)h
(completes)g(after)f(the)g(matc)o(hing)g(send)h(started,)75
735 y(a)g(windo)o(w)g(coun)o(ter)f(is)i(up)q(dated)f(after)g(the)g(corresp)q
(onding)g(RMA)h(call)g(started,)d(etc.\).)19 b(W)l(e)c(sa)o(y)g(that)75
791 y(Op)q(eration)h(B)f Fk(fol)r(lows)g Fu(op)q(eration)g(A)g(\(and)g(A)g
Fk(pr)n(e)n(c)n(e)n(des)f Fu(B\))g(if)i(AB)f(b)q(elongs)h(to)f(this)g
(partial)g(order;)g(A)75 847 y(and)h(B)g(are)f Fk(c)n(oncurr)n(ent)g
Fu(if)h(neither)h(A)f(precedes)g(B)g(nor)g(B)g(precedes)g(A.)g(Note)f(that)g
(this)h(is)g(a)g(logical)75 904 y(order)f(of)g(execution,)h(not)e(a)h(ph)o
(ysical,)h(w)o(all-clo)q(c)o(k)h(ordering.)166 960 y(Tw)o(o)9
b(accesses)i(to)f(the)g(same)g(memory)g(lo)q(cation)h(are)f
Fp(con\015icting)j Fu(if)e(there)f(is)h(a)f(lo)q(cation)h(accessed)75
1017 y(b)o(y)k(b)q(oth)g(and)h(up)q(dated)g(b)o(y)f(at)g(least)g(one)g(of)g
(the)g(t)o(w)o(o)f(op)q(erations.)166 1073 y(W)l(e)g(distinguish)j(b)q(et)o
(w)o(een)d Fk(lo)n(c)n(al)f Fu(memory)h(accesses,)g(and)h Fk(r)n(emote)f
Fu(memory)f(accesses.)20 b(Accesses)75 1130 y(executed)h(b)o(y)g(a)f(pro)q
(cess)h(to)f(its)g(o)o(wn)g(memory)g(using)i(loads)e(or)g(stores)g(are)g(lo)q
(cal;)k(also)d(are)f(lo)q(cal)75 1186 y(accesses)d(to)g(a)g(comm)o(unication)
g(bu\013er)h(p)q(erformed)f(b)o(y)g(a)g(send)g(or)g(receiv)o(e)h(MPI)f(call)h
(and)g(accesses)75 1243 y(to)13 b(the)i(origin)f(bu\013er)g(of)g(an)g(RMA)g
(call.)21 b(Accesses)15 b(to)e(the)h(target)f(bu\013er)h(of)g(an)g(RMA)g(op)q
(eration)g(are)75 1299 y(remote)e(\(including)k(the)d(case)f(of)h(an)g(RMA)g
(op)q(eration)g(where)g(a)g(pro)q(cess)g(accesses)g(its)g(o)o(wn)f(memory\).)
166 1356 y(MPI)k(has)g(to)f(sp)q(ecify)i(the)f(outcome)g(of)f(concurren)o(t,)
h(con\015icting)h(accesses)g(to)e(a)h(windo)o(w.)22 b(Tw)o(o)75
1412 y(issues)16 b(arise:)131 1506 y(1.)22 b(Con\015icting)16
b(remote)f(RMA)g(accesses)g(to)g(the)g(same)g(windo)o(w.)131
1600 y(2.)22 b(Con\015icts)15 b(b)q(et)o(w)o(een)h(lo)q(cal)g(and)f(remote)g
(accesses)h(to)e(a)h(windo)o(w.)75 1720 y Fn(Con\015icting)h(Remote)d
(Accesses)75 1806 y Fu(The)i(execution)h(of)f(a)g(Read)p 565
1806 V 17 w(Mo)q(dify)p 723 1806 V 17 w(W)l(rite)g(or)g(accum)o(ulate)g(up)q
(date)h(of)e(an)h(elemen)o(t)h(is)g(atomic)f(with)75 1862 y(resp)q(ect)e(to)e
(other)h(Read)p 500 1862 V 17 w(Mo)q(dify)p 658 1862 V 17 w(W)l(rite)h(and)f
(accum)o(ulate)h(up)q(dates)g(of)e(that)h(same)g(elemen)o(t.)20
b(I.e.,)12 b(the)75 1918 y(outcome)f(is)g(as)g(if)g(the)g(RMW)g(and)g(accum)o
(ulate)g(up)q(dates)h(of)e(that)g(elemen)o(t)i(w)o(ere)f(serialized.)20
b(Ho)o(w)o(ev)o(er,)75 1975 y(if)c(t)o(w)o(o)f(accum)o(ulate)h(op)q(erations)
g(up)q(date)g(the)g(same)g(t)o(w)o(o)e(elemen)o(ts,)j(then)f(these)g(t)o(w)o
(o)e(elemen)o(ts)j(ma)o(y)75 2031 y(b)q(e)22 b(up)q(dated)f(in)h(a)f
(di\013eren)o(t)g(order)g(b)o(y)g(the)g(t)o(w)o(o)f(op)q(erations:)31
b(while)23 b(eac)o(h)e(individual)j(up)q(date)d(is)75 2088
y(atomic,)15 b(the)g(en)o(tire)h(accum)o(ulate)f(op)q(eration)h(is)f(not)g
(atomic.)166 2144 y(It)g(is)h(erroneous)f(to)g(ha)o(v)o(e)g(concurren)o(t)g
(accum)o(ulate)h(or)e(RMW)h(up)q(dates)h(of)f(v)m(ariables)i(that)d(par-)75
2201 y(tially)19 b(o)o(v)o(erlap;)g(e.g.,)e(accum)o(ulating)i(in)o(to)f
(complex)h(n)o(um)o(b)q(ers,)f(of)g(t)o(yp)q(e)g Fl(MPI)p 1463
2201 13 2 v 14 w(COMPLEX)p Fu(,)f(and)h(con-)75 2257 y(curren)o(tly)c(accum)o
(ulating)g(in)o(to)g(the)f(real)h(or)f(imaginary)h(part)e(of)h(these)h(n)o
(um)o(b)q(ers,)g(of)f(t)o(yp)q(e)g Fl(MPI)p 1748 2257 V 15
w(REAL)p Fu(.)166 2314 y(The)k(execution)g(of)g(put)f(or)g(get)h(op)q
(erations)f(is)h(atomic)g(for)f(eac)o(h)g(accessed)i(b)o(yte;)e(i.e.,)h(for)f
(eac)o(h)75 2370 y(b)o(yte,)f(the)h(outcome)f(is)h(as)f(if)h(puts)g(and)g
(gets)f(w)o(ere)g(serialized)j(in)e(some)g(order.)k(Ho)o(w)o(ev)o(er,)15
b(the)g(en)o(tire)75 2427 y(put)i(or)g(get)g(op)q(eration)g(is)h(not)f
(atomic,)g(and)g(di\013eren)o(t)h(b)o(ytes)f(ma)o(y)f(b)q(e)i(accessed)g(in)g
(di\013eren)o(t)f(order)75 2483 y(b)o(y)e(the)g(same)g(t)o(w)o(o)f(op)q
(erations.)166 2539 y(The)j(outcome)g(is)h(unde\014ned)g(if)g(put)f(or)g(get)
f(op)q(erations)i(con\015ict)g(with)f(RMW)g(or)g(accum)o(ulate)75
2596 y(op)q(erations.)k(Also,)15 b(the)h(outcome)f(is)h(unde\014ned)h(if)f(t)
o(w)o(o)e(RMA)i(op)q(erations)f(using)i(di\013eren)o(t)e(comm)o(u-)75
2652 y(nicators)g(p)q(erform)g(con\015icting)i(accesses)e(to)g(the)g(same)g
(lo)q(cation.)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967
215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967
498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724
y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959
1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959
1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959
1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959
1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959
1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959
2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959
2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959
2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 23 25
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(23)189 49 y Fk(R)n(ationale.)61 b Fu(The)19
b(atomicit)o(y)g(requiremen)o(t)g(for)g(put/get)f(corresp)q(onds)h(to)f(the)h
(b)q(eha)o(vior)g(of)189 106 y(shared)c(memory)g(m)o(ultipro)q(cessors.)20
b(No)15 b(sp)q(ecial)i(e\013ort)d(is)i(required)g(to)f(satisfy)f(it.)189
180 y(Lo)q(c)o(ks)22 b(are)g(needed)i(to)d(guaran)o(tee)h(atomicit)o(y)g(of)f
(accum)o(ulate)i(or)f(RMW)g(op)q(erations)g(in)h(a)189 236
y(shared)17 b(memory)g(implemen)o(tation,)h(or)f(an)o(y)g(implemen)o(tation)i
(where)e(more)g(than)g(one)h(agen)o(t)189 293 y(ma)o(y)e(access)i(the)g
(windo)o(w.)26 b(The)18 b(curren)o(t)g(design)g(allo)o(ws)g(one)f(to)g(asso)q
(ciate)g(lo)q(c)o(ks)h(with)g(win-)189 349 y(do)o(ws)g(\(or)h(parts)g(of)g(a)
g(windo)o(w\),)h(not)f(with)g(memory)g(lo)q(cations,)i(and)e(not)g(to)g(use)h
(lo)q(c)o(ks)g(for)189 406 y(put/get)13 b(op)q(erations.)20
b(In)15 b(order)f(to)g(handle)i(put/get)e(accesses)g(that)g(con\015ict)h
(with)g(concurren)o(t)189 462 y(accum)o(ulate)j(or)f(RMW)g(accesses,)h(one)g
(w)o(ould)g(need)h(to)e(protect)g(the)h(put/get)f(accesses)h(with)189
519 y(lo)q(c)o(ks;)h(in)f(order)f(to)g(handle)i(con\015icting)g(accesses)f
(to)f(the)h(same)f(lo)q(cation)i(that)e(use)h(di\013eren)o(t)189
575 y(windo)o(ws,)d(one)h(w)o(ould)g(need)h(to)e(k)o(eep)h(trac)o(k)f(of)g
(windo)o(w)h(o)o(v)o(erlaps,)f(and)h(asso)q(ciate)f(lo)q(c)o(ks)i(with)189
631 y(memory)d(areas,)g(not)h(windo)o(ws.)20 b(\()p Fk(End)c(of)g(r)n
(ationale.)p Fu(\))189 731 y Fk(A)n(dvic)n(e)22 b(to)h(implementors.)82
b Fu(The)23 b(atomicit)o(y)f(requiremen)o(t)h(for)f(accum)o(ulate)h(and)f
(RMW)189 787 y(op)q(erations)c(can)h(b)q(e)g(implemen)o(ted)h(with)f(no)f(lo)
q(c)o(king)i(when)e(the)h(up)q(dates)g(to)f(a)g(windo)o(w)g(are)189
844 y(done)h(b)o(y)f(a)h(unique)h(RMA)f(agen)o(t:)26 b(the)19
b(up)q(dates)g(are)f(serialized)j(b)o(y)e(the)f(agen)o(t.)30
b(Lo)q(c)o(king)19 b(is)189 900 y(required)e(in)h(an)f(implemen)o(tation)g
(where)g(RMA)g(accesses)g(to)f(a)h(windo)o(w)f(ma)o(y)g(b)q(e)i(p)q(erformed)
189 957 y(b)o(y)g(m)o(ultiple)i(threads.)29 b(The)19 b(lo)q(c)o(k)g(gran)o
(ularit)o(y)f(can)g(b)q(e)h(as)f(large)g(as)g(the)h(en)o(tire)g(windo)o(w,)f
(or)189 1013 y(as)d(small)h(as)g(the)f(largest)h(en)o(tit)o(y)f(that)g(can)h
(b)q(e)g(up)q(dated)h(atomically)f({)g(one)f(can)h(trade-o\013)f(the)189
1070 y(added)g(o)o(v)o(erhead)g(of)g(\014ner)h(lo)q(c)o(ks)f(against)g(the)g
(additional)i(concurrency)f(they)f(pro)o(vide.)189 1144 y(In)j(systems)f
(where)i(w)o(ord-sized)f(v)m(alues)h(are)f(not)f(necessarily)i(w)o(ord)e
(aligned,)j(one)e(ma)o(y)f(need)189 1200 y(t)o(w)o(o)c(lo)q(c)o(ks)j(to)e
(protect)g(an)h(up)q(date,)h(unless)g(the)f(en)o(tire)g(windo)o(w)g(is)h(lo)q
(c)o(k)o(ed.)k(\()p Fk(End)c(of)g(advic)n(e)g(to)189 1257 y(implementors.)p
Fu(\))75 1376 y Fn(Con\015icting)g(Lo)q(cal)f(and)h(Remote)d(Accesses)75
1462 y Fu(W)l(e)19 b(prop)q(ose)f(to)g(use)h(a)f(dela)o(y)o(ed)h(consistency)
g(mo)q(del)g(to)f(handle)i(suc)o(h)f(con\015icts.)30 b(This)19
b(allo)o(ws)g(for)75 1518 y(more)c(e\016cien)o(t)h(implemen)o(tations,)g(and)
f(will)i(not)e(burden)h(unduly)h(users)e(\(see)g(rationale)h(b)q(elo)o(w\).)
166 1574 y(T)l(o)g(understand)h(the)g(coherence)h(mo)q(del,)f(one)g(ma)o(y)f
(think)h(that)f(there)h(are)f(t)o(w)o(o)f(copies)j(of)e(eac)o(h)75
1631 y(RMA)h(windo)o(w:)23 b(a)16 b Fk(private)h Fu(cop)o(y)g(that)f(is)h
(accessed)h(b)o(y)e(lo)q(cal)i(accesses,)f(and)g(a)f Fk(public)h
Fu(cop)o(y)f(that)g(is)75 1687 y(accessed)j(b)o(y)g(remote)f(accesses.)30
b(When)20 b(a)e(lo)q(cal)i(store)e(op)q(eration)g(is)i(complete,)g(then)f
(the)f(priv)m(ate)75 1744 y(windo)o(w)e(cop)o(y)f(has)h(b)q(een)g(up)q
(dated,)h(but)e(the)h(up)q(date)g(ma)o(y)f(not)g(b)q(e)i(y)o(et)e(visible)j
(in)e(the)g(public)h(cop)o(y;)75 1800 y(the)h(new)g(v)m(alue)h(will)h(b)q(e)f
(copied)g(\(b)o(yte)e(b)o(y)h(b)o(yte\))g(in)o(to)g(the)g(public)i(cop)o(y)e
(some)f(time)h(later.)29 b(When)75 1857 y(an)15 b(RMA)g(put)h(or)e(accum)o
(ulate)i(or)f(RMW)g(op)q(eration)g(has)g(completed)h(at)f(the)g(target)f
(memory)l(,)h(then)75 1913 y(the)22 b(public)i(windo)o(w)e(cop)o(y)g(at)f
(the)h(target)f(has)h(b)q(een)h(up)q(dated;)i(the)d(new)g(v)m(alue)i(will)f
(b)q(e)g(copied)75 1970 y(\(b)o(yte)d(b)o(y)h(b)o(yte\))f(to)g(the)h(priv)m
(ate)g(cop)o(y)f(at)g(that)g(pro)q(cess)h(some)g(time)g(later.)36
b(A)20 b(strict)h(coherence)75 2026 y(mo)q(del)15 b(w)o(ould)g(require)f
(that)g(an)g(up)q(date)h(is)f(propagated)g(from)f(one)h(cop)o(y)g(to)g
(another)f(as)h(so)q(on)g(as)g(the)75 2083 y(up)q(date)h(to)q(ok)f(place.)21
b(A)14 b(dela)o(y)o(ed)i(coherence)f(proto)q(col)g(allo)o(ws)f(one)h(to)f
(dela)o(y)h(the)g(cop)o(ying.)20 b(Windo)o(w)75 2139 y(sync)o(hronization)c
(p)q(oin)o(ts)f(are)g(de\014ned)h(where)f(the)g(coherence)h(b)q(et)o(w)o(een)
f(the)g(t)o(w)o(o)f(copies)h(is)h(restored.)75 2195 y(The)e(cop)o(ying)f
(from)g(one)g(windo)o(w)h(to)f(another)g(o)q(ccurs)g(some)g(time)h(after)e
(the)i(up)q(date)g(to)q(ok)e(place)j(and)75 2252 y(b)q(efore)g(the)h(cop)o(y)
f(sync)o(hronization)h(completes.)166 2308 y(Accesses)11 b(to)e(a)h(windo)o
(w)h(are)f(divided)i(in)f(successiv)o(e)h(ep)q(o)q(c)o(hs.)19
b(Ep)q(o)q(c)o(hs)10 b(are)g(either)h Fk(lo)n(c)n(al)e Fu(or)h
Fk(r)n(emote)p Fu(.)75 2365 y(During)i(a)f(lo)q(cal)i(ep)q(o)q(c)o(h)f(the)g
(\(priv)m(ate)g(cop)o(y)f(of)g(the\))g(windo)o(w)h(ma)o(y)f(b)q(e)h(up)q
(dated)g(lo)q(cally)l(,)i(but)e(the)f(\(pub-)75 2421 y(lic)j(cop)o(y)d(of)h
(the\))g(windo)o(w)g(should)h(not)f(b)q(e)h(up)q(dated)g(remotely)l(.)19
b(Lo)q(cal)13 b(and)f(remote)g(read)g(accesses)g(are)75 2478
y(allo)o(w)o(ed.)19 b(During)13 b(a)f(remote)g(ep)q(o)q(c)o(h,)h(the)g
(\(public)h(cop)o(y)e(of)g(the\))g(windo)o(w)h(ma)o(y)f(b)q(e)h(up)q(dated)g
(remotely)l(,)75 2534 y(but)k(the)f(\(lo)q(cal)i(cop)o(y)e(of)g(the\))g
(windo)o(w)h(should)h(not)e(b)q(e)h(up)q(dated)g(lo)q(cally)l(.)26
b(Lo)q(cal)17 b(and)g(remote)f(read)75 2591 y(accesses)g(are)g(allo)o(w)o
(ed.)24 b(A)16 b(lo)q(cal)h(ep)q(o)q(c)o(h)g(is)g(completed)g(b)o(y)g(a)e
(call)j(to)d Fn(MPI)p 1391 2591 14 2 v 16 w(WINDO)o(W)p 1603
2591 V 16 w(OUT\(comm)n(\))o Fu(,)75 2647 y(whic)o(h)23 b(completes)f(the)g
(transfer)f(of)g(all)i(lo)q(cal)g(up)q(dates)f(from)f(the)h(priv)m(ate)g
(windo)o(w)g(cop)o(y)g(to)f(the)75 2704 y(public)e(windo)o(w)f(cop)o(y)l(.)25
b(A)17 b(remote)g(ep)q(o)q(c)o(h)h(is)g(completed)g(b)o(y)f(a)f(call)j(to)d
Fn(MPI)p 1442 2704 V 16 w(WINDO)o(W)p 1654 2704 V 16 w(IN\(comm)m(\))o
Fu(,)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32
272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40
554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40
836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40
1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345
y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40
1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853
y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40
2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361
y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40
2643 y(47)-40 2699 y(48)p eop
%%Page: 24 26
bop 75 -100 a Fu(24)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fu(whic)o(h)e(completes)h(the)e(transfer)g(of)h(all)g(remote)
f(up)q(dates)h(from)f(the)h(public)i(cop)o(y)d(to)g(the)h(priv)m(ate)g(cop)o
(y)l(.)75 106 y(Th)o(us,)k(the)g(coherence)h(b)q(et)o(w)o(een)g(the)f(t)o(w)o
(o)f(windo)o(w)h(copies)h(is)g(restored)e(when)i(these)f(t)o(w)o(o)f
(functions)75 162 y(are)f(called.)75 313 y Fn(MPI)p 160 313
14 2 v 16 w(WINDO)o(W)p 372 313 V 16 w(IN\(com)o(m)n(\))117
390 y Fm(OUT)108 b Fn(comm)466 b Fm(comm)o(unicator)9 b(asso)q(ciated)k(with)
e(RMA)h(windo)o(w)f(\(handle\))75 515 y Fq(int)23 b(MPI)p 245
515 15 2 v 17 w(Window)p 406 515 V 17 w(in\(MPI)p 567 515 V
16 w(Comm)g(comm\))75 601 y(MPI)p 150 601 V 17 w(WINDOW)p 311
601 V 16 w(IN\()h(COMM,)f(IERROR\))170 658 y(INTEGER)g(COMM,)h(IERROR)166
744 y Fu(Completes)13 b(the)h(transfers)e(of)h(all)h(up)q(dates)f(to)g(the)g
(public)j(windo)o(w)d(cop)o(y)g(in)o(to)g(the)g(priv)m(ate)h(cop)o(y)l(.)75
895 y Fn(MPI)p 160 895 14 2 v 16 w(WINDO)o(W)p 372 895 V 16
w(OUT\(comm)m(\))117 972 y Fm(OUT)108 b Fn(comm)466 b Fm(comm)o(unicator)9
b(asso)q(ciated)k(with)e(RMA)h(windo)o(w)f(\(handle\))75 1097
y Fq(int)23 b(MPI)p 245 1097 15 2 v 17 w(Window)p 406 1097
V 17 w(out\(MPI)p 591 1097 V 16 w(Comm)g(comm\))75 1183 y(MPI)p
150 1183 V 17 w(WINDOW)p 311 1183 V 16 w(OUT\()g(COMM,)h(IERROR\))170
1240 y(INTEGER)f(COMM,)h(IERROR)166 1326 y Fu(Completes)15
b(the)f(transfer)g(of)g(all)h(up)q(dates)g(to)f(the)h(priv)m(ate)g(windo)o(w)
f(cop)o(y)h(in)o(to)f(the)h(public)h(cop)o(y)l(.)166 1383 y(Supp)q(ose)22
b(that)e(a)g(pro)q(cess)h(lo)q(cally)h(writes)f(a)f(windo)o(w)h(lo)q(cation,)
i(and)d(that)g(lo)q(cation)i(is)f(later)75 1439 y(accessed)15
b(remotely)f(b)o(y)g(a)g(get.)19 b(Then,)c(the)f(get)g(will)i(surely)f
(return)f(the)g(v)m(alue)i(of)d(the)i(store)e(pro)o(vided)75
1495 y(that)131 1583 y(1.)22 b(a)15 b(call)h(to)e Fn(MPI)p
450 1583 14 2 v 16 w(WINDO)o(W)p 662 1583 V 16 w(OUT)i Fu(o)q(ccurred)g
(after)e(the)i(store)e(and)i(b)q(efore)f(the)g(get.)131 1674
y(2.)22 b(An)o(y)16 b(other)h(lo)q(cal)g(or)g(remote)f(up)q(date)h(to)f(that)
g(same)g(lo)q(cation)i(either)f(precedes)h(the)f(store)e(or)189
1730 y(follo)o(ws)g(the)g(get.)75 1817 y(If)j(the)g(call)h(to)e
Fn(MPI)p 433 1817 V 15 w(WINDO)o(W)p 644 1817 V 16 w(OUT)i
Fu(did)g(not)e(tak)o(e)g(place,)i(the)f(the)f(get)h(ma)o(y)f(or)g(ma)o(y)g
(not)g(return)75 1874 y(the)f(stored)g(v)m(alue.)25 b(Note)16
b(that)f(in)i(order)f(that)g(the)g(call)i(to)d Fn(MPI)p 1220
1874 V 16 w(WINDO)o(W)p 1432 1874 V 16 w(OUT)i Fu(precede)g(the)g(get,)75
1930 y(it)i(is)g(necessary)g(that)f(the)h(t)o(w)o(o)f(pro)q(cesses)h(sync)o
(hronize.)31 b(E.g.,)19 b(a)f(message)g(could)i(b)q(e)g(sen)o(t)e(b)o(y)h
(the)75 1987 y(\014rst)j(pro)q(cess)g(after)f(the)h(call)h(to)e(windo)o(w)p
845 1987 V 17 w(out)h(and)g(receiv)o(ed)h(b)o(y)f(the)g(other)f(pro)q(cess)i
(b)q(efore)f(the)75 2043 y(call)e(to)e(get.)30 b(Suc)o(h)19
b(sync)o(hronization)g(ma)o(y)f(also)h(o)q(ccur)g(indirectly)l(,)j(via)c(a)h
(c)o(hain)g(of)f(sync)o(hronizing)75 2100 y(comm)o(unications,)d(or)g(ma)o(y)
g(b)q(e)h(done)f(via)h(a)e(collectiv)o(e)j(call,)f(suc)o(h)g(as)f(barrier.)
166 2156 y(Supp)q(ose)k(that)e(an)g(RMA)h(op)q(eration)g(\(suc)o(h)g(as)f
(put\))h(up)q(dates)g(a)f(windo)o(w)h(lo)q(cation,)h(and)f(that)75
2212 y(lo)q(cation)d(is)g(later)f(read)h(lo)q(cally)l(.)21
b(The)15 b(load)f(will)i(surely)f(return)g(the)f(v)m(alue)i(of)e(the)g(put)h
(pro)o(vided)g(that)131 2300 y(1.)22 b(a)15 b(call)h(to)e Fn(MPI)p
450 2300 V 16 w(COPY)p 587 2300 V 17 w(IN)h Fu(o)q(ccurred)h(after)f(the)g
(put)g(and)h(b)q(efore)f(the)g(load.)131 2391 y(2.)22 b(An)o(y)14
b(other)g(lo)q(cal)i(\(store\))d(or)h(global)h(\(put,)f(accum)o(ulate,)h
(RMW\))f(up)q(date)h(to)f(that)g(same)g(lo)q(ca-)189 2447 y(tion)h(either)h
(precedes)g(the)f(put)h(or)e(follo)o(ws)i(the)f(load.)75 2534
y(If)j(the)g(call)h(to)e Fn(MPI)p 433 2534 V 16 w(WINDO)o(W)p
645 2534 V 16 w(IN)h Fu(did)h(not)e(tak)o(e)g(place,)j(then)e(the)g(load)g
(ma)o(y)f(or)g(ma)o(y)h(not)f(return)75 2591 y(the)e(v)m(alue)i(of)d(the)i
(put.)166 2647 y(Note)g(that)h(races)f(will)j(o)q(ccur)e(if)h(a)e(windo)o(w)h
(lo)q(cation)h(is)f(b)q(oth)g(remotely)g(up)q(dated)h(and)f(lo)q(cally)75
2704 y(read)f(during)g(an)g(external)g(ep)q(o)q(c)o(h)h(for)e(that)g(windo)o
(w.)21 b(The)16 b(lo)q(cal)h(read)f(ma)o(y)f(return)g(an)o(y)h(v)m(alue)h
(that)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967
272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959
554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780
y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959
1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959
1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959
1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959
1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959
1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959
2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959
2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959
2643 y(47)1959 2699 y(48)p eop
%%Page: 25 27
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(25)75 49 y(is)18 b(consisten)o(t)g(with)g(the)g(t)o
(w)o(o-cop)o(y)f(mo)q(del;)i(i.e.,)f(the)g(v)m(alue)h(of)e(eac)o(h)h(b)o(yte)
g(read)f(ma)o(y)g(b)q(e)i(the)f(v)m(alue)75 106 y(of)d(the)h(b)o(yte)g(at)f
(the)h(start)f(of)h(the)g(ep)q(o)q(c)o(h,)g(or)f(an)o(y)h(of)f(the)h(v)m
(alues)h(successiv)o(ely)h(written)e(in)h(this)f(b)o(yte)75
162 y(b)o(y)h(remote)f(up)q(dates)h(that)f(precede)i(or)e(are)g(concurren)o
(t)h(with)g(the)g(lo)q(cal)g(read.)25 b(And,)17 b(similarly)l(,)i(for)75
219 y(remote)c(reads)g(during)h(an)f(in)o(ternal)h(ep)q(o)q(c)o(h.)189
320 y Fk(R)n(ationale.)38 b Fu(A)13 b(stronger)f(coherence)i(mo)q(del)f(is)h
(required)f(where)g(accesses)h(to)e(shared)h(v)m(ariables)189
376 y(are)h(used)g(to)g(implemen)o(t)i(sync)o(hronization)f(op)q(erations.)20
b(A)14 b(w)o(eak)o(er)f(coherence)j(mo)q(del,)f(where)189 433
y(coherence)21 b(is)g(dela)o(y)o(ed)g(to)f(sync)o(hronization)h(p)q(oin)o
(ts,)h(is)e(adequate)h(for)f(data)f(sharing,)j(since)189 489
y(correct)d(sharing)h(will)i(require)e(some)g(sync)o(hronization)g(b)q(et)o
(w)o(een)h(pro)q(ducer)f(and)g(consumer.)189 545 y(The)13 b(coherence)h(op)q
(eration)g(can)f(dela)o(y)o(ed)h(to)e(that)h(sync)o(hronization)h(p)q(oin)o
(t,)f(without)g(c)o(hanging)189 602 y(program)k(seman)o(tics.)30
b(Dela)o(y)o(ed)19 b(coherence)h(allo)o(ws)e(for)g(impro)o(v)o(ed)h(p)q
(erformance,)g(esp)q(ecially)189 658 y(on)c(systems)f(where)i(explicit)h
(action)f(is)f(needed)i(for)d(coherence.)189 733 y(W)l(e)f(assume)g(that)g
(pro)q(cesses)h(will)h(sync)o(hronize)f(using)g(MPI)g(calls:)20
b(either)14 b(message)f(passing)g(or)189 789 y(RMW)g(op)q(erations.)20
b(The)14 b(coherence)h(rules)g(for)f(RMA)g(op)q(erations)g(mak)o(e)g(sure)g
(that)f(suc)o(h)i(syn-)189 846 y(c)o(hronization)g(will)i(w)o(ork)d
(correctly)l(.)20 b(On)15 b(the)g(other)g(hand,)g(data)f(sharing)h(b)q(et)o
(w)o(een)g(pro)q(cesses,)189 902 y(where)10 b(one)h(pro)q(cess)g(accesses)g
(the)g(data)e(lo)q(cally)l(,)14 b(using)d(loads)g(and)f(stores,)h(and)g
(another)f(pro)q(cess)189 959 y(accesses)16 b(the)g(data)f(remotely)l(,)h
(using)h(puts)f(and)g(gets,)f(will)j(w)o(ork)d(correctly)h(with)g(dela)o(y)o
(ed)h(co-)189 1015 y(herence.)i(Dela)o(y)o(ed)12 b(coherence)h(could)g(b)q(e)
g(implemen)o(ted)g(implicitl)q(y)l(,)i(b)o(y)d(p)q(erforming)g(coherence)189
1071 y(actions)j(on)g(eac)o(h)g(windo)o(w,)h(for)e(eac)o(h)i(MPI)f(call.)21
b(But)15 b(this)h(w)o(ould)g(b)q(e)g(an)f(exp)q(ensiv)o(e)i(solution.)189
1128 y(Instead,)f(w)o(e)f(pro)o(vide)i(explicit)h(coherence)f(enforcing)f
(calls,)h(namely)f Fn(MPI)p 1516 1128 14 2 v 16 w(WINDO)o(W)p
1728 1128 V 16 w(IN)g Fu(and)189 1184 y Fn(MPI)p 274 1184 V
15 w(WINDO)o(W)p 485 1184 V 17 w(OUT)p Fu(.)f(\()p Fk(End)g(of)i(r)n
(ationale.)p Fu(\))189 1285 y Fk(A)n(dvic)n(e)i(to)i(implementors.)66
b Fu(In)21 b(an)e(implemen)o(tation)i(where)f(lo)q(cal)h(accesses)f(are)f
(coheren)o(t)189 1342 y(with)h(remote)f(accesses,)i(then)g(the)f(calls)h(to)e
Fn(MPI)p 1087 1342 V 16 w(WINDO)o(W)p 1299 1342 V 16 w(IN)g
Fu(and)h Fn(MPI)p 1554 1342 V 16 w(WINDO)o(W)p 1766 1342 V
16 w(OUT)189 1398 y Fu(are)e(no)q(ops:)26 b(in)19 b(e\013ect,)f(the)g(t)o(w)o
(o)f(windo)o(w)i(copies)g(are)f(iden)o(tical.)31 b(Suc)o(h)19
b(is)f(the)h(case)f(if)h(RMA)189 1455 y(remote)e(accesses)h(are)f(executed)i
(b)o(y)e(a)h(soft)o(w)o(are)d(agen)o(t)i(running)i(on)f(the)g(same)f(pro)q
(cessor)g(as)189 1511 y(the)e(lo)q(cal)h(pro)q(cess,)f(or)f(on)h(a)g(comm)o
(unication)h(copro)q(cessor)e(that)h(is)g(cac)o(he)h(coheren)o(t)f(with)g
(the)189 1568 y(computation)f(pro)q(cessor.)19 b(The)14 b(same)g(holds)g
(true)g(in)h(an)f(implemen)o(tation)h(of)f(RMA)g(on)g(top)f(of)189
1624 y(coheren)o(t)i(shared)h(memory)l(,)f(where)h(remote)f(accesses)h(are)f
(p)q(erformed)h(directly)h(b)o(y)f(the)f(origin)189 1681 y(pro)q(cess.)189
1755 y(The)h(non)o(trivial)g(case)g(is)g(where)g(RMA)g(remote)f(accesses)h
(are)f(not)g(cac)o(he)h(coheren)o(t)g(with)g(lo)q(cal)189 1811
y(accesses.)k(There)15 b(are)g(sev)o(eral)g(cases)h(to)e(consider)231
1904 y(1.)22 b(RMA)f(is)h(implemen)o(ted)h(using)g(non-coheren)o(t)e(shared)h
(memory)l(,)g(with)g(write-through)289 1960 y(incoheren)o(t)g(cac)o(hes.)37
b(Both)21 b(lo)q(cal)i(and)e(remote)f(writes)i(go)e(directly)i(to)f(memory)l
(,)h(but)289 2017 y(b)q(oth)15 b(lo)q(cal)h(and)g(remote)e(reads)h(ma)o(y)g
(go)f(to)h(cac)o(he)g(\(Cra)o(y)f(T90,)g(Cra)o(y)h(T3D?\).)231
2088 y(2.)22 b(RMA)e(is)h(implemen)o(ted)i(using)e(non-coheren)o(t)g(shared)f
(memory)l(,)h(with)g(write-bac)o(k)g(in-)289 2145 y(coheren)o(t)c(cac)o(hes.)
28 b(Both)17 b(lo)q(cal)i(and)f(remote)f(accesses)h(ma)o(y)f(go)g(to)g(cac)o
(he,)h(rather)f(than)289 2201 y(memory)d(\(NEC)h(Cenju-3,)g(Mercury)g(RA)o
(CE)g(i860?\).)231 2273 y(3.)22 b(RMA)d(is)h(implemen)o(ted)h(using)f(a)f
(non-coheren)o(t)h(DMA)e(engine.)34 b(Remote)19 b(accesses)h(go)289
2329 y(directly)g(to)e(memory)l(,)h(whereas)g(lo)q(cal)h(accesses)f(\(b)q
(oth)g(reads)g(and)g(writes\))f(ma)o(y)g(go)h(to)289 2385 y(cac)o(he)c(\(Cra)
o(y)f(T3D?\).)189 2478 y(Assume)h(that)g(the)h(windo)o(w)f(is)h(cac)o(he)g
(line)h(aligned.)22 b(A)16 b(call)g(to)f Fn(MPI)p 1404 2478
V 16 w(WINDO)o(W)p 1616 2478 V 16 w(OUT\(comm)m(\))189 2534
y Fu(is)k(a)f(no)q(op)g(in)h(the)g(\014rst)f(case)g(\(except)h(that)e(a)h
(memory)g(sync)h(ma)o(y)e(b)q(e)i(needed)h(to)e(force)g(the)189
2591 y(stores)12 b(out)h(of)g(the)g(store)g(queue,)h(on)f(some)g(systems\).)
19 b(In)14 b(the)f(second)h(and)f(third)h(case,)f(the)h(call)189
2647 y(will)k(\015ush)g(the)f(cac)o(he)g(lines)h(in)g(the)f(windo)o(w)g(from)
f(the)h(calling)i(pro)q(cessor)d(cac)o(he)h(to)g(memory)189
2704 y(\(it)e(is)g(su\016cien)o(t)h(to)f(\015ush)h(dirt)o(y)f(lines\).)-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 26 28
bop 75 -100 a Fu(26)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)189 49 y Fu(A)20 b(call)h(to)e Fn(MPI)p 476 49 14 2
v 16 w(WINDO)o(W)p 688 49 V 16 w(IN)h Fu(will)i(in)o(v)m(alidate)g(the)e(cac)
o(he)g(lines)i(in)f(the)f(windo)o(w)h(from)e(the)189 106 y(calling)e(pro)q
(cessor)e(cac)o(he)g(\(it)g(is)h(su\016cien)o(t)g(to)e(in)o(v)m(alidate)k
(lines)e(that)f(w)o(ere)g(remotely)g(up)q(dated)189 162 y(in)j(the)f(last)g
(ep)q(o)q(c)o(h\).)26 b(Since)19 b(no)e(lo)q(cal)h(accesses)f(w)o(ere)g(p)q
(erformed)h(in)f(the)h(last)f(ep)q(o)q(c)o(h,)h(all)g(the)189
219 y(windo)o(w)e(cac)o(he)h(lines)h(are)e(clean,)h(and)g(a)f(\015ush)h(call)
g(ma)o(y)f(b)q(e)h(used,)g(instead)g(of)f(an)g(in)o(v)m(alidate.)189
275 y(\(A)f(\\\015ush")h(copies)h(bac)o(k)e(the)h(dirt)o(y)g(lines)h(to)f
(memory)f(and)h(in)o(v)m(alidate)h(the)f(en)o(tries)g(in)h(cac)o(he;)189
332 y(an)g(\\in)o(v)m(alidate")i(do)q(es)f(not)g(cop)o(y)f(bac)o(k)h(the)f
(lines)j(in)e(memory)l(.)27 b(On)18 b(some)g(systems,)f(\015ush)h(is)189
388 y(allo)o(w)o(ed)c(in)g(user)g(mo)q(de,)g(whereas)g(in)o(v)m(alidate)h
(requires)g(sup)q(ervisory)f(mo)q(de,)g(so)g(that)f(a)g(\015ush)h(is)189
444 y(c)o(heap)q(er.)20 b(W)l(e)14 b(assume)h(that)f(the)g(\015ush)h
(implemen)o(tation)h(is)f(smart)f(enough)h(not)f(to)g(cop)o(y)g(bac)o(k)189
501 y(to)g(memory)h(clean)h(lines.\))189 576 y(In)h(addition,)g(in)g(the)f
(second)h(case,)g(the)f(co)q(de)h(for)f(put,)g(accum)o(ulate)h(or)e(RMW)i
(requires)g(that)189 632 y(the)f(written)h(data)f(b)q(e)h(\015ushed)h(from)e
(the)h(origin)g(cac)o(he)g(b)q(efore)g(the)g(RMA)g(call)g(is)h(allo)o(w)o(ed)
f(to)189 689 y(complete)c(at)f(the)h(target.)18 b(In)c(the)f(\014rst)f(case,)
h(a)g(memory)f(sync)h(ma)o(y)f(b)q(e)i(needed)g(to)e(\015ush)i(stores)189
745 y(out)g(of)h(the)g(store)g(queue.)189 820 y(An)d(additional)h(problem)f
(is)g(raised)g(b)o(y)g(p)q(ossible)h(memory)e(corruption)h(at)f(windo)o(w)h
(b)q(oundaries.)189 877 y(Supp)q(ose)j(that)f(a)h(lo)q(cal)g(access)g(up)q
(dates)g(a)g(b)o(yte)f(at)g(the)h(windo)o(w)g(b)q(oundary)l(,)g(on)g(systems)
f(that)189 933 y(supp)q(orts)i(only)i(w)o(ord)e(sized)h(memory)f(accesses.)25
b(The)17 b(pro)q(cessor)f(will)j(execute)e(a)f(w)o(ord)g(read,)189
989 y(b)o(yte)f(mo)q(dify)l(,)i(w)o(ord)e(write)h(cycle;)h(the)f(written)f(w)
o(ord)h(ma)o(y)f(o)o(v)o(erlap)g(with)h(the)g(windo)o(w.)22
b(This)189 1046 y(write)17 b(ma)o(y)f(o)q(ccur)h(during)h(a)e(remote)h(ep)q
(o)q(c)o(h)h(for)e(the)h(windo)o(w,)g(and)g(ma)o(y)f(cause)h(a)g(put)g(to)f
(b)q(e)189 1102 y(erroneously)f(o)o(v)o(erwritten.)189 1177
y(Supp)q(ose)c(that)f(a)g(cac)o(he)h(line)h(partially)f(o)o(v)o(erlaps)f(a)g
(windo)o(w.)19 b(A)10 b(lo)q(cal)i(access)e(to)g(the)g(line)i(outside)189
1234 y(the)k(windo)o(w)g(ma)o(y)f(cause)h(the)g(line)h(to)e(b)q(e)i(loaded)f
(in)h(cac)o(he.)22 b(The)16 b(line)h(ma)o(y)e(b)q(e)i(subsequen)o(tly)189
1290 y(written)e(bac)o(k,)g(and)g(erroneously)h(o)o(v)o(erwrite)e(a)h(put.)
189 1365 y(The)h(solution)h(to)e(these)i(problem)g(is)f(that)g(suc)o(h)g
(lines)i(or)e(w)o(ords)f(are)h(not)g(remotely)g(accessed.)189
1422 y(A)f(safet)o(y)e(area)i(is)g(preserv)o(ed)g(at)f(the)h(windo)o(w)g(b)q
(oundaries,)h(and)f(an)o(y)g(remote)f(up)q(date)h(to)g(suc)o(h)189
1478 y(b)q(oundary)g(is)h(stored)e(in)i(a)f(separate)g(bu\013er;)f(this)i
(bu\013er)f(is)h(used)f(to)g(satisfy)g(remote)f(accesses.)189
1535 y(When)c(a)g(call)i(to)d Fn(MPI)p 563 1535 V 16 w(WINDO)o(W)p
775 1535 V 16 w(IN)h Fu(o)q(ccurs,)h(then)g(the)f(calling)j(pro)q(cessor)d
(copies)h(this)f(separate)189 1591 y(bu\013er)17 b(on)o(to)f(the)h(windo)o(w)
g(b)q(oundary)l(.)26 b(\(In)17 b(e\013ect,)g(w)o(e)f(main)o(tain)i(in)g(soft)
o(w)o(are)d(the)i(coherence)189 1647 y(of)d(the)i(t)o(w)o(o)e(windo)o(w)h
(copies.\))189 1722 y(The)e(same)g(soft)o(w)o(are)e(approac)o(h)i(to)g
(coherence)h(can)f(b)q(e)h(used)g(on)f(systems)g(where)h(shared)f(mem-)189
1779 y(ory)g(is)i(a)o(v)m(ailable,)h(in)f(cases)f(where)h(the)f(windo)o(w)h
(is)f(not)g(in)i(shared)e(memory)l(.)19 b(One)c(will)h(reserv)o(e)189
1835 y(in)k(shared)f(memory)g(a)g(public)j(cop)o(y)d(of)g(the)g(windo)o(w.)33
b(Lo)q(cal)20 b(accesses)g(to)f(the)g(windo)o(w)h(ac-)189 1892
y(cess)14 b(the)h(priv)m(ate)g(cop)o(y)l(,)f(whereas)h(RMA)g(remote)e
(accesses)i(access)g(the)f(public)j(cop)o(y)e(in)g(shared)189
1948 y(memory)l(.)31 b(When)20 b Fn(MPI)p 612 1948 V 16 w(WINDO)o(W)p
824 1948 V 16 w(IN)f Fu(is)g(called,)j(then)d(the)h(public,)i(shared)d(cop)o
(y)g(is)h(copied)189 2005 y(in)o(to)e(the)h(priv)m(ate)h(cop)o(y;)g(when)f
Fn(MPI)p 854 2005 V 16 w(WINDO)o(W)p 1066 2005 V 16 w(OUT)h
Fu(is)f(called,)i(then)e(the)g(priv)m(ate)h(cop)o(y)e(is)189
2061 y(copied)k(on)o(to)d(the)i(public,)j(shared)d(cop)o(y)l(.)37
b(V)l(arious)21 b(optimizations)g(\(similar)h(to)e(those)g(used)189
2118 y(in)e(DSVM)g(systems\))f(can)i(b)q(e)f(applied)i(to)e(a)o(v)o(oid)f
(full)j(windo)o(w)e(cop)o(ying.)29 b(\()p Fk(End)18 b(of)h(advic)n(e)g(to)189
2174 y(implementors.)p Fu(\))166 2355 y Fi(Alternativ)o(es)o(:)34
b Fm(A)14 b(p)q(ossible)g(alternativ)o(e)f(is)g(to)h(drop)f(the)h(condition)f
(that)h(a)f(windo)o(w)f(can)i(b)q(e)g(up)q(dated)75 2405 y(only)d(lo)q(cally)
e(or)j(only)e(remotely)h(during)g(an)g(ep)q(o)q(c)o(h.)18 b(E.g.,)10
b(b)o(y)h(allo)o(wing)e(b)q(oth)j(t)o(yp)q(es)g(of)e(up)q(dates,)j(if)d(they)
i(access)75 2455 y(distinct)i(lo)q(cations.)166 2504 y(The)g(di\016cult)o(y)m
(,)d(as)i(compared)g(to)g(the)h(main)d(prop)q(osal,)i(is)g(that)g(it)g(is)g
(not)g(p)q(ermissible)g(an)o(ymore)f(to)h(cop)o(y)75 2554 y(the)j(en)o(tire)g
(windo)o(w)e(when)h(windo)o(w)p 667 2554 13 2 v 15 w(in)f(or)h(windo)o(w)p
921 2554 V 14 w(out)h(are)f(called.)21 b(This,)15 b(b)q(ecause)i(b)q(oth)e
(the)h(priv)n(ate)f(cop)o(y)75 2604 y(and)10 b(the)h(public)g(cop)o(y)f(ma)o
(y)f(ha)o(v)o(e)h(b)q(een)h(up)q(dated)h(during)e(the)h(last)f(ep)q(o)q(c)o
(h,)i(in)e(di\013eren)o(t)h(parts,)h(and)e(the)h(up)q(dates)75
2654 y(need)18 b(to)e(b)q(e)h(merged.)26 b(Supp)q(ose)17 b(one)g(main)o
(tains)d(a)j(dirt)o(y)f(bit)g(for)h(eac)o(h)g(b)o(yte.)26 b(The)17
b(dirt)o(y)g(bit)f(for)g(a)g(b)o(yte)h(in)75 2704 y(the)d(priv)n(ate)g(cop)o
(y)f(is)h(set)g(b)o(y)f(a)h(lo)q(cal)e(up)q(date)j(to)e(the)h(b)o(yte,)g(and)
f(the)h(dirt)o(y)g(bit)f(for)g(a)g(b)o(yte)h(in)g(the)g(public)f(cop)o(y)1967
46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 27 29
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(27)75 49 y Fm(is)16 b(set)g(b)o(y)g(a)f(remote)g(up)q
(date)i(to)e(the)h(b)o(yte.)24 b(When)16 b(a)f(call)g(to)h
Fl(MPI)p 1169 49 13 2 v 14 w(WINDO)o(W)p 1362 49 V 15 w(OUT)f
Fm(o)q(ccurs,)i(then)f(an)o(y)g(b)o(yte)75 99 y(that)i(is)f(dirt)o(y)g(in)g
(the)h(priv)n(ate)g(cop)o(y)f(is)h(copied)f(to)h(the)g(public)f(cop)o(y)h
(and)f(mark)o(ed)f(clean.)29 b(When)18 b(a)f(call)g(to)75 149
y Fl(MPI)p 152 149 V 14 w(WINDO)o(W)p 345 149 V 15 w(IN)12
b Fm(o)q(ccurs,)h(then)g(an)o(y)e(b)o(yte)h(that)g(is)g(dirt)o(y)g(in)f(the)i
(public)e(cop)o(y)h(is)g(copied)g(to)g(the)g(priv)n(ate)g(cop)o(y)75
199 y(and)i(mark)o(ed)e(clean.)166 249 y(It)19 b(is)f(hard)h(to)f(implemen)o
(t)e(this)i(proto)q(col)h(e\016cien)o(tly)f(on)h(systems)f(with)g
(non-coheren)o(t)i(cac)o(hes.)34 b(The)75 298 y(\015ush)12
b(in)f Fl(MPI)p 295 298 V 15 w(COPY)p 423 298 V 13 w(OUT)g
Fm(should)h(\015ush)g(only)f(dirt)o(y)g(b)o(ytes,)h(to)g(a)o(v)o(oid)e(o)o(v)
o(erwriting)h(b)o(ytes)i(that)f(w)o(ere)g(up)q(dated)75 348
y(remotely)m(.)26 b(But,)19 b(t)o(ypically)m(,)c(a)i(dirt)o(y)g(line)g(is)g
(\015ushed)h(in)f(its)g(en)o(tiret)o(y)m(,)g(ev)o(en)h(if)f(only)f(some)g(of)
h(the)h(b)o(ytes)g(are)75 398 y(dirt)o(y)d(\(the)h(cac)o(he)g(main)o(tains)d
(only)h(one)i(dirt)o(y)f(bit)g(p)q(er)h(line\).)22 b(Th)o(us,)15
b(if)f(part)i(of)f(the)g(line)g(w)o(as)g(up)q(dated)h(b)o(y)f(a)75
448 y(lo)q(cal)g(store)h(and)f(part)h(w)o(as)g(up)q(dated)g(b)o(y)f(a)g
(remote)h(put,)f(the)h(\015ush)g(will)e(o)o(v)o(erwrite)i(the)g(put.)24
b(Similarl)o(y)m(,)12 b(the)75 498 y(in)o(v)n(alidate)f(in)h
Fl(MPI)p 387 498 V 14 w(COPY)p 514 498 V 13 w(IN)g Fm(should)g(in)o(v)n
(alidate)f(only)h(b)o(ytes)h(that)f(where)i(remotely)d(up)q(dated;)i(ho)o(w)o
(ev)o(er,)g(one)75 547 y(can)j(only)g(in)o(v)n(alidate)e(full)h(lines.)24
b(Th)o(us,)17 b(if)e(part)h(of)g(the)h(line)e(w)o(as)h(up)q(dated)h(b)o(y)f
(a)g(lo)q(cal)f(store)i(and)f(part)g(w)o(as)75 597 y(up)q(dated)i(b)o(y)g(a)f
(remote)g(put,)h(the)g(in)o(v)n(alidate)e(will)g(cause)j(the)f(store)g(to)g
(b)q(e)g(lost.)28 b(It)18 b(w)o(ould)f(seem)g(that)h(one)75
647 y(needs)d(to)e(use)h(soft)o(w)o(are)f(tec)o(hniques)h(similar)d(to)i
(those)h(used)g(in)f(DSVM)g(systems)g(in)g(order)h(to)f(implem)o(en)o(t)e
(this)75 697 y(sc)o(heme.)166 747 y(Another)16 b(option)e(is)h(to)g(a)o(v)o
(oid)e(the)j(use)g(of)e(explicit)h(windo)o(w)p 1147 747 V 14
w(in)f(and)h(windo)o(w)p 1430 747 V 14 w(out)g(coherence)i(calls,)e(but)75
796 y(sp)q(ecify)j(that)f(con\015icting)g(lo)q(cal)g(and)g(remote)g(accesses)
j(are)d(erroneous:)27 b(suc)o(h)18 b(accesses)i(m)o(ust)c(b)q(e)i(explicitly)
75 846 y(sync)o(hronized.)166 896 y(Supp)q(ose)d(that)g(a)f(pro)q(cess)i
(stores)f(a)f(v)n(alue)g(at)g(a)g(lo)q(cation)g(in)f(a)h(windo)o(w,)g(and)g
(that)g(lo)q(cation)f(is)i(accessed)75 946 y(using)e(an)g(RMA)g(op)q(eration)
g(\(suc)o(h)h(as)f(get\))g(b)o(y)g(another)h(pro)q(cess)h(\(or)e(p)q(ossibly)
g(the)h(same)e(pro)q(cess\).)20 b(Then)13 b(the)75 996 y(get)h(returns)i(the)
e(v)n(alue)f(of)h(the)g(store)h(pro)o(vided)f(that)134 1070
y(1.)22 b(The)14 b(store)h(precedes)h(the)f(get.)134 1145 y(2.)22
b(An)o(y)11 b(other)h(lo)q(cal)e(\(store\))i(or)f(global)f(\(put,)i(accum)o
(ulate,)e(RMW\))g(up)q(date)i(to)f(that)h(same)e(lo)q(cation)g(either)189
1195 y(precedes)16 b(the)f(store)f(or)g(follo)o(ws)e(the)j(get.)166
1270 y(Supp)q(ose)e(that)g(an)f(RMA)g(op)q(eration)g(\(suc)o(h)i(as)e(put\))h
(up)q(dates)h(a)e(windo)o(w)f(lo)q(cation,)h(and)g(that)h(lo)q(cation)e(is)75
1319 y(later)i(accessed)i(b)o(y)d(the)h(lo)q(cal)f(pro)q(cess)j(using)d(a)g
(load.)17 b(The)c(load)f(returns)i(the)f(v)n(alue)f(of)g(the)i(put)e(pro)o
(vided)h(that)134 1394 y(1.)22 b(the)14 b(put)g(precedes)j(the)d(load.)134
1469 y(2.)22 b(An)o(y)11 b(other)h(lo)q(cal)e(\(store\))i(or)f(global)f
(\(put,)i(accum)o(ulate,)e(RMW\))g(up)q(date)i(to)f(that)h(same)e(lo)q
(cation)g(either)189 1519 y(precedes)16 b(the)f(put)f(or)f(follo)o(ws)g(the)h
(load.)166 1593 y(If)h(a)h(remote)f(RMA)h(get)g(follo)o(ws)e(a)i(lo)q(cal)e
(store)j(then)g(the)f(lo)q(cal)f(pro)q(cess)j(m)o(ust)c(ha)o(v)o(e)i
(executed)i(a)d(send-)75 1643 y(lik)o(e)h(MPI)i(call)e(after)i(the)g(store)g
(and)f(b)q(efore)h(the)f(get.)29 b(\(A)17 b(send-lik)o(e)g(op)q(eration)g(is)
h(a)e(send,)j(or)e(a)g(collectiv)o(e)75 1693 y(comm)o(unication)8
b(call,)j(an)h(RMA)f(call,)g(etc.;)h(i.e.,)f(an)g(op)q(eration)h(that)g(is)f
(kno)o(wn)g(to)h(precede)i(another)e(matc)o(hing)75 1743 y(op)q(eration)k(at)
f(another)h(pro)q(cess\).)26 b(If)16 b(a)f(remote)g(RMA)h(put)g(precedes)i(a)
e(lo)q(cal)f(load,)f(then)j(the)f(lo)q(cal)f(pro)q(cess)75
1793 y(m)o(ust)9 b(ha)o(v)o(e)g(executed)j(a)e(receiv)o(e-lik)o(e)g(MPI)g
(call)f(after)h(the)g(put)g(and)g(b)q(efore)h(the)f(load.)16
b(\(A)10 b(receiv)o(e-lik)o(e)g(op)q(eration)75 1843 y(is)16
b(a)g(receiv)o(e,)i(a)e(collectiv)o(e)h(call,)e(an)i(access)h(to)e(a)g(windo)
o(w)g(coun)o(ter,)h(a)f(get,)h(etc.;)h(i.e.,)d(an)h(op)q(eration)h(that)f(is)
75 1892 y(kno)o(wn)d(to)h(follo)o(w)e(a)h(matc)o(hing)f(op)q(eration)h(at)h
(another)g(pro)q(cess.\))20 b(The)14 b(seman)o(tics)g(of)f(the)h(last)f
(option)g(can)h(b)q(e)75 1942 y(obtained)i(b)o(y)f(treating)h(an)o(y)g
(send-lik)o(e)g(call)f(as)h(a)g(call)f(to)g Fl(MPI)p 1095 1942
V 15 w(WINDO)o(W)p 1289 1942 V 14 w(OUT)p Fm(,)g(and)h(an)o(y)f(receiv)o
(e-lik)o(e)i(MPI)75 1992 y(call)c(as)h(a)g(call)f(to)h Fl(MPI)p
441 1992 V 14 w(WINDO)o(W)p 634 1992 V 14 w(IN)g Fm(\(some)f(calls)h(can)g(b)
q(e)g(b)q(oth\).)166 2048 y(The)g(di\016cult)o(y)f(in)h(implemen)o(tatio)o(n)
d(is)j(that)g(eac)o(h)h(MPI)f(op)q(eration)g(will)e(no)o(w)i(a\013ect)h(all)d
(windo)o(ws.)18 b(This)75 2105 y(will)f(p)q(oten)o(tially)f(add)i(a)g
(signi\014can)o(t)g(o)o(v)o(erhead)g(to)g(all)f(MPI)h(calls)g(on)g
(non-coheren)o(t)h(systems.)31 b(A)18 b(p)q(ossible)75 2161
y(compromise)d(is)j(to)f(sp)q(ecify)i(that)e(con\015icting)h(accesses)i(in)o
(v)o(olving)15 b(a)j(windo)o(w)e(m)o(ust)h(b)q(e)h(co)q(ordinated)g(using)75
2218 y(sp)q(eci\014c)g(MPI)e(calls.)24 b(One)17 b(c)o(hoice)g(is)f(that)g
(accesses)j(to)d(a)g(windo)o(w)f(m)o(ust)g(b)q(e)i(sync)o(hronized)g(with)f
(MPI)g(calls)75 2274 y(whic)o(h)d(ha)o(v)o(e)h(the)g(windo)o(w)e(comm)o
(unicator)f(as)j(an)f(argumen)o(t.)k(This)c(includes)h(barrier,)g(fence,)g
(send)g(or)g(receiv)o(e)75 2331 y(and)d(windo)o(w)f(coun)o(ter)j(accesses.)19
b(Th)o(us,)12 b(only)e(MPI)i(calls)f(with)g(this)g(comm)o(unicator)d(argumen)
o(t)j(will)e(b)q(e)j(slo)o(w)o(ed)75 2387 y(do)o(wn)i(with)f(windo)o(w)g
(coherence)j(actions.)i(This)c(last)g(option)f(should)h(b)q(e)g(considered)h
(seriously)m(.)166 2526 y Fu(Windo)o(w)g(coun)o(ter)g(incremen)o(t)h(is)g
(alw)o(a)o(ys)e(atomic.)75 2633 y Fp(Example)k(4.5)k Fu(W)l(e)16
b(presen)o(t)f(sev)o(eral)h(examples)g(that)f(explain)i(the)e(use)h(of)f
(memory)f(coherence)j(op-)75 2689 y(erations.)-32 46 y Fj(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 28 30
bop 75 -100 a Fu(28)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)166 49 y Fu(Example)k(1:)25 b(pro)q(ducer-consumer;)20
b(pro)q(ducer)f(main)o(tains)g(queue)g(p)q(oin)o(ters)f(and)h(pushes)g(data)
75 106 y(to)c(the)g(consumer)g(memory)l(.)75 200 y Fq(main\(\))23
b({)75 256 y(typedef)g(struct)g({)147 312 y(float)g(somedata[6];)147
369 y(int)g(somedross[7];)147 425 y(char)g(someotherdross[8];)147
482 y(})g(SOMEOBJECT;)75 538 y(SOMEOBJECT)f(*object;)75 595
y(long)h(int)h(produced,)e(consumed;)75 708 y(blens[3])h(=)g({6,)h(7,)g(8};)
75 764 y(MPI_Datatypes)e(types[3])h(=)h({MPI_FLOAT,)e(MPI_INT,)h(MPI_CHAR};)
75 821 y(MPI_Datatype)f(objecttype;)75 877 y(MPI_Comm)h(comm1,)g(comm2;)75
990 y(MPI_Type_simple_struct\(3,)d(blens,)j(types,)h(objecttype\);)75
1046 y(MPI_Type_commit\(objecttype)o(\);)75 1159 y(if)g(\(myrank)e(==)i(0\))g
({)f(/*)h(producer)f(code)g(*/)147 1272 y(object)g(=)g
(\(SOMEOBJECT*\)malloc\(sizeof\(SO)o(MEOBJEC)o(T\)\);)147 1329
y(MPI_RMA_init\(NULL,)e(0,)j(0,)f(MPI_COMM_WORLD;)f(comm1\);)147
1385 y(MPI_RMA_init\(NULL,)f(0,)j(0,)f(MPI_COMM_WORLD;)f(comm2\);)147
1498 y(while)h(\(1\))g({)218 1554 y(produce\(object\);)218
1611 y(MPI_Put\(object,)f(1,)i(objecttype,)e(1,)i(produced\045QLEN,)910
1667 y(1,)g(objecttype,)e(1,)i(comm1\);)218 1724 y(while\()f(produced)g(>=)h
(\(consumed+QLEN\)\))290 1780 y(MPI_Get\(&consumed,)d(1,)j(MPI_LONGINT,)e(1,)
862 1837 y(0,)i(MPI_LONGINT,)e(0,)i(comm2\);)218 1893 y(produced++;)218
1950 y(})75 2063 y(elseif)f(\(myrank)g(==)h(1\))f({)h(/*)f(consumer)g(code)h
(*/)147 2175 y(object)f(=)g(\(SOMEOBJECT*\)malloc\(QLEN*size)o(of\(SOME)o
(OBJECT\))o(\);)147 2232 y(MPI_RMA_init\(&object,)e
(QLEN*sizeof\(SOMEOBJECT\))o(,)457 2288 y(sizeof\(SOMEOBJECT},)g
(MPI_COMM_WORLD,)h(comm1\);)147 2345 y(MPI_RMA_INIT\(&consumed,)e(1,)k
(sizeof\(consumed\),)d(MPI_COMM_WORLD,)h(comm2\);)147 2458
y(while)h(\(1\))g({)218 2514 y(while)g(\(consumed)g(>=)h(produced\))290
2571 y(MPI_Get_counter\(comm1,)d(&produced\);)218 2627 y
(MPI_Window_in\(comm1\);)218 2684 y(consume\(object+consumed\045QLE)o(N\);)
1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967
272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959
554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780
y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959
1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959
1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959
1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959
1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959
1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959
2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959
2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959
2643 y(47)1959 2699 y(48)p eop
%%Page: 29 31
bop 75 -100 a Fo(4.8.)29 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)g(COHERENCE)h
(OPERA)l(TIONS)406 b Fu(29)218 49 y Fq(consumed++;)218 106
y(MPI_Window_out\(comm2\))147 162 y(})75 219 y(})166 304 y
Fu(Example)16 b(2:)j(similar)e(consumer-pro)q(ducer)f(co)q(de,)f(but)h
(consumer)f(pulls)i(data)d(from)h(pro)q(ducer.)75 389 y Fq(main\(\))23
b({)75 445 y(typedef)g(struct)g({)147 502 y(float)g(somedata[6];)147
558 y(int)g(somedross[7];)147 615 y(char)g(someotherdross[8];)147
671 y(})g(SOMEOBJECT;)75 728 y(SOMEOBJECT)f(*object;)75 784
y(long)h(int)h(produced,)e(consumed;)75 897 y(blens[3])h(=)g({6,)h(7,)g(8};)
75 953 y(MPI_Datatypes)e(types[3])h(=)h({MPI_FLOAT,)e(MPI_INT,)h(MPI_CHAR};)
75 1010 y(MPI_Datatype)f(objecttype;)75 1066 y(MPI_Comm)h(comm1,)g(comm2;)75
1179 y(MPI_Type_simple_struct\(3,)d(blens,)j(types,)h(objecttype\);)75
1236 y(MPI_Type_commit\(objecttype)o(\);)75 1349 y(if)g(\(myrank)e(==)i(0\))g
({)f(/*)h(producer)f(code)g(*/)147 1462 y(object)g(=)g
(\(SOMEOBJECT*\)malloc\(QLEN*size)o(of\(SOME)o(OBJECT\))o(\);)147
1518 y(MPI_RMA_init\(&object,)e(QLEN*sizeof\(SOMEOBJECT\))o(,)457
1574 y(sizeof\(SOMEOBJECT},)g(MPI_COMM_WORLD,)h(comm1\);)147
1631 y(MPI_RMA_INIT\(&produced,)e(1,)k(sizeof\(produced\),)d(MPI_COMM_WORLD,)
h(comm2\);)147 1744 y(while)h(\(1\))g({)218 1800 y(while)g(\(produced)g(>=)h
(\(consumed+QLEN\)\))290 1857 y(MPI_Get_counter\(comm1,)d(&consumed\);)218
1913 y(produce\(object+produced\045QLE)o(N\);)218 1970 y
(MPI_Window_out\(comm1\);)218 2026 y(produced++;)218 2083 y
(MPI_Window_out\(comm2\);)147 2139 y(})75 2252 y(elseif)i(\(myrank)g(==)h
(1\))f({)h(/*)f(consumer)g(code)h(*/)147 2421 y(object)f(=)g
(\(SOMEOBJECT*\)malloc\(sizeof\(SO)o(MEOBJEC)o(T\)\);)147 2478
y(MPI_RMA_init\(NULL,)e(0,)j(0,)f(MPI_COMM_WORLD;)f(comm1\);)147
2534 y(MPI_RMA_init\(NULL,)f(0,)j(0,)f(MPI_COMM_WORLD;)f(comm2\);)147
2647 y(while)h(\(1\))g({)218 2704 y(while)g(\(produced)g(<=)h(consumed\))-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 30 32
bop 75 -100 a Fu(30)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)290 49 y Fq(MPI_Get\(&produced,)21 b(1,)j(MPI_LONGINT,)e(0,)i(0,)815
106 y(1,)f(MPI_LONGINT,)g(0,)g(comm2\);)218 162 y(MPI_Get\(object,)f(1,)i
(objecttype,)e(1,)i(produced\045QLEN,)910 219 y(1,)g(objecttype,)e(1,)i
(comm1\);)218 275 y(consume\(object\);)218 332 y(})75 388 y(})166
482 y Fu(Example)16 b(3:)j(lo)q(cal)e(store,)d(follo)o(w)o(ed)h(b)o(y)g(get,)
g(in)h(a)f(global)g(gather)g(comm)o(unication)h(pattern.)75
576 y Fq(...)75 632 y(double)23 b(a[1000],)g(b[1000];)75 689
y(int)g(index[1000];)75 745 y(...)75 801 y(MPI_RMA_init\(a,)f
(1000*sizeof\(double\),)f(sizeof\(double\),)934 858 y(MPI_WORLD,)i(comm\);)75
914 y(....)75 971 y(while)g(\(notdone\))g({)147 1027 y(update1\(a\);)147
1084 y(update2\(index\);)147 1140 y(MPI_Window_out\(comm\);)147
1197 y(MPI_Barrier\(comm\);)147 1253 y(for)g(\(i=0;)g(i<1000;)g(i++\))g({)218
1310 y(rank)g(=)h(index[i]/numproc;)218 1366 y(disp)f(=)h
(index[i]\045numproc;)218 1422 y(MPI_Get\(&b[i],)e(1,)i(MPI_REAL,)e(rank,)i
(disp,)f(1,)g(MPI_REAL,)g(comm\);)218 1479 y(})147 1535 y
(MPI_Window_in\(comm\);)147 1592 y(use\(b\);)75 1648 y(})75
1705 y(...)75 1799 y Fu(Assignmen)o(ts)14 b(to)f(arra)o(y)g
Fq(a)h Fu(are)f(not)h(allo)o(w)o(ed)g(after)f(the)h(call)h(to)e(Windo)o(w)p
1347 1799 14 2 v 17 w(out,)g(and)h(b)q(efore)h(the)f(call)h(to)75
1855 y(Windo)o(w)p 242 1855 V 17 w(in.)166 1911 y(Example)e(4:)19
b(pro)q(ducer-consumer)14 b(co)q(de.)20 b(The)13 b(pro)q(ducer)h(rep)q
(eatedly)g(pro)q(duces)g(a)e(new)i(v)o(ersion)75 1968 y(of)d(an)h(ob)s(ject.)
18 b(The)12 b(consumer)g(o)q(ccasionally)i(pic)o(ks)e(and)g(consumes)g(the)g
(latest)g(v)o(ersion)g(of)f(the)h(ob)s(ject.)75 2024 y(T)l(o)j(simplify)i
(the)e(example,)h(w)o(e)f(assume)g(a)g(homogeneous)g(en)o(vironmen)o(t.)75
2118 y Fq(main)23 b(\(\))h({)75 2175 y(struct)f({)147 2231
y(double)g(data[100];)147 2288 y(long)g(int)g(version;)147
2344 y(})g(object;)75 2400 y(double)g(localcopy[100];)75 2457
y(MPI_Comm)g(comm;)75 2570 y(MPI_RMA_init\(&object,)e(sizeof\(object\),)h(1,)
767 2626 y(MPI_COMM_WORLD,)g(comm\))1967 46 y Fj(1)1967 103
y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967
385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959
667 y(12)1959 724 y(13)1959 780 y(14)1959 836 y(15)1959 893
y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959 1119
y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959 1345
y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959 1570
y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959 1796
y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959 2022
y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959 2248
y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959 2474
y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959 2699
y(48)p eop
%%Page: 31 33
bop 75 -100 a Fo(4.8.)34 b(A)l(TOMICITY)16 b(AND)f(MEMOR)l(Y)h(COHERENCE)f
(OPERA)l(TIONS)401 b Fu(31)75 49 y Fq(if)24 b(\(myrank)e(==)i(0\))48
b({)23 b(/*)h(producer)f(*/)147 162 y(MPI_RMA_INIT\(NULL,)e(0,)j(0,)f
(MPI_WORLD,)g(comm\))147 275 y(while\(1\))f({)218 332 y(object.version++;)218
388 y(produce\(object.data\);)218 444 y(MPI_Fence\(comm,)g(1\);)218
501 y(MPI_Put\(&object.version,)f(1,)i(MPI_INT,)g(1,)481 557
y(100*sizeof\(double\),)e(1,)j(MPI_INT,)e(0,)i(comm\);)218
614 y(MPI_Fence\(comm,)e(1\);)218 670 y(MPI_Put\(&object.data,)f(100,)i
(MPI_FLOAT,)g(1,)h(0,)839 727 y(100,)f(MPI_FLOAT,)f(0,)i(comm\);)218
783 y(})75 840 y(elseif)f(\(myrank)g(==)h(1\))47 b({)24 b(/*)f(consumer)g(*/)
147 953 y(MPI_RMA_INIT\(&object,)e(sizeof\(object\),)g(1,)j(MPI_WORLD,)e
(comm\);)147 1065 y(while\(1\))g({)218 1122 y(do)i({)290 1178
y(MPI_Window_in\(comm\);)290 1235 y(current)f(=)g(object.version;)290
1291 y(for)g(\(i=0;)g(i)h(<)g(100;)f(i++\))g(localcopy[i])g(=)g
(object.data[i];)290 1348 y(MPI_Window_in\(comm\);)290 1404
y(})218 1461 y(while)g(\(current)g(!=)h(object.version\);)218
1517 y(consume\(localcopy\);)218 1574 y(})75 1667 y Fu(The)18
b(pro)q(ducer)h(rep)q(eatedly)h(up)q(dates)e(in)i(the)e(consumer)g(memory)g
(the)g(v)o(ersion)h(n)o(um)o(b)q(er,)g(next)f(the)75 1724 y(ob)s(ject)e
(data.)24 b(Successiv)o(e)19 b(up)q(dates)e(are)f(serialized)j(b)o(y)e(calls)
h(to)e(fence.)26 b(The)17 b(consumer)g(c)o(hec)o(ks)g(the)75
1780 y(v)o(ersion)22 b(n)o(um)o(b)q(er,)h(next)e(bu\013ers)h(the)f(ob)s(ject)
g(data,)h(next)g(rec)o(hec)o(k)g(the)f(v)o(ersion)h(n)o(um)o(b)q(er.)39
b(If)22 b(the)75 1837 y(v)o(ersion)15 b(n)o(um)o(b)q(er)h(has)f(not)g(c)o
(hanged,)g(then)g(all)i(ob)s(ject)d(data)h(comes)g(from)f(the)i(same)e(v)o
(ersion.)166 1893 y(The)21 b(co)q(de)h(is)g(correct,)f(but)h(starv)m(ation)e
(is)i(p)q(ossible,)i(if)e(consumer)f(rep)q(eatedly)h(up)q(dates)g(the)75
1950 y(v)o(ersion)14 b(n)o(um)o(b)q(er)f(b)q(efore)h(the)f(consumer)h
(bu\013ered)f(a)g(new)h(v)o(ersion.)19 b(Suc)o(h)14 b(co)q(de)g(should)h(b)q
(e)f(used)f(only)75 2006 y(when)k(new)g(v)o(ersions)f(are)h(pro)q(duced)g(at)
f(m)o(uc)o(h)h(greater)e(in)o(terv)m(als)j(than)e(the)g(time)h(needed)h(to)e
(bu\013er)75 2063 y(a)f(v)o(ersion.)75 2183 y Fn(Registers)h(and)h(Com)o
(piler)c(Optimizations)44 b Fu(Another)15 b(coherence)h(problem)g(exists)g(b)
q(et)o(w)o(een)g(v)m(ariables)75 2239 y(k)o(ept)21 b(in)g(registers)g(and)g
(the)f(memory)h(v)m(alue)h(of)e(these)h(v)m(ariables.)38 b(An)21
b(RMA)g(call)g(ma)o(y)f(access)h(a)75 2296 y(v)m(ariable)c(in)g(memory)f
(\(or)f(cac)o(he\),)h(while)h(the)f(up-to-date)g(v)m(alue)i(of)d(this)i(v)m
(ariable)g(is)g(in)g(register.)22 b(A)75 2352 y(get)12 b(will)j(not)d(return)
g(the)h(latest)g(v)m(ariable)h(v)m(alue,)g(and)f(a)f(put)h(ma)o(y)f(b)q(e)h
(o)o(v)o(erwritten)f(when)h(the)g(register)75 2408 y(is)j(stored)e(bac)o(k)i
(in)g(memory)l(.)166 2465 y(MPI)k(implemen)o(tations)i(will)f(a)o(v)o(oid)f
(this)h(problem)g(for)e(standard)h(conforming)h(C)f(programs.)75
2521 y(High)13 b(qualit)o(y)h(MPI)f(F)l(ortran)e(implemen)o(tations)j(will)h
(a)o(v)o(oid)d(this)i(problem)f(for)g(standard)f(conforming)75
2578 y(F)l(ortran)i(programs)h(as)g(w)o(ell,)h(without)g(disabling)h
(compiler)g(optimizations.)22 b(Ho)o(w)o(ev)o(er,)14 b(MPI)h(imple-)75
2634 y(men)o(tations)h(are)h(not)f(required)i(to)e(a)o(v)o(oid)h(altogether)f
(the)h(register)f(coherence)i(problem,)g(since)g(this)75 2691
y(requires)d(compiler)h(supp)q(ort)e(that)g(is)h(not)f(curren)o(tly)h(a)o(v)m
(ailable)h(on)f(most)e(F)l(ortran)g(en)o(vironmen)o(ts.)20
b(In)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32
272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40
554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40
836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40
1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345
y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40
1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853
y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40
2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361
y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40
2643 y(47)-40 2699 y(48)p eop
%%Page: 32 34
bop 75 -100 a Fu(32)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fu(order)f(to)f(a)o(v)o(oid)g(register)h(coherence)h
(problems,)f(users)g(should)h(restrict)e(their)h(use)h(of)e(RMA)h(windo)o(ws)
75 106 y(to)h(v)m(ariables)h(stored)f(in)h Fl(COMMON)f Fu(blo)q(c)o(ks,)h(or)
e(to)h(v)m(ariables)i(that)d(w)o(ere)h(declared)i Fl(V)o(OLA)m(TILE)d
Fu(\(while)75 162 y Fl(V)o(OLA)m(TILE)e Fu(is)h(no)g(a)f(standard)h(F)l
(ortran)f(declaration,)h(it)h(is)f(supp)q(orted)g(b)o(y)g(man)o(y)g(F)l
(ortran)e(compilers.\))189 268 y Fk(R)n(ationale.)70 b Fu(Supp)q(ose)21
b(that)e(remote)h(accesses)g(are)g(executed)h(b)o(y)g(a)e(separate)h(soft)o
(w)o(are)e(or)189 325 y(hardw)o(are)d(agen)o(t,)g(concurren)o(tly)i(with)f
(main)h(pro)q(cess)f(execution.)24 b(A)16 b(remote)f(get)h(will)i(access)189
381 y(the)f(memory)g(or)f(cac)o(he)i(v)m(alue)g(of)f(a)g(v)m(ariable,)i
(while)g(the)e(latest)g(v)m(alue)i(of)d(this)i(v)m(ariable)h(ma)o(y)189
438 y(b)q(e)g(in)g(a)f(register.)29 b(Similarly)l(,)21 b(a)d(remote)f(put)i
(will)h(up)q(date)f(the)f(memory)g(or)g(cac)o(he)g(lo)q(cation)189
494 y(of)g(this)i(v)m(ariable,)h(while)g(the)e(v)m(ariable)h(is)g(held)g(in)g
(register.)32 b(The)19 b(memory)g(up)q(date)h(will)g(b)q(e)189
551 y(o)o(v)o(erwritten)14 b(when)i(the)f(register)g(spills)i(bac)o(k)e(to)g
(memory)l(.)189 626 y(The)g(same)h(problem)g(will)h(o)q(ccur)f(ev)o(en)g(if)g
(remote)f(accesses)g(are)g(executed)i(b)o(y)e(a)h(callbac)o(k)g(pro-)189
682 y(cedure)d(on)g(the)g(lo)q(cal)h(pro)q(cess.)20 b(The)13
b(v)m(ariable)h(that)e(is)i(accessed)f(remotely)g(do)q(es)g(not)g(app)q(ear)g
(in)189 739 y(the)k(argumen)o(t)f(list)h(of)g(the)f(callbac)o(k)i(pro)q
(cedure.)26 b(Therefore,)17 b(an)f(optimizing)j(compiler)f(ma)o(y)189
795 y(infer)f(that)e(the)i(pro)q(cedure)g(do)q(es)f(not)g(access)h(this)f(v)m
(ariable.)25 b(If)16 b(the)h(v)m(ariable)g(w)o(as)f(in)h(register)189
852 y(b)q(efore)f(the)f(pro)q(cedure)i(call,)g(the)e(compiler)i(ma)o(y)e(lea)
o(v)o(e)h(it)g(there.)21 b(The)16 b(pro)q(cedure)h(will)g(either)189
908 y(lea)o(v)o(e)h(this)i(register)e(unp)q(erturb)q(ed,)j(or)d(will)i(sa)o
(v)o(e)e(it)h(on)g(the)f(stac)o(k)g(up)q(on)i(en)o(try)e(and)h(restore)189
965 y(it)f(up)q(on)g(exit.)28 b(The)18 b(address)g(of)f(this)h(v)m(ariable)h
(is)g(not)e(kno)o(wn)h(to)f(the)g(pro)q(cedure.)29 b(In)18
b(either)189 1021 y(cases,)d(a)g(read)g(to)g(the)h(lo)q(cation)g(of)f(this)h
(v)m(ariable)h(in)g(memory)e(on)g(b)q(ehalf)i(of)e(a)g(get)g(will)i(fail)g
(to)189 1077 y(return)d(the)g(latest)h(v)m(alue)g(of)f(this)h(v)m(ariable,)h
(and)e(a)g(write)h(to)f(this)h(lo)q(cation)g(on)f(b)q(ehalf)i(of)e(a)g(put)
189 1134 y(will)i(b)q(e)g(o)o(v)o(erwritten)f(when)g(the)h(pro)q(cedure)g
(returns.)189 1209 y(A)i(lo)q(cal)h(access)f(to)f(suc)o(h)h(v)m(ariable)h(is)
g(alw)o(a)o(ys)e(separated)g(from)h(a)f(con\015icting)i(remote)f(access)189
1266 y(b)o(y)13 b(a)g(lo)q(cal)h(MPI)f(call,)i(suc)o(h)e(as)g
Fn(MPI)p 827 1266 14 2 v 16 w(WINDO)o(W)p 1039 1266 V 16 w(IN)g
Fu(or)f Fn(MPI)p 1248 1266 V 16 w(WINDO)o(W)p 1460 1266 V 16
w(OUT)p Fu(.)i(Therefore,)f(the)189 1322 y(coherence)j(problem)h(will)h(not)d
(o)q(ccur)h(in)h(a)e(standard)h(conforming)g(C)f(program.)21
b(If)16 b(a)g(v)m(ariable)189 1378 y(o)q(ccurs)22 b(in)g(a)g(comm)o
(unication)h(windo)o(w,)g(then)f(this)g(v)m(ariable)i(has)d(b)q(een)i
(dereferenced:)35 b(its)189 1435 y(address)15 b(has)h(b)q(een)h(used)f(to)f
(create)h(the)f(windo)o(w,)h(or)f(the)h(v)m(ariable)h(has)f(b)q(een)g
(heap-allo)q(cated)189 1491 y(at)21 b(a)h(kno)o(wn)f(lo)q(cation.)42
b(This)22 b(v)m(ariable)i(ma)o(y)d(b)q(e)i(accessed)f(b)o(y)g(a)g(pro)q
(cedure,)i(ev)o(en)e(if)h(the)189 1548 y(v)m(ariable)17 b(is)f(not)g(on)g
(the)f(argumen)o(t)h(list)g(of)g(the)g(pro)q(cedure.)22 b(If)16
b(the)g(MPI)g(library)h(is)f(compiled)189 1604 y(separately)l(,)c(and)f(at)f
(least)i(one)f(external)g(p)q(oin)o(ter)h(is)g(declared,)h(then)e(an)g
(optimizing)i(C)e(compiler)189 1661 y(m)o(ust)16 b(assume)g(that)g(MPI)h
(functions)g(ma)o(y)f(access)h(the)g(v)m(ariable.)25 b(Therefore,)17
b(the)f(v)m(ariable)i(is)189 1717 y(written)d(bac)o(k)g(to)f(memory)h(b)q
(efore)h(the)f(MPI)g(function)h(is)g(called.)189 1792 y(The)11
b(coherence)h(problem)g(ma)o(y)e(o)q(ccur)i(in)g(F)l(ortran)e(since)i
(explicit)h(aliasing)f(via)g(p)q(oin)o(ters)f(cannot)189 1849
y(o)q(ccur.)33 b(Ho)o(w)o(ev)o(er,)19 b(if)h(a)f(v)m(ariable)h(is)g(in)h(a)e
Fl(COMMON)g Fu(blo)q(c)o(k,)i(then)e(an)h(optimizing)h(compiler)189
1905 y(m)o(ust)12 b(assume)h(that)f(this)i(v)m(ariable)g(ma)o(y)e(b)q(e)i
(accessed)g(b)o(y)f(MPI)g(calls,)h(and)f(the)g(v)m(ariable)i(will)f(b)q(e)189
1962 y(written)f(bac)o(k)g(to)f(memory)h(b)q(efore)g(the)g(MPI)g(function)h
(is)g(called.)21 b(A)13 b(declaration)h(of)e Fl(V)o(OLA)m(TILE)189
2018 y Fu(will)22 b(force)e(the)h(v)m(ariable)h(to)e(b)q(e)i(written)e(bac)o
(k)h(to)f(memory)g(after)g(eac)o(h)h(up)q(date,)h(and)f(read)189
2075 y(from)e(memory)g(b)q(efore)h(eac)o(h)g(use,)g(so)g(that)f(the)h
(coherence)g(problem)h(disapp)q(ears.)34 b(\()p Fk(End)20 b(of)189
2131 y(r)n(ationale.)p Fu(\))-170 2215 y Fh(?)p Fu(\()p Fp(June)p
Fu(\))75 2370 y Fr(4.9)59 b(F)n(urther)20 b(Examples)75 2471
y Fp(Example)e(4.6)k Fu(W)l(e)17 b(assume)g(that)e(t)o(w)o(o)h(datat)o(yp)q
(e)g(transfer)g(functions)h(are)f(built,)i(using)f(the)g(facili-)75
2527 y(ties)h(de\014ned)h(for)f(datat)o(y)o(e)e(transfer)h(in)i(Chapter)e
Fp(??)p Fu(.)27 b(The)18 b(function)h Fn(MPI)p 1416 2527 V
16 w(TYPE)p 1549 2527 V 17 w(SEND\(datat)o(yp)q(e,)75 2584
y(rank,)j(tag,)h(comm)m(\))18 b Fu(send)k(a)e(description)j(of)e(a)g(datat)o
(yp)q(e)g(to)f(another)h(pro)q(cess)g(and)h(the)f(function)75
2640 y Fn(MPI)p 160 2640 V 16 w(TYPE)p 293 2640 V 17 w(RECV\(t)o(yp)q(e,)16
b(rank,)g(tag,)f(comm)m(\))d Fu(receiv)o(es)17 b(this)f(description)h(and)e
(create)h(an)f(equiv)m(alen)o(t)1967 46 y Fj(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 33 35
bop 75 -100 a Fo(4.9.)34 b(FUR)l(THER)16 b(EXAMPLES)1153 b
Fu(33)75 49 y(datat)o(yp)q(e)14 b(on)h(the)g(destination)g(pro)q(cess)g
(\(the)g(same)f(datat)o(yp)q(e)g(that)g(w)o(ould)h(b)q(e)h(created)e(if)h
(the)g(desti-)75 106 y(nation)g(pro)q(cess)f(had)h(executed)h(the)e(same)g
(sequence)i(of)e(datat)o(yp)q(e)g(construction)h(calls)h(as)e(the)g(origin)75
162 y(pro)q(cess\).)166 219 y(In)19 b(the)h(follo)o(wing)f(co)q(de)h(pro)q
(cess)f(one)g(gets)g(a)f(structure)h(from)f(pro)q(cess)h(zero.)32
b(The)19 b(co)q(de)g(will)75 275 y(w)o(ork)14 b(correctly)i(in)g(a)f
(heterogeneous)g(en)o(vironmen)o(t.)75 381 y Fq(main\(\))23
b({)75 438 y(int)g(rank;)75 494 y(MPI_Comm)g(getcomm;)147 607
y(struct)g({)147 664 y(char)g(a;)147 720 y(int)g(b[10];)147
777 y(})g(str;)75 889 y(MPI_Comm_rank\(MPI_COMM_WOR)o(LD,)e(rank\);)75
946 y(MPI_RMA_Init\(&str,)g(sizeof\(str\),)i(1,)g(MPI_COMM_WORLD,)f
(getcomm\);)75 1059 y(/*)i(construct)e(local)h(datatype)g(for)h(str)f(*/)75
1115 y(int)g(disps[2];)75 1172 y(disps[0])g(=)g(0;)75 1228
y(disps[1])g(=)g(&str.b)h(-)f(&str;)75 1285 y(int)g(blens[2])g(=)h({1,)f
(10};)75 1341 y(MPI_Datatype)f(types[2])h(=)h({MPI_CHAR,)e(MPI_INT};)75
1398 y(MPI_Datatype)g(strtype;)75 1510 y(MPI_Type_struc\(2,)g(blens,)h
(disps,)g(types,)g(strtype\);)75 1567 y(MPI_Type_commit\(&strtype\);)75
1680 y(if\(rank)g(==)g(0\))h({)147 1736 y(int)f(i;)147 1849
y(/*)g(initialize)g(str)g(*/)147 1906 y(str.a)g(=)g('z';)147
1962 y(for)g(\(i=0;)g(i)h(<)g(10;)f(i++\))g(str.b[i])g(=)h(i;)147
2075 y(/*)f(send)g(datatype)g(for)h(str)f(*/)147 2131 y
(MPI_Type_send\(strtype,)d(1,)k(0,)g(getcomm\);)147 2188 y(})75
2244 y(elseif\(rank)e(==)i(1\))g({)147 2301 y(/*)f(get)h(remote)f(datatype)f
(for)i(str)f(*/)147 2357 y(MPI_Datatype)f(remstrtype;)147 2414
y(MPI_Recv\(&remstrtype,)f(0,)i(0,)h(getcomm\);)147 2527 y(/*)f(get)h(remote)
f(structure)f(*/)147 2583 y(MPI_Get\(str,)g(1,)h(srttype,)g(0,)h(0,)f(1,)h
(remstrtype,)e(1,)i(getcomm\);)147 2640 y(})75 2696 y(})-32
46 y Fj(1)-32 103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32
328 y(6)-32 385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40
611 y(11)-40 667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40
893 y(16)-40 949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119
y(20)-40 1175 y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40
1401 y(25)-40 1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627
y(29)-40 1683 y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40
1909 y(34)-40 1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135
y(38)-40 2191 y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40
2417 y(43)-40 2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643
y(47)-40 2699 y(48)p eop
%%Page: 34 36
bop 75 -100 a Fu(34)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 49 y Fp(Example)j(4.7)k Fu(W)l(e)c(outline)h(b)q(elo)o(w)g(a)e
(design)i(for)e Fk(glob)n(al)h(p)n(ointers)p Fu(.)27 b(Global)18
b(p)q(oin)o(ters)h(are)e(sp)q(eci\014c)75 106 y(to)g(an)g(RMA)g(comm)o
(unicator,)g(and)h(consist)f(of)g(a)g(\(rank,)f(displacemen)o(t\))j(pair.)26
b(The)18 b(example)g(uses)75 162 y(a)f(windo)o(w,)h(with)g(displacemen)o(t)h
(one,)f(that)f(co)o(v)o(ers)g(the)h(heap)g(area)f(used)h(allo)q(cate)g
(globally)h(link)o(ed)75 219 y(structures.)75 381 y Fq(int)k(i;)75
551 y(/*)h(global)f(pointer)g(*/)75 607 y(typedef)g(struct)g({)123
664 y(int)g(rank;)123 720 y(int)g(disp;)75 777 y(})h(GPTR;)75
889 y(/*)g(local)f(to)g(global)g(pointer)g(conversion)g(\(error)g(checking)g
(omitted\))f(*/)75 946 y(void)h(local2global\(void)f(*ptr,)h(MPI_Comm)g
(comm,)g(GPTR)g(*gptr\))75 1002 y({)75 1059 y(int)g(flag,)h(disp_unit;)75
1115 y(MPI_Aint)f(addr,)g(base;)75 1172 y(void)g(*attr;)75
1285 y(MPI_Comm_rank\(comm,)e(&\(gptr->rank\)\);)75 1341 y
(MPI_Attr_get\(comm,)g(MPI_WINDOW_BASE,)h(&attr,)h(&flag\);)75
1398 y(base)g(=)h(\(MPI_Aint\)attr;)75 1454 y(MPI_Attr_get\(comm,)d
(MPI_WINDOW_DISP_UNIT,)g(&attr,)i(&flag\);)75 1510 y(disp_unit)g(=)g
(\(int\)attr;)75 1567 y(gptr->disp)f(=)i(\(\(MPI_Aint\)ptr)e(-)i
(base\)/disp_unit;)75 1623 y(return\(\);)75 1680 y(})75 1849
y(/*)g(MPI)f(Datatype)g(for)g(global)g(pointer)g(*/)75 1906
y(MPI_Datatype)f(GPTR_TYPE;)75 1962 y(MPI_Type_contiguous\(2,)f(MPI_INT,)i
(&GPTR_TYPE\);)75 2019 y(MPI_Type_commit\(&GPTR_TYPE)o(\);)75
2188 y(/*)h(nodes)f(of)g(local)h(list)f(*/)75 2244 y(typedef)g(lnode)g
(struct)g({)75 2301 y(double)g(content;)75 2357 y(struct)g(lnode)g(*next;)75
2414 y(})h(LNODE;)75 2583 y(/*)g(node)f(of)g(global)h(list;)f(the)g(list)g
(can)h(be)g(distributed)e(among)147 2640 y(several)g(processes)h(*/)1967
46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967 272 y(5)1967
328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959 554 y(10)1959
611 y(11)1959 667 y(12)1959 724 y(13)1959 780 y(14)1959 836
y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959 1062 y(19)1959
1119 y(20)1959 1175 y(21)1959 1232 y(22)1959 1288 y(23)1959
1345 y(24)1959 1401 y(25)1959 1457 y(26)1959 1514 y(27)1959
1570 y(28)1959 1627 y(29)1959 1683 y(30)1959 1740 y(31)1959
1796 y(32)1959 1853 y(33)1959 1909 y(34)1959 1966 y(35)1959
2022 y(36)1959 2078 y(37)1959 2135 y(38)1959 2191 y(39)1959
2248 y(40)1959 2304 y(41)1959 2361 y(42)1959 2417 y(43)1959
2474 y(44)1959 2530 y(45)1959 2587 y(46)1959 2643 y(47)1959
2699 y(48)p eop
%%Page: 35 37
bop 75 -100 a Fo(4.9.)29 b(FUR)l(THER)16 b(EXAMPLES)1158 b
Fu(35)75 49 y Fq(typedef)23 b(struct)g({)75 106 y(GPTR)g(next;)75
162 y(double)g(content;)75 219 y(})h(GNODE;)75 388 y(/*)g(MPI)f(Datatype)g
(for)g(global)g(node)h(*/)75 444 y(MPI_Datatype)e(GNODE_TYPE;)75
501 y(blens)h(=)h({1,)f(1};)75 557 y(types)g(=)h({GPRT_TYPE,)e(MPI_DOUBLE};)
75 614 y(MPI_Type_contiguous_struct)o(\(2,)f(blens,)i(types,)g
(&GNODE_TYPE\);)75 670 y(MPI_Type_commit\(&GNODE_TYP)o(E\);)75
840 y(/*)h(insertion)e(function.)47 b(Inserts)23 b(node)g(in)h(global)f(list)
g(*/)75 896 y(void)g(insert\()g(double)g(content,)g(GPTR)g(*gptr,)g(MPI_Comm)
g(comm\);)75 953 y({)75 1009 y(GNODE)g(*gnode;)75 1065 y(GPTR)g(me;)75
1178 y(gnode)g(=)h(\(GNODE*\)malloc\(sizeof\(GNO)o(DE\)\);)75
1235 y(gnode->content)e(=)i(content;)75 1291 y(/*)g(compute)e(global)i
(pointer)e(to)i(inserted)f(node)g(*/)75 1348 y(local2global\(gnode,)e(comm,)i
(&me\);)75 1461 y(/*)h(insert)f(*/)75 1517 y(MPI_Rmw\(me,)f
(&\(gnode->next\),)g(2,)i(MPI_INT,)266 1574 y(gptr->rank,)e(gptr->disp,)h
(MPI_SWAP,)f(0,)i(comm\);)75 1630 y(return\(\);)75 1686 y(})75
1856 y(/*)g(global)f(to)g(local)g(list)h(copying)f(function.)46
b(Creates)23 b(local)g(copy)h(of)147 1912 y(global,)e(distributed)h(list)g
(\(error)g(checking)g(and)g(locking)g(omitted\))147 1969 y(an)g(undefined)g
(rank)g(is)h(used)f(as)h(list)f(terminator)f(*/)75 2025 y(void)h
(global2local_list_copy\(GPTR)d(*ghead,)j(MPI_Comm)g(comm,)g(LNODE)h
(**head\);)75 2082 y({)75 2138 y(GNODE)f(gnode;)75 2195 y(LNODE)g(*tail;)75
2307 y(if)h(\(ghead->rank)e(==)h(MPI_UNDEFINED\))f({)123 2364
y(*head)h(=)h(NULL;)123 2420 y(return\(\);)123 2477 y(})75
2533 y(else)f({)147 2590 y(*head)g(=)g(\(LNODE*\)malloc\(sizeof\(LNODE\)\);)
147 2646 y(MPI_Get\(&gnode,)e(1,)j(GNODE_TYPE,)e(ghead->rank,)h(ghead->disp,)
337 2703 y(1,)h(GNODE_TYPE,)e(0,)i(comm\);)-32 46 y Fj(1)-32
103 y(2)-32 159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32
385 y(7)-32 441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40
667 y(12)-40 724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40
949 y(17)-40 1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175
y(21)-40 1232 y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40
1457 y(26)-40 1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683
y(30)-40 1740 y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40
1966 y(35)-40 2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191
y(39)-40 2248 y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40
2474 y(44)-40 2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699
y(48)p eop
%%Page: 36 38
bop 75 -100 a Fu(36)728 b Fo(CHAPTER)15 b(4.)30 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)147 49 y Fq(\(*head\)->content)21 b(=)j(gnode.content;)147
106 y(tail)f(=)h(*head;)147 162 y(while\(gnode->next.rank)c(!=)k
(MPI_UNDEFINED\))e({)218 219 y(tail->next)h(=)g
(\(LNODE*\)malloc\(sizeof\(LNODE\)\))o(;)218 275 y(tail)g(=)h(tail->next;)218
332 y(MPI_Get\(&gnode,)e(1,)i(GNODE_TYPE,)e(gnode->next.rank,)f
(gnode->next.disp,)409 388 y(1,)j(GNODE_TYPE,)e(0,)i(comm\);)218
444 y(tail->content)e(=)i(gnode.content;)218 501 y(})147 557
y(tail->next)e(=)i(NULL;)147 614 y(})75 670 y(return\(\);)75
727 y(})166 909 y Fi(Discussion:)61 b Fm(As)21 b(the)g(last)g(example)e(sho)o
(ws,)j(the)g(curren)o(t)g(design)f(allo)o(ws)e(for)h(global)f(structures)75
965 y(\(structures)g(using)d(global)f(p)q(oin)o(ters\))j(to)e(b)q(e)h
(accessed)i(lo)q(cally)c(or)i(remotely;)f(it)g(allo)o(ws)g(for)g(lo)q(cal)f
(structures)75 1022 y(\(structures)f(with)c(lo)q(cal)h(p)q(oin)o(ters\))g(to)
g(b)q(e)h(accessed)h(lo)q(cally)m(,)c(and)i(con)o(v)o(erted)h(to)f(global)f
(structures.)19 b(It)11 b(do)q(es)h(not)75 1078 y(allo)o(w)d(for)h(lo)q(cal)g
(structures)j(to)d(b)q(e)h(accessed)i(remotely)m(.)j(The)11
b(last)f(is)g(reasonably)h(easy)g(to)f(do)h(in)f(a)g(homogeneous)75
1134 y(en)o(vironmen)o(t.)24 b(An)17 b(MPI)f(extension)h(w)o(ould)f(b)q(e)h
(needed)h(to)e(supp)q(ort)h(the)g(last:)22 b(e.g.,)16 b(allo)o(wing)e(v)n
(alues)i(to)g(b)q(e)75 1191 y(sen)o(t)f(with)f(datat)o(yp)q(e)g
Fl(MPI)p 508 1191 13 2 v 14 w(POINTER)f Fm(and)h(receiv)o(ed)h(with)f(t)o(yp)
q(e)g Fl(MPI)p 1217 1191 V 15 w(INT)p Fm(,)f(on)h(a)g(windo)o(w)f(comm)o
(unicator;)e(if)75 1247 y(the)j(p)q(oin)o(ter)g(w)o(as)f(p)q(oin)o(ting)f(to)
h(a)h(lo)q(cation)e(inside)h(the)h(windo)o(w,)f(then)h(it)f(is)g(con)o(v)o
(erted,)h(at)f(the)h(receiving)g(side,)75 1304 y(in)o(to)f(a)h(displacemen)o
(t)f(within)g(this)h(windo)o(w.)75 1486 y Fp(Example)k(4.8)k
Fu(This)15 b(example)f(implemen)o(ts)h(a)e(master-sla)o(v)o(e)g(framew)o
(ork,)f(where)i(sla)o(v)o(es)g(pic)o(k)g(w)o(ork)75 1542 y(from)e(a)h(w)o
(ork)f(pile)j(and)e(execute)h(it.)19 b(A)14 b(form)e(of)h(guided)h(self-sc)o
(heduling)i(is)d(used)h(for)f(load)g(balancing,)75 1599 y(assuming)h(that)e
(task)h(execution)h(time)g(is)g(not)f(constan)o(t.)19 b(The)13
b(main)h(sla)o(v)o(e)f(routine)h(is)g(sho)o(wn)f(b)q(ello)o(w.)75
1687 y Fq(...)75 1744 y(void)23 b(function)g(slave\()147 1800
y(MPI_Comm)f(pile,)47 b(/*)24 b(communicator)e(for)i(window)f(containing)1053
1857 y(shared)g(work)h(pile)214 b(*/)147 1913 y(MPI_Comm)22
b(counter,)h(/*)h(communicator)e(for)h(window)g(containing)910
1970 y(shared)g(work)h(pile)f(counter)166 b(*/)147 2026 y(int)47
b(master,)71 b(/*)23 b(rank)g(of)h(master)739 b(*/)242 2083
y(numitems,)47 b(/*)23 b(number)g(of)h(work)f(items)g(in)h(pile)405
b(*/)242 2139 y(itemsize,)47 b(/*)23 b(size)g(of)h(work)f(item)668
b(*/)242 2195 y(minchunk,)47 b(/*)23 b(min)h(number)f(of)g(work)h(items)f
(picked)g(by)g(process)71 b(*/)242 2252 y(maxchunk,)47 b(/*)23
b(max)h(number)f(of)g(work)h(items)f(picked)g(by)g(process)71
b(*/)147 2308 y(float)23 b(bfactor,)g(/*)g(fudge)g(factor)g(for)h(load)f
(balancing)g(\(greater)g(than)552 2365 y(1,)h(and)f(larger)g(if)h(tasks)f
(exhibit)g(more)g(variance\))g(*/)147 2421 y(void)g(\(*task\))g(\()h(char)f
(*\))g(/*)h(pointer)f(to)h(work)f(function)309 b(*/)147 2478
y(\))75 2534 y({)75 2591 y(int)23 b(numproc,)g(mychunk,)g(diff,)g(mystart,)g
(i;)75 2647 y(char*)g(mypile;)1967 46 y Fj(1)1967 103 y(2)1967
159 y(3)1967 215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967
441 y(8)1967 498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959
724 y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949
y(17)1959 1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175
y(21)1959 1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401
y(25)1959 1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627
y(29)1959 1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853
y(33)1959 1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078
y(37)1959 2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304
y(41)1959 2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530
y(45)1959 2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 37 39
bop 75 -100 a Fo(4.10.)34 b(OPTIMIZA)l(TION)16 b(HINTS)1119
b Fu(37)75 49 y Fq(MPI_Comm_size\(comm,)21 b(numproc\);)75
106 y(mypile)i(=)h(\(char)f(*\)malloc\(numitems*itemsize)o(/\(numpr)o
(oc*bfac)o(tor\)\);)75 219 y(while\(1\))g({)147 275 y(mychunk)f(=)i
(numitem/\(numproc*bfactor\);)147 332 y(if)f(\(mychunk)g(>)h(maxchunk\))e
(mychunk)h(=)h(maxchunk;)147 388 y(if)f(\(mychunk)g(<)h(minchunk\))e(mychunk)
h(=)h(minchunk;)147 444 y(diff)f(=)h(-mychunk;)147 501 y(MPI_Rmw\(&diff,)e
(&mystart,)g(1,)i(MPI_INT,)f(master,)f(0,)i(MPI_SUM\);)147
557 y(if)f(\(\(mystart)g(+)h(mychunk\))e(<)i(0\))g(break;)147
614 y(if)f(\(mystart)g(<)h(0\))f({)h(mychunk)f(+=)g(mystart;)g(mystart)g(=)h
(0;})147 670 y(MPI_Get\(mypile,)d(mychunk*itemsize,)h(MPI_CHAR,)h(master,)337
727 y(mystart*itemsize,)f(mychunk*itemsize,)g(MPI_CHAR,)g(pile\);)147
783 y(for)h(\(i=0;)g(i<mychunk;)g(i++\))g(task\(mypile+i*itemsize\);)147
840 y(})75 983 y Fr(4.10)60 b(Optimization)18 b(Hints)75 1167
y Fi(Discussion:)d Fm(This)f(is)g(the)g(text)h(of)e(a)h(prop)q(osal)f(b)o(y)h
(Rolf)-170 1235 y Fh(>)p Fu(\()p Fp(June)p Fu(\))75 1382 y
Fr(4.11)60 b(Optimization)18 b(Hints)75 1483 y Fu(The)i(application)h(can)f
(giv)o(e)g(some)f(information)h(on)f(ho)o(w)g(it)h(will)i(use)d(one-sided)j
(comm)o(unication.)75 1540 y(Based)f(on)g(these)g(hin)o(ts)g(the)g(MPI)f
(library)i(can)f(p)q(ossibly)h(optimize)g(some)e(RMA)h(functions.)38
b(F)l(or)75 1596 y(this)21 b(purp)q(ose)g(the)g(application)h(can)e(store)g
(some)g(attribute)h(v)m(alues)g(with)g Fn(MPI)p 1525 1596 14
2 v 16 w(A)l(TTR)p 1658 1596 V 17 w(PUT)g Fu(on)f(a)75 1653
y(comm)o(unicator)10 b(used)h(for)e(RMA)h(comm)o(unications.)19
b(T)l(o)10 b(store)g(an)o(y)f(com)o(binations)i(of)f(attribute)g(v)m(alues)75
1709 y(for)k(eac)o(h)i(k)o(ey)l(,)e(the)h(v)m(alues)i(can)e(b)q(e)g(added.)21
b(The)15 b(giv)o(en)h(hin)o(ts)f(are)g(v)m(alid)i(un)o(til)f(they)f(are)g(mo)
q(di\014ed)h(b)o(y)75 1766 y(a)g(later)g Fn(MPI)p 306 1766
V 16 w(A)l(TTR)p 439 1766 V 17 w(PUT)g Fu(or)g(canceled)i(b)o(y)e
Fn(MPI)p 949 1766 V 16 w(A)l(TTR)p 1082 1766 V 17 w(DELETE)p
Fu(.)g(The)g(k)o(ey)g(and)h(attribute)f(v)m(alues)75 1822 y(are)f
(prede\014ned.)21 b(The)16 b(k)o(ey)f(v)m(alues)h(itself)g(remain)g(usable)g
(after)f Fn(MPI)p 1293 1822 V 15 w(A)l(TTR)p 1425 1822 V 17
w(DELETE)p Fu(.)166 1879 y(The)j(follo)o(wing)h(attribute)f(v)m(alues)h(sp)q
(ecify)g(whether)f(the)h(application)g(will)h(use)e(the)g(one-sided)75
1935 y(commin)o(ucation)i(in)g(the)g(same)f(manner)g(on)h(eac)o(h)f(no)q(de)h
(of)f(the)h(comm)o(unicator.)32 b(If)19 b(non)h(of)f(these)75
1991 y(v)m(alues)d(are)f(sp)q(eci\014ed)j(then)d(eac)o(h)g(no)q(de)h(can)g
(ha)o(v)o(e)e(its)i(o)o(wn)e(pro\014le)i(\(this)g(is)f(the)h(default\).)75
2072 y(Key)g(v)m(alue:)21 b Fn(MPI)p 390 2072 V 16 w(1SIDED)p
560 2072 V 15 w(A)l(TTR)-32 46 y Fj(1)-32 103 y(2)-32 159 y(3)-32
215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32 441 y(8)-32
498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40 724 y(13)-40
780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40 1006 y(18)-40
1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232 y(22)-40 1288
y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40 1514 y(27)-40
1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740 y(31)-40 1796
y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40 2022 y(36)-40
2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248 y(40)-40 2304
y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40 2530 y(45)-40
2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 38 40
bop 75 -100 a Fu(38)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)75 43 y Fu(A)o(ttribute)g(v)m(alues:)434 b(Seman)o(tics)75
117 y Fn(MPI)p 160 117 14 2 v 16 w(1SIDED)p 330 117 V 16 w(A)l(TTR)p
463 117 V 17 w(GLOBAL)193 b Fu(On)24 b(eac)o(h)f(no)q(de)h(of)f(the)g(comm)o
(unicator)g(the)h(same)f(at-)843 174 y(tributes)j(for)f(one-sided)i(comm)o
(unication)g(will)g(b)q(e)g(de-)843 230 y(\014ned.)e(Storing)16
b(and)h(deleting)h(of)f(these)f(attributes)h(with)843 287 y
Fn(MPI)p 928 287 V 16 w(A)l(TTR)p 1061 287 V 17 w(PUT)28 b
Fu(and)h Fn(MPI)p 1381 287 V 16 w(A)l(TTR)p 1514 287 V 17 w(DELETE)g
Fu(can)g(b)q(e)843 343 y(implemen)o(ted)17 b(as)d(a)h(collectiv)o(e)i
(routine.)75 400 y Fn(MPI)p 160 400 V 16 w(1SIDED)p 330 400
V 16 w(A)l(TTR)p 463 400 V 17 w(EXPOSE)195 b(MPI)p 928 400
V 16 w(EXPOSE)25 b Fu(can)f(b)q(e)h(implemen)o(ted)g(as)f(a)f(collectiv)o(e)
843 456 y(routine.)75 512 y Fn(MPI)p 160 512 V 16 w(1SIDED)p
330 512 V 16 w(A)l(TTR)p 463 512 V 17 w(DELIVER)182 b(MPI)p
928 512 V 16 w(DELIVER)22 b Fu(can)g(b)q(e)g(implemen)o(ted)i(as)e(a)f
(collectiv)o(e)843 569 y(routine)g(and)h(after)e Fn(MPI)p 1298
569 V 16 w(PUT)h Fu(the)g(used)h(bu\013er)f(is)h(not)843 625
y(mo)q(di\014ed)16 b(un)o(til)h(the)e(next)g Fn(MPI)p 1401
625 V 16 w(DELIVER)p Fu(.)166 702 y(The)i(follo)o(wing)h(attribute)f(v)m
(alues)i(sp)q(ecify)f(the)g(RMA)f(routines)h(that)e(will)j(b)q(e)f(used)g
(only)g(after)75 758 y(storing)c(this)h(v)m(alues.)21 b(If)15
b(non)f(of)g(these)h(v)m(alues)h(are)e(sp)q(eci\014ed)j(then)d(eac)o(h)h(RMA)
g(routine)g(can)f(b)q(e)h(used)75 815 y(on)g(this)h(comm)o(unicator)f(\(this)
g(is)h(the)f(default\).)75 895 y(Key)h(v)m(alue:)21 b Fn(MPI)p
390 895 V 16 w(1SIDED)p 560 895 V 15 w(OP)75 971 y Fu(A)o(ttribute)15
b(v)m(alues:)434 b(Seman)o(tics)75 1045 y Fn(MPI)p 160 1045
V 16 w(1SIDED)p 330 1045 V 16 w(OP)p 408 1045 V 16 w(PUT)331
b(MPI)p 928 1045 V 16 w(PUT)15 b Fu(ma)o(y)f(b)q(e)i(used)75
1101 y Fn(MPI)p 160 1101 V 16 w(1SIDED)p 330 1101 V 16 w(OP)p
408 1101 V 16 w(IPUT)318 b(MPI)p 928 1101 V 16 w(IPUT)15 b
Fu(ma)o(y)f(b)q(e)i(used)75 1158 y Fn(MPI)p 160 1158 V 16 w(1SIDED)p
330 1158 V 16 w(OP)p 408 1158 V 16 w(GET)334 b(MPI)p 928 1158
V 16 w(GET)15 b Fu(ma)o(y)g(b)q(e)g(used)75 1214 y Fn(MPI)p
160 1214 V 16 w(1SIDED)p 330 1214 V 16 w(OP)p 408 1214 V 16
w(IGET)321 b(MPI)p 928 1214 V 16 w(IGET)15 b Fu(ma)o(y)f(b)q(e)i(used)75
1271 y Fn(MPI)p 160 1271 V 16 w(1SIDED)p 330 1271 V 16 w(OP)p
408 1271 V 16 w(A)o(CCUMULA)l(TE)124 b(MPI)p 928 1271 V 16
w(A)o(CCUMULA)l(TE)15 b Fu(ma)o(y)g(b)q(e)h(used)75 1327 y
Fn(MPI)p 160 1327 V 16 w(1SIDED)p 330 1327 V 16 w(OP)p 408
1327 V 16 w(IA)o(CCUMULA)l(TE)111 b(MPI)p 928 1327 V 16 w(IA)o(CCUMULA)l(TE)
15 b Fu(ma)o(y)f(b)q(e)i(used)75 1384 y Fn(MPI)p 160 1384 V
16 w(1SIDED)p 330 1384 V 16 w(OP)p 408 1384 V 16 w(RMW)310
b(MPI)p 928 1384 V 16 w(RMW)15 b Fu(ma)o(y)f(b)q(e)i(used)75
1440 y Fn(MPI)p 160 1440 V 16 w(1SIDED)p 330 1440 V 16 w(OP)p
408 1440 V 16 w(PUT)p 515 1440 V 17 w(EA)o(CH)p 649 1440 V
17 w(SAME)58 b Fu(F)l(or)13 b(eac)o(h)g(target)g(no)q(de)h
Fn(MPI)p 1349 1440 V 16 w(PUT)g Fu(is)g(alw)o(a)o(ys)f(called)i(with)843
1497 y(the)j(same)g(bu\013er)g(address,)h(datat)o(yp)q(e)f(and)g(coun)o(t)h
(argu-)843 1553 y(men)o(ts.)42 b(F)l(or)22 b(di\013eren)o(t)h(target)e(no)q
(des)i(di\013eren)o(t)g(bu\013er)843 1609 y(argumen)o(ts)14
b(ma)o(y)h(b)q(e)g(used.)75 1666 y Fn(MPI)p 160 1666 V 16 w(1SIDED)p
330 1666 V 16 w(OP)p 408 1666 V 16 w(PUT)p 515 1666 V 17 w(EVERY)p
675 1666 V 18 w(SAME)31 b Fu(F)l(or)10 b(ev)o(ery)i(target)e(no)q(de)i
Fn(MPI)p 1358 1666 V 16 w(PUT)f Fu(is)h(alw)o(a)o(ys)f(called)i(with)843
1722 y(the)18 b(same)g(bu\013er)g(address,)h(datat)o(yp)q(e)f(and)g(coun)o(t)
h(argu-)843 1779 y(men)o(ts.)35 b(F)l(or)20 b(di\013eren)o(t)g(target)g(no)q
(des)h(the)f(same)g(bu\013er)843 1835 y(argumen)o(ts)14 b(ma)o(y)h(b)q(e)g
(used.)166 1912 y(The)20 b(follo)o(wing)g(attribute)f(v)m(alues)i(sp)q(ecify)
g(whic)o(h)f(RMA)g(sp)q(eci\014c)h(sync)o(hronization)g(metho)q(ds)75
1968 y(are)13 b(used.)20 b(If)14 b(non)g(of)g(these)g(v)m(alues)h(are)e(sp)q
(eci\014ed)j(then)e(eac)o(h)g(metho)q(d)g(can)g(b)q(e)g(used)g(on)g(this)g
(comm)o(u-)75 2024 y(nicator)h(\(this)g(is)h(the)f(default\).)75
2105 y(Key)h(v)m(alue:)21 b Fn(MPI)p 390 2105 V 16 w(1SIDED)p
560 2105 V 15 w(SYNC)1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967
215 y(4)1967 272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967
498 y(9)1959 554 y(10)1959 611 y(11)1959 667 y(12)1959 724
y(13)1959 780 y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959
1006 y(18)1959 1062 y(19)1959 1119 y(20)1959 1175 y(21)1959
1232 y(22)1959 1288 y(23)1959 1345 y(24)1959 1401 y(25)1959
1457 y(26)1959 1514 y(27)1959 1570 y(28)1959 1627 y(29)1959
1683 y(30)1959 1740 y(31)1959 1796 y(32)1959 1853 y(33)1959
1909 y(34)1959 1966 y(35)1959 2022 y(36)1959 2078 y(37)1959
2135 y(38)1959 2191 y(39)1959 2248 y(40)1959 2304 y(41)1959
2361 y(42)1959 2417 y(43)1959 2474 y(44)1959 2530 y(45)1959
2587 y(46)1959 2643 y(47)1959 2699 y(48)p eop
%%Page: 39 41
bop 75 -100 a Fo(4.11.)34 b(OPTIMIZA)l(TION)16 b(HINTS)1119
b Fu(39)75 43 y(A)o(ttribute)15 b(v)m(alues:)552 b(Seman)o(tics)75
117 y Fn(MPI)p 160 117 14 2 v 16 w(1SIDED)p 330 117 V 16 w(SYNC)p
462 117 V 17 w(COUNTER)273 b(MPI)p 1046 117 V 16 w(GET)p 1150
117 V 16 w(COUNTER)17 b Fu(ma)o(y)d(b)q(e)i(used.)75 174 y
Fn(MPI)p 160 174 V 16 w(1SIDED)p 330 174 V 16 w(SYNC)p 462
174 V 17 w(THRESHOLD)218 b(MPI)p 1046 174 V 16 w(SET)p 1145
174 V 16 w(COUNTER)p 1373 174 V 18 w(THRESHOLD)43 b Fu(ma)o(y)f(b)q(e)961
230 y(used.)75 287 y Fn(MPI)p 160 287 V 16 w(1SIDED)p 330 287
V 16 w(SYNC)p 462 287 V 17 w(THRESHOLD)p 746 287 V 17 w(NOPOLL)24
b(MPI)p 1046 287 V 16 w(SET)p 1145 287 V 16 w(COUNTER)p 1373
287 V 18 w(THRESHOLD)43 b Fu(ma)o(y)f(b)q(e)961 343 y(used)15
b(and)g(a)g Fn(MPI)p 1275 343 V 15 w(W)l(AIT)g Fu(with)g(its)g(request)g
(handle)961 400 y(should)h(not)f(b)q(e)h(implemen)o(ted)h(b)o(y)e(p)q
(olling.)75 456 y Fn(MPI)p 160 456 V 16 w(1SIDED)p 330 456
V 16 w(SYNC)p 462 456 V 17 w(BARRIER)298 b(MPI)p 1046 456 V
16 w(BARRIER)25 b Fu(with)f(an)g(RMA)h(comm)o(unicator)961
512 y(ma)o(y)14 b(b)q(e)i(used.)75 569 y Fn(MPI)p 160 569 V
16 w(1SIDED)p 330 569 V 16 w(SYNC)p 462 569 V 17 w(FENCE)344
b(MPI)p 1046 569 V 16 w(FENCE)15 b Fu(ma)o(y)g(b)q(e)h(used.)166
645 y(If)i(an)h(application)g(giv)o(e)g(some)f(of)g(these)g(hin)o(ts)h(then)f
(it)h(m)o(ust)e(not)h(use)h(other)e(RMA)i(metho)q(ds)75 702
y(than)c(it)h(has)f(sp)q(eci\014ed.)75 808 y Fp(Example)j(4.9)k
Fk(On)14 b(e)n(ach)g(no)n(de)g(of)h(an)f(RMA)g(c)n(ommunic)n(ator)h(the)f
(fol)r(lowing)g(applic)n(ation)h(runs.)k(F)m(or)75 864 y(e)n(ach)c(tar)n(get)
g(always)g(the)g(same)g(bu\013er)h(ar)n(guments)e(ar)n(e)h(use)n(d)g(with)h
Fn(MPI)p 1341 864 V 16 w(PUT)p Fk(.)f(The)f(synchr)n(onization)75
921 y(after)f(the)g Fn(MPI)p 337 921 V 16 w(PUT)g Fk(is)f(done)h(with)g(c)n
(ounters,)g(b)n(e)n(c)n(ause)f(then)h(the)g(wait)g(c)n(an)f(b)n(e)g(c)n
(ompute)n(d)i(lo)n(c)n(al)r(ly.)k(The)75 977 y(applic)n(ation)12
b(waits)f(on)h(the)f(thr)n(eshold)h(but)g(do)n(es)f(not)h(set)f
Fn(MPI)p 1127 977 V 16 w(1SIDED)p 1297 977 V 16 w(SYNC)p 1429
977 V 17 w(THRESHOLD)p 1713 977 V 17 w(NOPOLL)75 1034 y Fk(b)n(e)n(c)n(ause)
18 b(the)i(applic)n(ation)f(exp)n(e)n(cts)g(that)h(ther)n(e)f(is)g(never)g(a)
h(longer)e(wait)i(at)g Fn(MPI)p 1525 1034 V 16 w(W)l(AIT)f
Fk(due)h(to)g(the)75 1090 y(overlapping)c(of)h(the)f(c)n(ommunic)n(ation)g
(and)g(the)h(se)n(c)n(ond)e(c)n(omputing)h(p)n(erio)n(d.)170
1196 y Fq(int)24 b(attr_val;)170 1253 y(attr_val)f(=)h
(MPI_1SIDED_ATTR_GLOBAL)d(+)j(MPI_1SIDED_ATTR_DELIVER)o(;)170
1309 y(MPI_Attr_put\(comm,)e(MPI_1SIDED_ATTR,)g(&attr_val\);)170
1366 y(attr_val)h(=)h(MPI_1SIDED_OP_PUT)d(+)j(MPI_1SIDED_OP_PUT_EACH_SAME;)
170 1422 y(MPI_Attr_put\(comm,)e(MPI_1SIDED_OP,)g(&attr_val\);)170
1479 y(attr_val)h(=)h(MPI_1SIDED_SYNC_THRESHOLD;)170 1535 y
(MPI_Attr_put\(comm,)e(MPI_1SIDED_SYNC,)g(&attr_val\);)170
1648 y(while\(...\))h({)218 1705 y(/*)h(compute)f(some)g(data)g(for)h(other)f
(nodes)g(and)h(store)f(it)g(into)h(buff)f(*/)218 1761 y(MPI_Put\(&buff_a,)f
(...,)h(rank_a,)g(...,)g(1,)h(comm\);)218 1817 y(MPI_Put\(&buff_b,)e(...,)h
(rank_b,)g(...,)g(1,)h(comm\);)218 1874 y(...)218 1930 y(/*)g(second)f
(computing)f(period)h(\(buff_a)g(and)h(buff_b)f(are)g(not)h(modified\)*/)218
1987 y(...)218 2043 y(/*)g(wait)f(until)g(from)h(n)f(other)g(nodes)h(the)f
(data)g(is)h(there)f(*/)218 2100 y(MPI_Set_counter_threshold\(c)o(omm,)e(n,)i
(request\);)218 2156 y(MPI_Wait\(request,)f(status\);)218 2213
y(MPI_Deliver\(comm\);)170 2269 y(})166 2451 y Fi(Discussion:)34
b Fm(Are)14 b(these)i(hin)o(ts)e(useful)g(for)f(optimization?)75
2501 y(They)19 b(are)f(useful)g(for)g(sp)q(ecial)h(hardw)o(are)f(systems,)h
(e.g.)31 b(with)18 b Fl(MPI)p 1217 2501 13 2 v 14 w(1SIDED)p
1372 2501 V 14 w(OP)p 1444 2501 V 14 w(PUT)p 1542 2501 V 14
w(EA)o(CH)p 1664 2501 V 15 w(SAME)g Fm(sys-)75 2551 y(tems)e(can)h(use)h
(virtual)e(c)o(hannels,)h(with)g Fl(MPI)p 826 2551 V 14 w(1SIDED)p
981 2551 V 14 w(A)m(TTR)p 1103 2551 V 15 w(GLOBAL)f Fm(&)h
Fl(MPI)p 1417 2551 V 14 w(1SIDED)p 1572 2551 V 14 w(A)m(TTR)p
1694 2551 V 15 w(DELIVER)75 2601 y Fm(systems)c(without)g(reliable)f(in)o
(terconnect)j(can)e(implemen)o(t)d(fast)j(one-sided)g(comm)o(unication.)i
(The)e(absence)i(of)75 2651 y Fl(MPI)p 152 2651 V 14 w(1SIDED)p
307 2651 V 14 w(SYNC)p 428 2651 V 15 w(THRESHOLD)p 687 2651
V 16 w(NOPOLL)e Fm(allo)o(ws)g(that)g Fl(MPI)p 1166 2651 V
15 w(W)m(AIT)f Fm(on)i(a)f(threshold)h(request)i(handle)d(is)75
2700 y(implemen)o(ted)f(b)o(y)h(p)q(olling,)f(a)i(go)q(o)q(d)f(strategy)i(on)
e(shared)i(memory)d(systems.)-32 46 y Fj(1)-32 103 y(2)-32
159 y(3)-32 215 y(4)-32 272 y(5)-32 328 y(6)-32 385 y(7)-32
441 y(8)-32 498 y(9)-40 554 y(10)-40 611 y(11)-40 667 y(12)-40
724 y(13)-40 780 y(14)-40 836 y(15)-40 893 y(16)-40 949 y(17)-40
1006 y(18)-40 1062 y(19)-40 1119 y(20)-40 1175 y(21)-40 1232
y(22)-40 1288 y(23)-40 1345 y(24)-40 1401 y(25)-40 1457 y(26)-40
1514 y(27)-40 1570 y(28)-40 1627 y(29)-40 1683 y(30)-40 1740
y(31)-40 1796 y(32)-40 1853 y(33)-40 1909 y(34)-40 1966 y(35)-40
2022 y(36)-40 2078 y(37)-40 2135 y(38)-40 2191 y(39)-40 2248
y(40)-40 2304 y(41)-40 2361 y(42)-40 2417 y(43)-40 2474 y(44)-40
2530 y(45)-40 2587 y(46)-40 2643 y(47)-40 2699 y(48)p eop
%%Page: 40 42
bop 75 -100 a Fu(40)723 b Fo(CHAPTER)15 b(4.)35 b(ONE)16 b(SIDED)f(COMMUNICA)
l(TIONS)166 49 y Fm(Are)g(v)o(endor)f(sp)q(eci\014c)h(metho)q(ds)f(b)q
(etter?)75 99 y(Y)m(es,)g(but)f(only)g(if)g(they)h(need)h(really)d
(additional)g(functions.)18 b(If)13 b(those)i(hin)o(ts)e(are)h(enough)g(then)
g(for)f(p)q(ortabilit)o(y)75 149 y(reasons)i(a)e(standardized)i(in)o(terface)
g(is)e(b)q(etter)j(for)d(the)i(users.)166 199 y(And)f(when)g(additional)e
(hin)o(ts)i(seem)g(to)g(b)q(e)g(useful?)75 249 y(If)g(v)o(endors)h(w)o(an)o
(t)f(to)h(add)f(additional)f(hin)o(ts)h(to)g(the)h(list)f(ab)q(o)o(v)o(e)h
(then)g(this)f(should)h(b)q(e)g(p)q(ossible)f(un)o(til)g(the)h(last)75
298 y(draft)f(of)f(MPI-2)h(is)f(written.)19 b(And)14 b(also)f(in)h(MPI-2.1.)
166 348 y(SYNC)p 283 348 13 2 v 15 w(THRESHOLD)p 562 348 V
15 w(POLL)g(or)g(SYNC)p 869 348 V 16 w(THRESHOLD)p 1149 348
V 15 w(NOPOLL?)75 398 y(Is)j(it)f(b)q(etter)i(to)e(de\014ne)i
Fl(MPI)p 546 398 V 14 w(1SIDED)p 701 398 V 14 w(SYNC)p 822
398 V 15 w(THRESHOLD)p 1081 398 V 15 w(POLL)f Fm(with)f(the)h(meaning)e(that)
h(w)o(aiting)f(on)h(a)75 448 y(threshold)d(ma)o(y)d(b)q(e)j(implem)o(en)o
(ted)e(b)o(y)g(p)q(olling?)16 b(No,)c(b)q(ecause)i(without)e(an)o(y)f(hin)o
(t)h(implemen)o(tation)d(b)o(y)j(p)q(olling)75 498 y(or)f(in)o(terrupt)h
(should)f(b)q(e)h(p)q(ossible.)17 b(P)o(olling)9 b(w)o(as)i(addressed)i(b)o
(y)e(a)g(hin)o(t)g(b)q(ecause)i(its)e(e\016ciency)h(dep)q(ends)g(on)f(the)75
547 y(a)o(v)o(erage)j(w)o(aiting)e(p)q(erio)q(d.)19 b(\(And)14
b(therefore)h(there)g(is)f(no)g(hin)o(t)f(ab)q(out)h(implemen)o(tation)c
(with)k(an)g(in)o(terrupt.\))166 597 y(When)g(should)g(the)g(hin)o(ts)g(b)q
(e)h(giv)o(en?)75 647 y(A)o(t)c(an)o(y)g(time)f(or)h(only)f(direct)i(after)f
Fl(MPI)p 723 647 V 14 w(RMA)p 828 647 V 15 w(INIT)g Fm(and)g(b)q(efore)h(the)
f(\014rst)h(RMA)f(is)g(done.)17 b(There)c(is)d(no)h(reason)75
697 y(for)i(a)f(restriction.)19 b(Esp)q(ecially)13 b(if)f Fl(MPI)p
696 697 V 14 w(1SIDED)p 851 697 V 14 w(OP)p 923 697 V 14 w(PUT)p
1021 697 V 15 w(EA)o(CH)p 1144 697 V 14 w(SAME)g Fm(is)h(giv)o(en)g(there)h
(should)f(b)q(e)g(a)g(c)o(hance)75 747 y(to)h(recon\014gure)h(\(e.g.)j(for)c
(load)e(balacing\))h(the)i(c)o(hannels)f(b)o(y)g(deleting)g(this)g(hin)o(t)f
(and)h(storing)g(it)f(again.)166 803 y(Whic)o(h)g(sync)o(hronization)h(metho)
q(d)f(should)h(b)q(e)g(used)h(in)f(the)g(example?)75 860 y(Because)19
b Fl(MPI)p 314 860 V 14 w(BARRIER)p Fm(,)14 b Fl(MPI)p 603
860 V 15 w(IBARRIER)g Fm(or)i Fl(MPI)p 946 860 V 15 w(FENCE)g
Fm(in)g(com)o(bination)e(with)i(a)h(an)o(y)f(other)h(sync)o(hro-)75
916 y(nization)11 b(causes)i(an)e(additional)f(net)o(w)o(ork)i(tra\016c)g
(and)f(b)q(ecause)j Fl(MPI)p 1187 916 V 14 w(BARRIER)c Fm(causes)j
(additional)d(idle)h(times)75 972 y(in)16 b(some)f(user)i(pro)q(cesses,)i
(and)d(after)g(reading)g(the)h(one-sided)f(c)o(hapter,)h(it)f(seems)g(that)h
(the)f(use)h(of)f(coun)o(ters)75 1029 y(and)i(w)o(aiting)f(on)h(the)g
(threshold)h(is)f(the)h(fastest)g(metho)q(d.)30 b(The)19 b(user)g(probably)f
(o)o(v)o(erlo)q(oks)f(that)h(up)q(dating)75 1085 y(the)f(coun)o(ters)h(is)e
(also)g(an)g(additional)e(o)o(v)o(erhead.)26 b(Nev)o(ertheless)19
b(coun)o(ters)f(seem)e(to)g(b)q(e)h(the)g(fastest)g(but)g(not)75
1142 y(simplest)c(metho)q(d.)k(Therefore)e(it)f(is)g(used)g(for)g(this)g
(optimization)d(example.)-170 1210 y Fh(?)p Fu(\()p Fp(June)p
Fu(\))1967 46 y Fj(1)1967 103 y(2)1967 159 y(3)1967 215 y(4)1967
272 y(5)1967 328 y(6)1967 385 y(7)1967 441 y(8)1967 498 y(9)1959
554 y(10)1959 611 y(11)1959 667 y(12)1959 724 y(13)1959 780
y(14)1959 836 y(15)1959 893 y(16)1959 949 y(17)1959 1006 y(18)1959
1062 y(19)1959 1119 y(20)1959 1175 y(21)1959 1232 y(22)1959
1288 y(23)1959 1345 y(24)1959 1401 y(25)1959 1457 y(26)1959
1514 y(27)1959 1570 y(28)1959 1627 y(29)1959 1683 y(30)1959
1740 y(31)1959 1796 y(32)1959 1853 y(33)1959 1909 y(34)1959
1966 y(35)1959 2022 y(36)1959 2078 y(37)1959 2135 y(38)1959
2191 y(39)1959 2248 y(40)1959 2304 y(41)1959 2361 y(42)1959
2417 y(43)1959 2474 y(44)1959 2530 y(45)1959 2587 y(46)1959
2643 y(47)1959 2699 y(48)p eop
%%Page: 41 43
bop 75 381 a Fs(Bibliograph)m(y)952 2828 y Fu(41)p eop
%%Trailer
end
userdict /end-hook known{end-hook}if
%%EOF

Next message: Dennis Cottel: "Re: MPI handlers need preemption"
Previous message: (no name): "Re: MPI handlers need preemption"
Next in thread: Rolf Rabenseifner: "Re: draft of chapter 4 for next meeting -- pls comment"