I have a lot comments and proposals for additional clarifications,
most related to "caching":
1) Please correct and add on page 16, lines 20-21:
outvar = targetvar[disp];
targetvar[disp] = targetvar[disp] op invar;
provided that "targetvar" is defined as "int"
and "disp_unit==sizeof(int)"
2) I believe, page 23, lines 5-11 should be deleted.
3) Page 23, lines 46 and 48 should be clarified both:
A ... epoch must be completed ... (instead of "is completed")
^^^^
4) Page 24, lines 42-43 is wrong because after a local store
the target process must first issue a MPI_WINDOW_OUT due
to the "epoch rule" in line 46 on page 23.
Therefore I propose to add on page 24, line 43:
A preceding store must be followed by a MPI_WINDOW_OUT
that also precedes the put
(see right case in Fig.\,\ref{fig-window-in-out}).
5) The explanations on page 24, lines 33-37 for "get"
must be also given for "put". But there is an evident difference
to the "get" case.
Therefore on page 24, line 46 the following should be added:
"Because the call to MPI_WINDOW_IN must follow the put,
it is necessary that the two processes synchronize with
one of the following RMA specific methods: a) MPI_BARRIER
on the RMA communicator, b) MPI_FENCE followed by any
other synchronization or c) waiting or polling on the
window counter in the target process."
6) I propose to add the following little figures to explain
the cases on page 24
- lines 25-37 "get and MPI_WINDOW_OUT and any sync."
- lines 38-46 "put and MPI_WINDOW_IN and RMA sync."
- the addition in comment 4) above.
-------------------------------------------------------------------------------
Origin Target | Origin Target | Origin Target
| |
local stores | PUT | local stores
| |
WINDOW_OUT | RMA-sync. | WINDOW_OUT
| to guarantee |
some sync. to | \... that | some sync.to
guarantee... | the data is | guarantee...
/ | stored into | /
... | the target | ...
being after | window | being after
window_out | | window_out
has finished | WINDOW_IN | has finished
local loads | |
GET allowed | local access | PUT
-------------------------------------------------------------------------------
The left figure explains the case "page 24, lines 25-37",
the middle one "page 24, lines 38-46",
and the right one the case in the comment 4) above.
I have added the tex code of the figure at the end of the mail.
7) Page 26, lines 24-29: Because the given solution is not the
only possible -- Karl Feind has an other one --
you should modify line 24:
One solution of these problems ...
^^^-modification ^-typo
8) Also Page 26, lines 24-29: The algorithm only is correct, if
you add:
MPI_WINDOW_OUT must do the reverse operation.
9) Page 32, lines 5-38:
I think you should replace this "Rationale" by the following short
sentences that say all that is necessary for the user:
After line 3:
Then the use of MPI_WINDOW_IN and MPI_WINDOW_OUT
will prevent register coherence problems.
Details are described in the Section \ref{onesided-volatile}
on page \pageref{onesided-volatile}.
10) Please change
on page 38, lines 7-11 and page 39, line 45:
EXPOSE into WINDOW\_OUT
DELIVER into WINDOW\_IN
and on page 39, lines 21-40:
DELIVER into WINDOW_IN
Deliver into Window_in
11) Typos:
page 24, line 32: "then the" instead of "the the"
page 24, line 41: WINDOW instead of COPY
page 37, line 24 must be deleted
12) In addition to the already marked sections, the following lines
have changed in the now official voted sections of this chapter
and should be marked also with \begchangejune and \endchangejune:
page 2 lines 5-6
page 3 lines 12-14
page 5 lines 1-2, 14-16
page 6 lines 18-19, 28-29
page 7 lines 12, 19, 23
page 8 lines 20-23
page 15 lines 6-11, 43
page 17 lines 18-24
page 18 lines 15-22, 26-28, 38- page 19, line 21
page 19 lines 29-32
page 20 lines 20-24, 31-34
page 21 lines 10-15, 17-41, 46- page 22, line 4
page 32 lines 44- page 33, line 48
page 34 lines 2-4
page 2 lines 5-6
Do you agree with my comments?
Rolf
PS: Although I have a lot of comments I want to say, that your idea,
to write the caching section on the basis of a fictive modell
with the private and public copy, is better than my idea to
explain WINDOW_IN and OUT on the basis of the real caching
problems.
With my comments above I think that your text is semantically
identical with my proposal from May 6th.
--------------------------------------------------------------
\begin{figure}[ht]
\begin{minipage}[t]{49.7mm}
\begin{tabular}[t]{@{}l@{}c@{}l@{}}
\underline{Origin}&& \underline{Target}\\[5pt]
&& local stores\\[3pt]
&& \mpifunc{WINDOW\_OUT}\\[3pt]
&& some sync.\,to\\[-1pt]
&& guarantee...\\
...& $\swarrow$\\[-1pt]
being after\\[-1pt]
window\_out\\[-1pt]
has finished && \raisebox{-4pt}[4pt]{local loads}\\[3pt]
\mpifunc{GET} && allowed
\end{tabular}
\end{minipage}%
\begin{minipage}[t]{50.3mm}
\begin{tabular}[t]{@{\hspace{1mm}}|l@{}c@{\hspace{0.5mm}}l@{}}
\underline{Origin}&& \underline{Target}\\[5pt]
\mpifunc{PUT}\\[3pt]
RMA-sync.\\[-1pt]
to guarantee
&& ... that\\
& $\searrow$ &
the data is \\[-1pt]
&& stored into \\[-1pt]
&& the target \\[-1pt]
&& window \\[3pt]
&& \mpifunc{WINDOW\_IN} \\[3pt]
&& local access
\end{tabular}
\end{minipage}%
\begin{minipage}[t]{51.8mm}
\begin{tabular}[t]{@{\hspace{1mm}}|l@{\hspace{-1mm}}c@{}l@{}}
\underline{Origin}&& \underline{Target}\\[5pt]
&& local stores\\[3pt]
&& \mpifunc{WINDOW\_OUT}\\[3pt]
&& some sync.\,to\\[-1pt]
&& guarantee...\\
...& $\swarrow$\\[-1pt]
being after\\[-1pt]
window\_out\\[-1pt]
has finished\\[3pt]
\mpifunc{PUT}
\end{tabular}
\end{minipage}
\caption{The usage of \mpifunc{MPI\_WINDOW\_IN}
and \mpifunc{MPI\_WINDOW\_OUT}}
\label{fig-window-in-out}
\end{figure}
--------------------------------------------------------------
Rolf Rabenseifner (Computer Center )
Rechenzentrum Universitaet Stuttgart (University of Stuttgart)
Allmandring 30 Phone: ++49 711 6855530
D-70550 Stuttgart 80 FAX: ++49 711 6787626
Germany rabenseifner@rus.uni-stuttgart.de