\section{Examples\label{examples}}
A sample program is described by completing the program fragments given earlier
(Figures~\ref{highlevel_code},~\ref{init_options} and~\ref{init_mv_structs}).
In Figure~\ref{highlevel_code}, {\sf AZ\_set\_proc\_config} is an \Az{} utility
which initializes the array {\it proc\_config} to reflect the number of
processors being used and the node number of this processor. The function {\sf
  AZ\_solve} is also supplied by \Az{} to solve the user supplied linear
system. Thus, the only functions that the user must supply which have not
already been discussed include: {\sf init\_guess\_and\_rhs} in
Figure~\ref{highlevel_code} and {\sf create\_matrix} in
Figure~\ref{init_mv_structs}.

The function {\sf init\_guess\_and\_rhs} initializes the initial guess
\begin{figure}[Htbp]
  \shadowbox{
%    \begin{minipage}{\textwidth}
    \begin{minipage}{6.2in}
      \vspace{0.5em}
      {\large \flushleft{\bf Example}} \hrulefill %
      \vspace{0.5em}
%%%
\begin{verbatim}
void init_guess_and_rhs(x, rhs, data_org, update, update_index)
{
  N_update = data_org[AZ_N_internal] + data_org[AZ_N_border];
  for (i = 0; i < N_update ; i = i + 1) {
    rhs[i] = (double) update[i];
    x[i] = 0.0;
  }
  AZ_reorder_vec(rhs,data_org,update_index,NULL); 
  AZ_reorder_vec(  x,data_org,update_index,NULL); 
                            /* reorder 'rhs' and 'x' to correspond to    */
                            /* AZ_transformed matrix. Note: convert      */
                            /* solution from AZ_solve (or AZ_iterate)    */
                            /* back to user ordering via AZ_invorder_vec */
}
\end{verbatim}
%%%
      \vspace{0.1em}
    \end{minipage}}
  \caption{{\sf init\_guess\_and\_rhs}.}\label{init_guess_rhs}
\end{figure}
and the right hand side. In Figure~\ref{init_guess_rhs}, a sample routine is
given which sets the initial guess vector to zero and sets the right hand side
vector equal to the global indices (where the local element {\it
  update\_index[i]\/} corresponds to global element {\it update[i]\/}, see
Section \ref{highlevel_data_inter})\footnote{Alternatively, $i$ could replace
  {\sl update\_index[i]} by invoking {\sl AZ\_reorder\_vec()} once {\sl rhs}
  was initialized.}.

A {\sf create\_matrix} function to initialize an MSR matrix is illustrated in
Figure~\ref{create_matrix}.  Different matrix problems can be implemented by
changing the function {\sf add\_row} which computes the MSR entries
corresponding to a new row of the matrix.
%
\begin{figure}[Htbp]
  \shadowbox{
%    \begin{minipage}{\textwidth}
    \begin{minipage}{6.2in}
      \vspace{0.5em}
      {\large \flushleft{\bf Example}} \hrulefill %
      \vspace{0.5em}
%%%
\begin{verbatim}
void create_matrix(bindx, val, update, N_update)
{
  N_nonzeros = N_update + 1;
  bindx[0] = N_nonzeros;

  for (i = 0; i < N_update; i = i + 1)
    add_row(update[i], i, val, bindx);
}
\end{verbatim}
%%%
      \vspace{0.1em}
    \end{minipage}}
  \caption{{\sf create\_matrix}.}\label{create_matrix}
\end{figure}
%
The specific {\sf add\_row} function for implementing a 5-point 2D Poisson
operator on an $n \times n$ grid is shown in Figure~\ref{add_row} ($n$ is a
global variable set by the user).  With
\begin{figure}[t]
  \shadowbox{
%    \begin{minipage}{\textwidth}
    \begin{minipage}{6.2in}
      \vspace{0.5em}
      {\large \flushleft{\bf Example}} \hrulefill %
      \vspace{0.5em}
%%%
\begin{verbatim}
void add_row(row, location, val, bindx)
{
   k = bindx[location];   val[location] = 4.; /* matrix diagonal */

   /* check neighboring points in each direction and add nonzero */
   /* entry if neighbor exists.                                  */

   bindx[k]  = row + 1;   if (row%n !=     n-1) val[k++] = -1.;
   bindx[k]  = row - 1;   if (row%n !=       0) val[k++] = -1.;
   bindx[k]  = row + n;   if ((row/n)%n != n-1) val[k++] = -1.;
   bindx[k]  = row - n;   if ((row/n)%n !=   0) val[k++] = -1.;

   bindx[location+1] = k;
}
\end{verbatim}
%%%
      \vspace{0.1em}
    \end{minipage}}
  \caption{{\sf add\_row} for a 2D Poisson problem}\label{add_row}
\end{figure}
these few lines of code and the functions described earlier, the user
initializes and solves a 2D Poisson problem. While for simplicity of
presentation this specific example is structured the \Az{} library does not
assume any structure in the sparse matrix.  All the communication and variable
renumbering is done automatically without the assumption of structured
communication.

Other {\sf add\_row} functions corresponding to a 3D Poisson equation and a
high order 2D Poisson equation are distributed with \Az{} (file
\verb'az_examples.c').  We recommend that potential users review these
examples.  In many cases, new applications can be written by simply editing
these programs.  The interested reader should note that only a few lines of
code are different between the functions for the 5-pt Poisson, the high order
Poisson and the 3D Poisson codes.  Further, the {\sf add\_row} routines are
essentially identical to those that would be used to set up sparse matrices in
serial applications and that there are no references to processors,
communications or anything specific to parallel programming.
\begin{figure}[Htbp]
  \shadowbox{
%    \begin{minipage}{\textwidth}
    \begin{minipage}{6.2in}
      \vspace{0.5em}
      {\large \flushleft{\bf Example}} \hrulefill %
      \vspace{0.5em}
%%%
\begin{tabbing}
\tt void create\_matrix(bindx, val, update, N\_update);\\
%\tt 
\{\\
\in \tt read\_triangles(T, N\_triangles);\\
\>  \tt init\_msr(val, bindx, N\_update);\\[1em]
\>  \tt for (triangle = 0; triangle $<$ N\_triangles; triangle = triangle + 1)\\
\>\in {\tt for (i = 0; i $<$ 3; i = i + 1)} \{\\
\>\>\in \tt row = \underline{AZ\_find\_index(}T[triangle][i]\underline{, update,N\_update)}; \\
\> \>\> {\tt for (j = 0; j $<$ 3; j = j + 1) } \{\\
\>\>\>\in \tt \underline{if (row != NOT\_FOUND)}\\
\>\>\>\>\in \tt add\_to\_element(row, T[triangle][j], \underline{0.0}, val, bindx, i==j);\\
%\> \>\> \tt \}\\
%\> \> \tt \}\\
%\>  \tt \} \} \} \\
\>  \} \} \} \\
\>  \tt compress\_matrix(val, bindx, N\_update);\\
\}
\end{tabbing}
%%%
\vspace{0.1em}
\end{minipage}}
\caption{{\sf create\_matrix} for the Poisson finite element
problem.}\label{fem}
\end{figure}

While \Az{} simplifies the parallel coding associated with structured problems,
it is for unstructured problems that \Az{} makes a significant programming
difference. To illustrate this, a 2D finite element example is given where the
underlying grid is a triangulation of a complex geometry. Unlike the previous
example
%Additionally, this example
%will illustrate another use of \Az{}.
%is defined.
%main utility com
%To illusrate the simplicity of the
%The parallel computing aspects of unstructured problems are no more
%difficult than structured problems.
%We conclude this section with two applications where the values of the
%matrix entries are not defined inside {\sf create\_matrix}.
%In particular,
{\sf create\_matrix} defines a sparsity pattern (i.e. {\it bindx\/}) but not
the actual nonzero entries (i.e. {\it val\/}) as interprocessor communication
is required before they can be computed.  Thus, in this example {\sf
  AZ\_transform} takes the sparsity pattern and initializes the communication
data structures. Using these structures, communication can be performed at a
later stage in computing the matrix nonzeros.

Figure~\ref{fem} depicts {\sf create\_matrix} while
Figure~\ref{matrix_fill_fem} depicts an additional function {\sf matrix\_fill}
that must be included before {\sf AZ\_solve} is invoked in
Figure~\ref{highlevel_code}.  We have not made any effort to optimize these
routines. In both figures the new lines that have been added specifically for a
parallel implementation are underlined.  That is, {\sf create\_matrix} and {\sf
  matrix\_fill} have been created by taking a serial program that creates the
finite element discretization, splitting this program over the two functions
and adding a few new lines necessary for the parallel implementation.  The only
additional change is to replace the single data file containing the triangle
connectivity read using {\sf read\_triangles} by a set of data files containing
the triangle connectivity for each processor.  We do not discuss the details of
this program but only wish to draw the readers attention to the small number of
lines that need changing to convert the serial unstructured application to
parallel.  Most of the main routines such as {\sf setup\_Ke} which computes the
element contributions and {\sf add\_to\_element} which stores the element
contributions in the MSR data structures remain the same.  In fact, almost all
the new lines of code correspond to adding the communication ({\sf
  AZ\_exchange\_bdry}) (which was the main reason that the calculation of the
matrix nonzeros was deferred) and the conversion of global index values by
local index values with the help of {\sf AZ\_find\_index}.  As in the Poisson
example, all of the details with respect to communication are hidden from the
user.
%%%
\begin{figure}[p]
  \shadowbox{
%    \begin{minipage}{\textwidth-.5in}
    \begin{minipage}{6.2in}
      \vspace{0.5em}
      {\large \flushleft{\bf Example}} \hrulefill %
      \vspace{0.5em}
%%%
\begin{tabbing}
\tt void matrix\_fill(bindx, val, N\_update, update, update\_index,\\
\hskip 1.45in \tt     N\_external, external, extern\_index)\\[1em]

\in\tt/* read the x and y coordinates from an input file */\\[1em]

\>{\tt for (i = 0; i $<$ N\_update; i = i + 1)} \{\\
\>\tt\in read\_from\_file(x[\underline{update\_index[i]}], y[\underline{update\_index[i]}]);\\
\> \}\\
\>\tt \underline{AZ\_exchange\_bdry(x);}\\
\>\tt \underline{AZ\_exchange\_bdry(y);}\\[1em]


\in\tt/* Locally renumber rows and columns of the new sparse matrix */\\[1em]

\>\tt \underline{for (triangle = 0; triangle $<$ N\_triangles; triangle = triangle + 1)}\\
\>\>{\tt \underline{for (i = 0; i $<$ 3; i = i + 1)}} \underline{\{}\\
\>\>\in\tt \underline{row = AZ\_find\_index(T[triangle][i], update, N\_update);}\\
\>\>\>{\tt  \underline{if (row == NOT\_FOUND)} } \{\\
\>\>\>\in\tt  \underline{row = AZ\_find\_index(T[triangle][i], external, N\_external);}\\
\>\>\>\>\tt   \underline{T[triangle][i] = extern\_index[row];}\\
\>\>\>  \underline{\}}\\
\>\>\>\tt  \underline{else T[triangle][i] = update\_index[row];}\\
%\>\>\>\>\tt
%\>\>\>\tt  \underline{\}}\\
\>\> \underline{\}}\\
\> \underline{\}}\\[1em]


\in\tt/* Fill the element stiffness matrix Ke */\\[1em]

\>{\tt for (triangle = 0; triangle $<$ N\_triangles; triangle = triangle + 1)}\{\\
\>\>\tt setup\_Ke(Ke, x[T[triangle][0]], y[T[triangle][0]],\\
\>\>\hskip 1.1in  \tt x[T[triangle][1]], y[T[triangle][1]],\\
\>\>\hskip 1.1in  \tt x[T[triangle][2]], y[T[triangle][2]]);\\[1em]


\in\tt/* Fill sparse matrix by scattering Ke to appropriate locations */\\[1em]

\>\>{\tt for (i = 0; i $<$ 3; i = i + 1) } \{\\
\>\>\>{\tt  for (j = 0; j $<$ 3; j = j + 1)}\{\\
\>\>\>\>{\tt   \underline{if (T[triangle][i] $<$ N\_update)}} \underline{\{}\\
\>\>\>\>\in\tt   add\_to\_element(T[triangle][i], T[triangle][j], Ke[i][j],\\
\hskip 2.18in     \tt              val, bindx, i==j);\\
\>\>\>\>   \}\\
\>\>\>  \}\\
\>\> \}\\
\> \}\\
\end{tabbing}
%%%
\vspace{0.1em}
\end{minipage}}
\caption{{\sf matrix\_fill} for the Poisson finite element
problem.}\label{matrix_fill_fem}
\vspace{-8em}
\end{figure}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "az_ug_20"
%%% End:
