\section{Advanced Topics\label{advanced_topics}}
\subsection{Data Layout\label{comm_vars}}
The \Az{} function {\sf AZ\_transform} initializes the integer array {\it
  data\_org}.  This array specifies how the matrix is set up on the parallel
machine.  In many cases, the user need not be concerned with the contents of
this array. However, in some situations it is useful to initialize these
elements without the use of {\sf AZ\_transform}, to access these array elements
(e.g. determine how many {\it internal} components are used), or to change
these array elements (e.g. when reusing factorization information, see
Section~\ref{reusing}). When using the transformation software, the user can
ignore the size of {\it data\_org} as it is allocated in {\sf AZ\_transform}.
However, when this is not used, {\it data\_org} must be allocated of size {\sf
  AZ\_COMM\_SIZE} $+$ number of vector elements sent to other processors during
matrix-vector multiplies.  The contents of {\it data\_org} are as follows:
\vspace{2em} {\flushleft{\bf Specifications} \hrulefill}
\nopagebreak \\[0.5em]
%
\optionbox{data\_org[{\sf AZ\_matrix\_type}]}{Specifies matrix
format.}
%
        \choicebox{AZ\_VBR\_MATRIX}{Matrix corresponds to VBR format.}
%
        \choicebox{AZ\_MSR\_MATRIX}{Matrix corresponds to MSR format.}
%
\optionbox{data\_org[{\sf AZ\_N\_internal}]}{Number of elements
                                      updated by this processor that
                                      can be computed without
                                      information from neighboring
                                      processors
                                      ({\it N\_internal\/}). This also
                                      corresponds to the number of internal
                                      rows assigned to this processor.}
%
\optionbox{data\_org[{\sf AZ\_N\_border}]}{Number of elements
                                      updated by this processor that
                                      use information from
                                      neighboring processors ({\it
                                      N\_border\/}).}
%
\optionbox{data\_org[{\sf AZ\_N\_external}]}{Number of {\it external}
                                      components needed by this
                                      processor ({\it
                                      N\_external\/}).}
%
\optionbox{data\_org[{\sf AZ\_N\_int\_blk}]}{Number of internal
                                      VBR block rows owned by this
                                      processor.  Set to
                                      data\_org[{\sf AZ\_N\_internal}]
                                      for MSR matrices.  }
%
\optionbox{data\_org[{\sf AZ\_N\_bord\_blk}]}{Number of border VBR block
                                      rows owned by this processor.
                                      Set to
                                      data\_org[{\sf AZ\_N\_border}]
                                      for MSR matrices.  }
%
\optionbox{data\_org[{\sf AZ\_N\_ext\_blk}]}{Number of external VBR
                                      block rows on this
                                      processor.  Set to
                                      data\_org[{\sf AZ\_N\_external}]
                                      for MSR matrices.  }
%
\optionbox{data\_org[{\sf AZ\_N\_neigh}]}{Number of processors with
                                      which we exchange information
                                      (send or receive) in performing
                                      matrix-vector products.}
%
\optionbox{data\_org[{\sf AZ\_total\_send}]}{Total number of vector
                                      elements sent to other processors
                                      during matrix-vector products.}
%
\optionbox{data\_org[{\sf AZ\_name}]}{Name of the matrix. This
                                      name is utilized when deciding
                                      which previous factorization to
                                      use as a preconditioner (see
                                      Section~\ref{reusing}).
                                      (positive integer value).}
%
\optionbox{data\_org[{\sf AZ\_neighbors}]}{Start of vector containing
                                      node i.d.'s of neighboring
                                      processors.  That is, {\it
                                      data\_org[{\sf
                                      AZ\_neighbors}+i]\/} gives the
                                      node i.d. of the ({\it
                                      i+1\/})'th neighbor.}
%
\optionbox{data\_org[{\sf AZ\_rec\_length}]}{Start of vector containing
                                      the number of elements to
                                      receive from each neighbor.
                                      We receive from the ({\it
                                      i+1\/})'th neighbor {\it
                                      data\_org[{\sf AZ\_rec\_length}+i]\/}
                                      elements.}
%
\optionbox{data\_org[{\sf AZ\_send\_length}]}{Start of vector containing
                                      the number of elements to send
                                      to each neighbor. We send to the
                                      ({\it i+1\/})'th neighbor {\it
                                      data\_org[{\sf AZ\_rec\_length}+i]\/}
                                      elements.}
%
\optionbox{data\_org[{\sf AZ\_send\_list}]}{Start of vector indicating the
                                      elements that we will send to
                                      other processors during
                                      communication. The first {\it
                                      data\_org[{\sf AZ\_send\_length}]\/}
                                      components correspond to the
                                      elements for the first neighbor
                                      and the next {\it
                                      data\_org[{\sf AZ\_send\_length}+1]\/}
                                      components correspond to element
                                      indices for the second neighbor,
                                      and so on.} $\hphantom{hi}$
%
\subsection{Reusing factorizations}\label{reusing}
By solving a problem with {\it options[{\sf AZ\_keep\_info}]} set to '1', 
\Az{} will not remove information so that it can be reused
later. In most cases, this information corresponds to either matrix scaling
factors or preconditioning factorization information for LU or ILU.  This
information is saved internally and referenced by the matrix name given by {\it
  data\_org[{\sf AZ\_name]}}. By changing {\it options[{\sf AZ\_pre\_calc}]}
and {\it data\_org[{\sf AZ\_name}]} a number of different \Az{} possibilities
can be realized. As an example, consider the following situation. A user needs
to solve the linear systems in the order shown below:
\[
A_1 x = b , A_2 y = x , \mbox{ and } A_1 z = y.
\]
The first and second systems are solved with {\it options[{\sf AZ\_pre\_calc}]}
set to {\sf AZ\_calc}. However, the name (i.e. {\it data\_org[{\sf AZ\_name}]})
is changed between these two solves. In this way, scaling and preconditioning
information computed from the first solve is not overwritten during the second
solve.  By then setting {\it options[{\sf AZ\_pre\_calc}]} to {\sf AZ\_reuse}
and {\it data\_org[{\sf AZ\_name}]} to the name used during the first solve,
the third system is solved reusing the scaling information (to scale the right
hand side, initial guess, and rescale the final solution\footnote{The matrix
  does not need to be rescaled as the scaling during the first solve overwrites
  the original matrix.}) and the preconditioning factorizations (e.g. ILU) used
during the first solve. While in this example the same matrix system is solved
for the first and third solve, this is not necessary. In particular,
preconditioners can be reused from previous nonlinear iterates even though the
linear system being solved are changing. Of course, many times information from
previous linear solves is not reused. In this case the user must explicitly
free the space associated with the matrix or this information will remain
allocated for the duration of the program. Space is cleared by invoking {\sf
  AZ\_free\_memory({\it data\_org[{\sf AZ\_name}]})}.

\subsection{Important Constants}
\Az{} uses a number of constants which are defined in the file
\verb'az_aztec_defs.h'. Most users can ignore these constants.  However, there
may be situations where they should be changed.  Below
is a list of these constants with a brief description:\\[2em]
\choicebox{AZ\_MAX\_NEIGHBORS}{Maximum number of processors with which
  information can be exchanged during matrix-vector products.}
\choicebox{AZ\_MSG\_TYPE\\ AZ\_NUM\_MSGS}{All message types used inside \Az{}
  lie between AZ\_MSG\_TYPE and AZ\_MSG\_TYPE + AZ\_NUM\_MSGS - 1.}
\choicebox{AZ\_MAX\_BUFFER\_SIZE}{Maximum message information that can be sent
  by any processor at any given time before receiving. This is used to
  subdivide large messages to avoid buffer overflows.}
\choicebox{AZ\_MAX\_MEMORY\_SIZE}{Maximum available memory.  Used primarily for
  the LU-factorizations where a large amount of memory is first allocated and
  then unused portions are freed after factorization.}
\choicebox{AZ\_TEST\_ELE}{Internal algorithm parameter that can effect the
  speed of the {\sf AZ\_find\_procs\_for\_externs} calculation.  Reduce {\sf
    AZ\_TEST\_ELE} if communication buffers are exceeded during this
  calculation.}  $\hphantom{h}$

\subsection{{\sf AZ\_transform} Subtasks}\label{subtasks}
The function {\sf AZ\_transform} described in
Section~\ref{highlevel_data_inter} is actually made up of 5 subtasks. In most
cases the user need not be concerned with the individual tasks. However, there
might arise situations where additional information is available such that some
of the subtasks can be omitted. In this case, it is possible for the user to
edit the code for {\sf AZ\_transform} located in the file \verb'az_tools.c' to
suit the application.  In this section we briefly describe the five subroutines
which make up the transformation function.  More detailed descriptions are
given in~\cite{aztec-utils}.  Prototypes for these subroutines as well as for
{\sf AZ\_transform} are given in Section~\ref{subroutines}.

{\sf AZ\_transform} begins by identifying the {\it external} set needed by each
processor.  Here, each column entry must correspond to either an element
updated by this processor or an {\it external} component.  The function {\sf
  AZ\_find\_local\_indices} checks each column entry.  If a column is in {\it
  update\/}, its number is replaced by the appropriate index into {\it
  update\/} (i.e. {\it update[new column index]\/} = old column index).  If a
column number is not found in {\it update\/}, it is stored in the {\it
  external} list and the column number is replaced by an index into {\it
  external\/} (i.e.  {\it external[new column index - N\_update]\/} = old
column index).

{\sf AZ\_find\_procs\_for\_externs} queries the other processors to determine
which processors update each of its {\it external} components. The array {\it
  extern\_proc\/} is set such that {\it extern\_proc[i]\/} indicates which
processor updates {\it external[i]\/}.

{\sf AZ\_order\_ele} reorders the {\it external} components such that elements
updated by the same processor are contiguous. This new ordering is given by
{\it extern\_index\/} where {\it extern\_index[i]\/} indicates the local
numbering of {\it external[i]\/}.  Additionally, {\it update} components are
reordered so the {\it internal} components precede the {\it border} components.
This new ordering is given by {\it update\_index\/} where {\it
  update\_index[i]\/} indicates the local numbering of {\it update[i]\/}.

{\sf AZ\_set\_message\_info} initializes {\it data\_org\/} (see
Section~\ref{comm_vars}) This is done by computing the number of neighbors,
making a list of the neighbors, computing the number of values to be sent and
received with each neighbor and computing the list of elements which will be
sent to other processors during communication steps.

Finally, {\sf AZ\_reorder\_matrix} permutes and reorders the matrix nonzeros so
that its entries correspond to the newly reordered vector elements.

