<br>

<i>This program was contributed by Timo Heister, Martin Kronbichler, and Wolfgang
Bangerth.
<br>
This material is based upon work partly supported by the National
Science Foundation under Award No. EAR-0426271 and The California Institute of
Technology. Any opinions, findings, and conclusions or recommendations
expressed in this publication are those of the author and do not
necessarily reflect the views of the National Science Foundation or of The
California Institute of Technology.
</i>


@note As a prerequisite of this program, you need to have both PETSc and the
p4est library installed. The installation of deal.II
together with these two additional libraries is described in the <a
href="../../readme.html" target="body">README</a> file. Note also that
to work properly, this program needs access to the Hypre
preconditioner package implementing algebraic multigrid; it can be
installed as part of PETSc but has to be explicitly enabled during
PETSc configuration; see the page linked to from the installation
instructions for PETSc.


<a name="step-40-Intro"></a>
<h1>Introduction</h1>

@dealiiVideoLecture{41.5,41.75}

Given today's computers, many finite element computations can be done on
a single machine. The majority of previous tutorial programs therefore
shows only this, possibly splitting up work among a number of
processors that, however, can all access the same, shared memory
space. That said, there are problems that are simply too big for a
single machine and in that case the problem has to be split up in a
suitable way among multiple machines each of which contributes its
part to the whole. A simple way to do that was shown in step-17 and
step-18, where we show how a program can use <a
href="http://www.mpi-forum.org/" target="_top">MPI</a> to parallelize
assembling the linear system, storing it, solving it, and computing
error estimators. All of these operations scale relatively trivially
(for a definition of what it means for an operation to "scale", see
@ref GlossParallelScaling "this glossary entry"),
but there was one significant drawback: For this to be moderately
simple to implement, each MPI processor had to keep its own copy of
the entire Triangulation and DoFHandler objects. Consequently, while
we can suspect (with good reasons) that the operations listed above
can scale to thousands of computers and problem sizes of billions of
cells and billions of degrees of freedom, building the one big mesh for the
entire problem these thousands of computers are solving on every last
processor is clearly not going to scale: it is going to take forever,
and maybe more importantly no single machine will have enough memory
to store a mesh that has a billion cells (at least not at the time of
writing this). In reality, programs like step-17 and step-18 can therefore
not be run on more than maybe 100 or 200 processors, and even then storing
the Triangulation and DoFHandler objects consumes the vast majority of
memory on each machine.

Consequently, we need to approach the problem differently: to scale to
very large problems, each processor can only store its own little piece
of the Triangulation and DoFHandler objects. deal.II implements such a
scheme in the parallel::distributed namespace and the classes
therein. It builds on an external library, <a
href="http://www.p4est.org/">p4est</a> (a word play on the expression
<i>parallel forest</i> that describes the parallel storage of a
hierarchically constructed mesh as a forest of quad- or
oct-trees). You need to <a
href="../../external-libs/p4est.html">install p4est and configure deal.II
to use it</a>,
but apart from that, all of its workings are hidden under the surface
of deal.II.

In essence, what the parallel::distributed::Triangulation class and
code inside the DoFHandler class do is to split
the global mesh so that every processor only stores a small bit it
"owns" along with one layer of "ghost" cells that surround the ones it
owns. What happens in the rest of the domain on which we want to solve
the partial differential equation is unknown to each processor and can
only be inferred through communication with other machines if such
information is needed. This implies that we also have to think about
problems in a different way than we did in, for example, step-17 and
step-18: no processor can have the entire solution vector for
postprocessing, for example, and every part of a program has to be
parallelized because no processor has all the information necessary
for sequential operations.

A general overview of how this parallelization happens is described in
the @ref distributed documentation topic. You should read it for a
top-level overview before reading through the source code of this
program. A concise discussion of many terms we will use in the program
is also provided in the @ref distributed_paper "Distributed Computing paper".
It is probably worthwhile reading it for background information on how
things work internally in this program.


<h3>Linear algebra</h3>

step-17 and step-18 already used parallel linear algebra classes, but since
the current program is the first one that *really* covers parallel computing,
it is probably worth giving a broad overview of parallel linear algebra here
as well.

First, the general mantra mentioned above was that *everything* has to be
distributed. It does not scale if one process (or in fact *all* processes)
have to keep a complete triangulation or even a substantial share of it; it
all only works if every one of the $N$ processes in the parallel universe
keep at most a small multiple of one $N$th of the triangulation.
Similarly, each process can only hold a small multiple of one $N$th of
each solution or right hand side vector, and of the system matrix.

To this end, deal.II has acquired interfaces to a number of packages
that provide these kind of distributed linear algebra data structures.
More specifically, deal.II comes with a number of "sub-packages" that
all provide vector, matrix, and linear solver classes that are typically
named the same or very similarly, but live in different namespaces:
- deal.II's own linear algebra classes. These are what we have been
  using in step-1 to step-6, for example, along with most of the
  other programs in the deal.II tutorial. These classes are all not
  parallel in the sense that they do not use MPI, can not subdivide
  the data among processes, or work on them on processes that cannot
  access each other's memory directory. (On the other hand, many of
  these classes actually use multiple threads internally, to use
  the multiple processor cores available on today's laptops and
  work stations.) These classes reside in the top-level
  `namespace dealii`.
- Interfaces to the PETSc library's implementations of linear
  algebra functionality. These are found in namespace `PETScWrappers`.
  PETSc is a library that has built a large collection of linear algebra,
  linear solvers, nonlinear solvers, time steppers, and other functionality
  that runs on some of the largest machines in the world in parallel,
  using MPI.
- The classes in the TrilinosWrapper namespace wrap functionality
  provided by the Trilinos project. Trilinos is, in many regards, similar
  in functionality to what PETSc provides, but it does so in a very different
  way (namely, as a bunch of independent and loosely coupled sub-projects,
  rather than as a single library). This nothwithstanding, the classes
  deal.II provides in namespace TrilinosWrappers are very similar to the
  ones in namespace PETScWrappers. Trilinos, like PETSc, is run on some of
  the biggest machines in the world.
- The classes in namespace TrilinosWrappers are generally written for the
  functionality Trilinos provides via its "Epetra" collection of vector
  and matrix classes, and everything that builds on it. Epetra has been
  slated for removal since the early 2020s (and may already have been removed
  by the time you read this), with replacements provided in the "Tpetra"
  collection. Tpetra replaces the old classes by ones that are templatized
  on the type of objects it stores (thus the "T" at the beginning of the
  name; "petra" being the word for "rock" or "foundation" since the rest of
  Trilinos builds on these packages). deal.II is wrapping the Tpetra classes
  in namespace LinearAlgebra::TpetraWrappers, but this process is still
  ongoing and may be available in a version of deal.II after release 9.5.
- deal.II also implements a subset of parallel linear algebra functionality
  internally through the LinearAlgebra::distributed::Vector class (compatible
  with most Trilinos classes) and matrix-free linear solvers, which are run
  on some of the biggest machines in the world as well. For further details, see
  the step-37 tutorial program.

For the current program, we need to use parallel linear algebra classes
to represent the matrix and vectors. Both the PETScWrapper and TrilinosWrapper
classes will fit the bill and, depending on whether deal.II was configured
with one or the other (or both), the top of the program selects one or the
other set of wrappers by putting the respective class names into a
namespace `LA`.



<h3>The testcase</h3>

This program essentially re-solves what we already do in
step-6, i.e. it solves the Laplace equation
@f{align*}{
  -\Delta u &= f \qquad &&\text{in}\ \Omega=[0,1]^2, \\
  u &= 0 \qquad &&\text{on}\ \partial\Omega.
@f}
The difference of course is now that we want to do so on a mesh that
may have a billion cells, with a billion or so degrees of
freedom. There is no doubt that doing so is completely silly for such
a simple problem, but the point of a tutorial program is, after all,
not to do something useful but to show how useful programs can be
implemented using deal.II. Be that as it may, to make things at least
a tiny bit interesting, we choose the right hand side as a
discontinuous function,
@f{align*}{
  f(x,y)
  =
  \left\{
  \begin{array}{ll}
    1 & \text{if}\ y > \frac 12 + \frac 14 \sin(4\pi x), \\
    -1 & \text{otherwise},
  \end{array}
  \right.
@f}
so that the solution has a singularity along the sinusoidal line
snaking its way through the domain. As a consequence, mesh refinement
will be concentrated along this line. You can see this in the mesh
picture shown below in the results section.

Rather than continuing here and giving a long introduction, let us go
straight to the program code. If you have read through step-6 and the
@ref distributed documentation topic, most of things that are going
to happen should be familiar to you already. In fact, comparing the two
programs you will notice that the additional effort necessary to make things
work in %parallel is almost insignificant: the two programs have about the
same number of lines of code (though step-6 spends more space on dealing with
coefficients and output). In either case, the comments below will only be on
the things that set step-40 apart from step-6 and that aren't already covered
in the @ref distributed documentation topic.


@note This program will be able to compute on as many processors as you want
to throw at it, and for as large a problem as you have the memory and patience
to solve. However, there <i>is</i> a limit: the number of unknowns can not
exceed the largest number that can be stored with an object of type
types::global_dof_index. By default, this is an alias for <code>unsigned
int</code>, which on most machines today is a 32-bit integer, limiting you to
some 4 billion (in reality, since this program uses PETSc, you will be limited
to half that as PETSc uses signed integers). However, this can be changed
during configuration to use 64-bit integers, see the ReadMe file. This will
give problem sizes you are unlikely to exceed anytime soon.
