<h1>Results</h1>


The output of the program looks as follows:
@code
Mesh refinement step 0
  Initial residual: 1.53143
  Residual: 1.08746
  Residual: 0.966748
  Residual: 0.859602
  Residual: 0.766462
  Residual: 0.685475

Mesh refinement step 1
  Initial residual: 0.868959
  Residual: 0.762125
  Residual: 0.677792
  Residual: 0.605762
  Residual: 0.542748
  Residual: 0.48704

Mesh refinement step 2
  Initial residual: 0.426445
  Residual: 0.382731
  Residual: 0.343865
  Residual: 0.30918
  Residual: 0.278147
  Residual: 0.250327

Mesh refinement step 3
  Initial residual: 0.282026
  Residual: 0.253146
  Residual: 0.227414
  Residual: 0.20441
  Residual: 0.183803
  Residual: 0.165319

Mesh refinement step 4
  Initial residual: 0.154404
  Residual: 0.138723
  Residual: 0.124694
  Residual: 0.112124
  Residual: 0.100847
  Residual: 0.0907222

....
@endcode

Obviously, the scheme converges, if not very fast. We will come back to
strategies for accelerating the method below.

One can visualize the solution after each set of five Newton
iterations, i.e., on each of the meshes on which we approximate the
solution. This yields the following set of images:

<div class="twocolumn" style="width: 80%">
  <div>
    <img src="https://www.dealii.org/images/steps/developer/step_15_solution_1.png"
         alt="Solution after zero cycles with contour lines." width="230" height="273">
  </div>
  <div>
    <img src="https://www.dealii.org/images/steps/developer/step_15_solution_2.png"
         alt="Solution after one cycle with contour lines." width="230" height="273">
  </div>
  <div>
    <img src="https://www.dealii.org/images/steps/developer/step_15_solution_3.png"
         alt="Solution after two cycles with contour lines." width="230" height="273">
  </div>
  <div>
    <img src="https://www.dealii.org/images/steps/developer/step_15_solution_4.png"
         alt="Solution after three cycles with contour lines." width="230" height="273">
  </div>
</div>

It is clearly visible, that the solution minimizes the surface
after each refinement. The solution converges to a picture one
would imagine a soap bubble to be that is located inside a wire loop
that is bent like
the boundary. Also it is visible, how the boundary
is smoothed out after each refinement. On the coarse mesh,
the boundary doesn't look like a sine, whereas it does the
finer the mesh gets.

The mesh is mostly refined near the boundary, where the solution
increases or decreases strongly, whereas it is coarsened on
the inside of the domain, where nothing interesting happens,
because there isn't much change in the solution. The ninth
solution and mesh are shown here:

<div class="onecolumn" style="width: 60%">
  <div>
    <img src="https://www.dealii.org/images/steps/developer/step_15_solution_9.png"
         alt="Grid and solution of the ninth cycle with contour lines." width="507" height="507">
  </div>
</div>



<a name="step-15-extensions"></a>
<h3>Possibilities for extensions</h3>

The program shows the basic structure of a solver for a nonlinear, stationary
problem. However, it does not converge particularly fast, for good reasons:

- The program always takes a step size of 0.1. This precludes the rapid,
  quadratic convergence for which Newton's method is typically chosen.
- It does not connect the nonlinear iteration with the mesh refinement
  iteration.

Obviously, a better program would have to address these two points.
We will discuss them in the following.


<h4> Step length control </h4>

Newton's method has two well known properties:
- It may not converge from arbitrarily chosen starting points. Rather, a
  starting point has to be close enough to the solution to guarantee
  convergence. However, we can enlarge the area from which Newton's method
  converges by damping the iteration using a <i>step length</i> 0<$\alpha^n\le
  1$.
- It exhibits rapid convergence of quadratic order if (i) the step length is
  chosen as $\alpha^n=1$, and (ii) it does in fact converge with this choice
  of step length.

A consequence of these two observations is that a successful strategy is to
choose $\alpha^n<1$ for the initial iterations until the iterate has come
close enough to allow for convergence with full step length, at which point we
want to switch to $\alpha^n=1$. The question is how to choose $\alpha^n$ in an
automatic fashion that satisfies these criteria.

We do not want to review the literature on this topic here, but only briefly
mention that there are two fundamental approaches to the problem: backtracking
line search and trust region methods. The former is more widely used for
partial differential equations and essentially does the following:
- Compute a search direction
- See if the resulting residual of $u^n + \alpha^n\;\delta u^n$ with
  $\alpha^n=1$ is "substantially smaller" than that of $u^n$ alone.
- If so, then take $\alpha^n=1$.
- If not, try whether the residual is "substantially smaller" with
  $\alpha^n=2/3$.
- If so, then take $\alpha^n=2/3$.
- If not, try whether the residual is "substantially smaller" with
  $\alpha^n=(2/3)^2$.
- Etc.
One can of course choose other factors $r, r^2, \ldots$ than the $2/3,
(2/3)^2, \ldots$ chosen above, for $0<r<1$. It is obvious where the term
"backtracking" comes from: we try a long step, but if that doesn't work we try
a shorter step, and ever shorter step, etc. The function
<code>determine_step_length()</code> is written the way it is to support
exactly this kind of use case.

Whether we accept a particular step length $\alpha^n$ depends on how we define
"substantially smaller". There are a number of ways to do so, but without
going into detail let us just mention that the most common ones are to use the
Wolfe and Armijo-Goldstein conditions. For these, one can show the following:
- There is always a step length $\alpha^n$ for which the conditions are
  satisfied, i.e., the iteration never gets stuck as long as the problem is
  convex.
- If we are close enough to the solution, then the conditions allow for
  $\alpha^n=1$, thereby enabling quadratic convergence.

We will not dwell on this here any further but leave the implementation of
such algorithms as an exercise. We note, however, that when implemented
correctly then it is a common observation that most reasonably nonlinear
problems can be solved in anywhere between 5 and 15 Newton iterations to
engineering accuracy &mdash; substantially fewer than we need with the current
version of the program.

More details on globalization methods including backtracking can be found,
for example, in @cite GNS08 and @cite NW99.

A separate point, very much worthwhile making, however, is that in practice
the implementation of efficient nonlinear solvers is about as complicated as
the implementation of efficient finite element methods. One should not
attempt to reinvent the wheel by implementing all of the necessary steps
oneself. Substantial pieces of the puzzle are already available in
the LineMinimization::line_search() function and could be used to this end.
But, instead, just like building finite element solvers on libraries
such as deal.II, one should be building nonlinear solvers on libraries such
as [SUNDIALS](https://computing.llnl.gov/projects/sundials). In fact,
deal.II has interfaces to SUNDIALS and in particular to its nonlinear solver
sub-package KINSOL through the SUNDIALS::KINSOL class. It would not be
very difficult to base the current problem on that interface --
indeed, that is what step-77 does.



<h4> Integrating mesh refinement and nonlinear and linear solvers </h4>

We currently do exactly 5 iterations on each mesh. But is this optimal? One
could ask the following questions:
- Maybe it is worthwhile doing more iterations on the initial meshes since
  there, computations are cheap.
- On the other hand, we do not want to do too many iterations on every mesh:
  yes, we could drive the residual to zero on every mesh, but that would only
  mean that the nonlinear iteration error is far smaller than the
  discretization error.
- Should we use solve the linear systems in each Newton step with higher or
  lower accuracy?

Ultimately, what this boils down to is that we somehow need to couple the
discretization error on the current mesh with the nonlinear residual we want
to achieve with the Newton iterations on a given mesh, and to the linear
iteration we want to achieve with the CG method within each Newton
iterations.

How to do this is, again, not entirely trivial, and we again leave it as a
future exercise.



<h4> Using automatic differentiation to compute the Jacobian matrix </h4>

As outlined in the introduction, when solving a nonlinear problem of
the form
  @f[
    F(u) \dealcoloneq
    -\nabla \cdot \left( \frac{1}{\sqrt{1+|\nabla u|^{2}}}\nabla u \right)
    = 0
  @f]
we use a Newton iteration that requires us to repeatedly solve the
linear partial differential equation
  @f{align*}{
    F'(u^{n},\delta u^{n}) &=- F(u^{n})
  @f}
so that we can compute the update
  @f{align*}{
    u^{n+1}&=u^{n}+\alpha^n \delta u^{n}
  @f}
with the solution $\delta u^{n}$ of the Newton step. For the problem
here, we could compute the derivative $F'(u,\delta u)$ by hand and
obtained
  @f[
  F'(u,\delta u)
  =
  - \nabla \cdot \left( \frac{1}{\left(1+|\nabla u|^{2}\right)^{\frac{1}{2}}}\nabla
  \delta u \right) +
  \nabla \cdot \left( \frac{\nabla u \cdot
  \nabla \delta u}{\left(1+|\nabla u|^{2}\right)^{\frac{3}{2}}} \nabla u
  \right).
  @f]
But this is already a sizable expression that is cumbersome both to
derive and to implement. It is also, in some sense, duplicative: If we
implement what $F(u)$ is somewhere in the code, then $F'(u,\delta u)$
is not an independent piece of information but is something that, at
least in principle, a computer should be able to infer itself.
Wouldn't it be nice if that could actually happen? That is, if we
really only had to implement $F(u)$, and $F'(u,\delta u)$ was then somehow
done implicitly? That is in fact possible, and runs under the name
"automatic differentiation". step-71 discusses this very
concept in general terms, and step-72 illustrates how this can be
applied in practice for the very problem we are considering here.


<h4> Storing the Jacobian matrix in lower-precision floating point variables </h4>

On modern computer systems, *accessing* data in main memory takes far
longer than *actually doing* something with it: We can do many floating
point operations for the time it takes to load one floating point
number from memory onto the processor. Unfortunately, when we do things
such as matrix-vector products, we only multiply each matrix entry once
with another number (the corresponding entry of the vector) and then we
add it to something else -- so two floating point operations for one
load. (Strictly speaking, we also have to load the corresponding vector
entry, but at least sometimes we get to re-use that vector entry in
doing the products that correspond to the next row of the matrix.) This
is a fairly low "arithmetic intensity", and consequently we spend most
of our time during matrix-vector products waiting for data to arrive
from memory rather than actually doing floating point operations.

This is of course one of the rationales for the "matrix-free" approach to
solving linear systems (see step-37, for example). But if you don't quite
want to go all that way to change the structure of the program, then
here is a different approach: Storing the system matrix (the "Jacobian")
in single-precision instead of double precision floating point numbers
(i.e., using `float` instead of `double` as the data type). This reduces
the amount of memory necessary by a factor of 1.5 (each matrix entry
in a SparseMatrix object requires storing the column index -- 4 bytes --
and the actual value -- either 4 or 8 bytes), and consequently
will speed up matrix-vector products by a factor of around 1.5 as well because,
as pointed out above, most of the time is spent loading data from memory
and loading 2/3 the amount of data should be roughly 3/2 times as fast. All
of this could be done using SparseMatrix<float> as the data type
for the system matrix. (In principle, we would then also like it if
the SparseDirectUMFPACK solver we use in this program computes and
stores its sparse decomposition in `float` arithmetic. This is not
currently implemented, though could be done.)

Of course, there is a downside to this: Lower precision data storage
also implies that we will not solve the linear system of the Newton
step as accurately as we might with `double` precision. At least
while we are far away from the solution of the nonlinear problem,
this may not be terrible: If we can do a Newton iteration in half
the time, we can afford to do a couple more Newton steps if the
search directions aren't as good.
But it turns out that even that isn't typically necessary: Both
theory and computational experience shows that it is entirely
sufficient to store the Jacobian matrix in single precision
*as long as one stores the right hand side in double precision*.
A great overview of why this is so, along with numerical
experiments that also consider "half precision" floating point
numbers, can be found in @cite Kelley2022 .
