<h1>Results</h1>

<h3>Program output</h3>

Like in step-37, we evaluate the multigrid solver in terms of run time.  In
two space dimensions with elements of degree 8, a possible output could look
as follows:
@code
Running with 12 MPI processes, element FE_DGQHermite<2>(8)

Cycle 0
Number of degrees of freedom: 5184
Total setup time              0.0282445 s
Time solve (14 iterations)    0.0110712 s
Verification via L2 error:    1.66232e-07

Cycle 1
Number of degrees of freedom: 20736
Total setup time              0.0126282 s
Time solve (14 iterations)    0.0157021 s
Verification via L2 error:    2.91505e-10

Cycle 2
Number of degrees of freedom: 82944
Total setup time              0.0227573 s
Time solve (14 iterations)    0.026568 s
Verification via L2 error:    6.64514e-13

Cycle 3
Number of degrees of freedom: 331776
Total setup time              0.0604685 s
Time solve (14 iterations)    0.0628356 s
Verification via L2 error:    5.57513e-13

Cycle 4
Number of degrees of freedom: 1327104
Total setup time              0.154359 s
Time solve (13 iterations)    0.219555 s
Verification via L2 error:    3.08139e-12

Cycle 5
Number of degrees of freedom: 5308416
Total setup time              0.467764 s
Time solve (13 iterations)    1.1821 s
Verification via L2 error:    3.90334e-12

Cycle 6
Number of degrees of freedom: 21233664
Total setup time              1.73263 s
Time solve (13 iterations)    5.21054 s
Verification via L2 error:    4.94543e-12
@endcode

Like in step-37, the number of CG iterations remains constant with increasing
problem size. The iteration counts are a bit higher, which is because we use a
lower degree of the Chebyshev polynomial (2 vs 5 in step-37) and because the
interior penalty discretization has a somewhat larger spread in
eigenvalues. Nonetheless, 13 iterations to reduce the residual by 12 orders of
magnitude, or almost a factor of 9 per iteration, indicates an overall very
efficient method. In particular, we can solve a system with 21 million degrees
of freedom in 5 seconds when using 12 cores, which is a very good
efficiency. Of course, in 2D we are well inside the regime of roundoff for a
polynomial degree of 8 &ndash; as a matter of fact, around 83k DoFs or 0.025s
would have been enough to fully converge this (simple) analytic solution
here.

Not much changes if we run the program in three spatial dimensions, except for
the fact that we now use do something more useful with the higher polynomial
degree and increasing mesh sizes, as the roundoff errors are only obtained at
the finest mesh. Still, it is remarkable that we can solve a 3D Laplace
problem with a wave of three periods to roundoff accuracy on a twelve-core
machine pretty easily - using about 3.5 GB of memory in total for the second
to largest case with 24m DoFs, taking not more than eight seconds. The largest
case uses 30GB of memory with 191m DoFs.

@code
Running with 12 MPI processes, element FE_DGQHermite<3>(8)

Cycle 0
Number of degrees of freedom: 5832
Total setup time              0.0210681 s
Time solve (15 iterations)    0.0956945 s
Verification via L2 error:    0.0297194

Cycle 1
Number of degrees of freedom: 46656
Total setup time              0.0452428 s
Time solve (15 iterations)    0.113827 s
Verification via L2 error:    9.55733e-05

Cycle 2
Number of degrees of freedom: 373248
Total setup time              0.190423 s
Time solve (15 iterations)    0.218309 s
Verification via L2 error:    2.6868e-07

Cycle 3
Number of degrees of freedom: 2985984
Total setup time              0.627914 s
Time solve (15 iterations)    1.0595 s
Verification via L2 error:    4.6918e-10

Cycle 4
Number of degrees of freedom: 23887872
Total setup time              2.85215 s
Time solve (15 iterations)    8.30576 s
Verification via L2 error:    9.38583e-13

Cycle 5
Number of degrees of freedom: 191102976
Total setup time              16.1324 s
Time solve (15 iterations)    65.57 s
Verification via L2 error:    3.17875e-13
@endcode

<h3>Comparison of efficiency at different polynomial degrees</h3>

In the introduction and in-code comments, it was mentioned several times that
high orders are treated very efficiently with the FEEvaluation and
FEFaceEvaluation evaluators. Now, we want to substantiate these claims by
looking at the throughput of the 3D multigrid solver for various polynomial
degrees. We collect the times as follows: We first run a solver at problem
size close to ten million, indicated in the first four table rows, and record
the timings. Then, we normalize the throughput by recording the number of
million degrees of freedom solved per second (MDoFs/s) to be able to compare
the efficiency of the different degrees, which is computed by dividing the
number of degrees of freedom by the solver time.

<table align="center" class="doxtable">
  <tr>
   <th>degree</th>
   <th>1</th>
   <th>2</th>
   <th>3</th>
   <th>4</th>
   <th>5</th>
   <th>6</th>
   <th>7</th>
   <th>8</th>
   <th>9</th>
   <th>10</th>
   <th>11</th>
   <th>12</th>
  </tr>
  <tr>
   <th>Number of DoFs</th>
   <td>2097152</td>
   <td>7077888</td>
   <td>16777216</td>
   <td>32768000</td>
   <td>7077888</td>
   <td>11239424</td>
   <td>16777216</td>
   <td>23887872</td>
   <td>32768000</td>
   <td>43614208</td>
   <td>7077888</td>
   <td>8998912</td>
  </tr>
  <tr>
   <th>Number of iterations</th>
   <td>13</td>
   <td>12</td>
   <td>12</td>
   <td>12</td>
   <td>13</td>
   <td>13</td>
   <td>15</td>
   <td>15</td>
   <td>17</td>
   <td>19</td>
   <td>18</td>
   <td>18</td>
  </tr>
  <tr>
   <th>Solver time [s]</th>
   <td>0.713</td>
   <td>2.150</td>
   <td>4.638</td>
   <td>8.803</td>
   <td>2.041</td>
   <td>3.295</td>
   <td>5.723</td>
   <td>8.306</td>
   <td>12.75</td>
   <td>19.25</td>
   <td>3.530</td>
   <td>4.814</td>
  </tr>
  <tr>
   <th>MDoFs/s</th>
   <td>2.94</td>
   <td>3.29</td>
   <td>3.62</td>
   <td>3.72</td>
   <td>3.47</td>
   <td>3.41</td>
   <td>2.93</td>
   <td>2.88</td>
   <td>2.57</td>
   <td>2.27</td>
   <td>2.01</td>
   <td>1.87</td>
  </tr>
</table>

We clearly see how the efficiency per DoF initially improves until it reaches
a maximum for the polynomial degree $k=4$. This effect is surprising, not only
because higher polynomial degrees often yield a vastly better solution, but
especially also when having matrix-based schemes in mind where the denser
coupling at higher degree leads to a monotonously decreasing throughput (and a
drastic one in 3D, with $k=4$ being more than ten times slower than
$k=1$!). For higher degrees, the throughput decreases a bit, which is both due
to an increase in the number of iterations (going from 12 at $k=2,3,4$ to 19
at $k=10$) and due to the $\mathcal O(k)$ complexity of operator
evaluation. Nonetheless, efficiency as the time to solution would be still
better for higher polynomial degrees because they have better convergence rates (at least
for problems as simple as this one): For $k=12$, we reach roundoff accuracy
already with 1 million DoFs (solver time less than a second), whereas for $k=8$
we need 24 million DoFs and 8 seconds. For $k=5$, the error is around
$10^{-9}$ with 57m DoFs and thus still far away from roundoff, despite taking 16
seconds.

Note that the above numbers are a bit pessimistic because they include the
time it takes the Chebyshev smoother to compute an eigenvalue estimate, which
is around 10 percent of the solver time. If the system is solved several times
(as e.g. common in fluid dynamics), this eigenvalue cost is only paid once and
faster times become available.

<h3>Evaluation of efficiency of ingredients</h3>

Finally, we take a look at some of the special ingredients presented in this
tutorial program, namely the FE_DGQHermite basis in particular and the
specification of MatrixFree::DataAccessOnFaces. In the following table, the
third row shows the optimized solver above, the fourth row shows the timings
with only the MatrixFree::DataAccessOnFaces set to `unspecified` rather than
the optimal `gradients`, and the last one with replacing FE_DGQHermite by the
basic FE_DGQ elements where both the MPI exchange are more expensive and the
operations done by FEFaceEvaluation::gather_evaluate() and
FEFaceEvaluation::integrate_scatter().

<table align="center" class="doxtable">
  <tr>
   <th>degree</th>
   <th>1</th>
   <th>2</th>
   <th>3</th>
   <th>4</th>
   <th>5</th>
   <th>6</th>
   <th>7</th>
   <th>8</th>
   <th>9</th>
   <th>10</th>
   <th>11</th>
   <th>12</th>
  </tr>
  <tr>
   <th>Number of DoFs</th>
   <td>2097152</td>
   <td>7077888</td>
   <td>16777216</td>
   <td>32768000</td>
   <td>7077888</td>
   <td>11239424</td>
   <td>16777216</td>
   <td>23887872</td>
   <td>32768000</td>
   <td>43614208</td>
   <td>7077888</td>
   <td>8998912</td>
  </tr>
  <tr>
   <th>Solver time optimized as in tutorial [s]</th>
   <td>0.713</td>
   <td>2.150</td>
   <td>4.638</td>
   <td>8.803</td>
   <td>2.041</td>
   <td>3.295</td>
   <td>5.723</td>
   <td>8.306</td>
   <td>12.75</td>
   <td>19.25</td>
   <td>3.530</td>
   <td>4.814</td>
  </tr>
  <tr>
   <th>Solver time MatrixFree::DataAccessOnFaces::unspecified [s]</th>
   <td>0.711</td>
   <td>2.151</td>
   <td>4.675</td>
   <td>8.968</td>
   <td>2.243</td>
   <td>3.655</td>
   <td>6.277</td>
   <td>9.082</td>
   <td>13.50</td>
   <td>20.05</td>
   <td>3.817</td>
   <td>5.178</td>
  </tr>
  <tr>
   <th>Solver time FE_DGQ [s]</th>
   <td>0.712</td>
   <td>2.041</td>
   <td>5.066</td>
   <td>9.335</td>
   <td>2.379</td>
   <td>3.802</td>
   <td>6.564</td>
   <td>9.714</td>
   <td>14.54</td>
   <td>22.76</td>
   <td>4.148</td>
   <td>5.857</td>
  </tr>
</table>

The data in the table shows that not using MatrixFree::DataAccessOnFaces
increases costs by around 10% for higher polynomial degrees. For lower
degrees, the difference is obviously less pronounced because the
volume-to-surface ratio is more beneficial and less data needs to be
exchanged. The difference is larger when looking at the matrix-vector product
only, rather than the full multigrid solver shown here, with around 20% worse
timings just because of the MPI communication.

For $k=1$ and $k=2$, the Hermite-like basis functions do obviously not really
pay off (indeed, for $k=1$ the polynomials are exactly the same as for FE_DGQ)
and the results are similar as with the FE_DGQ basis. However, for degrees
starting at three, we see an increasing advantage for FE_DGQHermite, showing
the effectiveness of these basis functions.

<h3>Possibilities for extension</h3>

As mentioned in the introduction, the fast diagonalization method as realized
here is tied to a Cartesian mesh with constant coefficients. When dealing with
meshes that contain deformed cells or with variable coefficients, it is common
to determine a nearby Cartesian mesh cell as an approximation. This can be
done with the class TensorProductMatrixSymmetricSumCollection. Here, one can
insert cell matrices similarly to the PreconditionBlockJacobi::initialize()
function of this tutorial program. The benefit of the collection class is that
cells on which the coefficient of the PDE has the same value can re-use the
same Laplacian matrix, which reduces the memory consumption for the inverse
matrices. As compared to the algorithm implemented in this tutorial program,
one would define the length scales as the distances between opposing
faces. For continuous elements, the code project <a
href=https://github.com/peterrum/dealii-dd-and-schwarz">Cache-optimized and
low-overhead implementations of multigrid smoothers for high-order FEM
computations</a> presents the computation for continuous elements. There is
currently no infrastructure in deal.II to automatically generate the 1D
matrices for discontinuous elements with SIP-DG discretization, as opposed to
continuous elements, where we provide
TensorProductMatrixCreator::create_laplace_tensor_product_matrix().

Another way of extending the program would be to include support for adaptive
meshes. While the classical approach of defining interface operations at edges
of different refinement level, as discussed in step-39, is one possibility,
for Poisson-type problems another option is typically more beneficial. Using
the class MGTransferGlobalCoarsening, which is explained in the step-75
tutorial program, one can deal with meshes of hanging nodes on all levels. An
algorithmic improvement can be obtained by combining the discontinuous
function space with the auxiliary continuous finite element space of the same
polynomial degree. This idea, introduced by Antonietti et al.
@cite antonietti2016uniform in 2016, allows making the multigrid convergence
independent of the penalty parameter. As demonstrated by Fehn et al.
@cite fehn2020hybrid, this also gives considerably lower iteration counts than
a multigrid solver directly working on levels with discontinuous function
spaces. The latter work also proposes p-multigrid techniques and combination
with algebraic multigrid coarse spaces as a means to efficiently solve Poisson
problems with high-order discontinuous Galerkin discretizations on complicated
geometries, representing the current state-of-the-art for simple Poisson-type
problems. The class MGTransferGlobalCoarsening provides features for each of
these three coarsening variants, the discontinuous-continuous auxiliary
function concept, p-multigrid, and traditional h-multigrid. The main
ingredient is to define an appropriate MGTwoLevelTransfer object and call
MGTwoLevelTransfer::reinit_geometric_transfer() or
MGTwoLevelTransfer::reinit_polynomial_transfer(), respectively.
