###############
Counterexamples
###############

Counterexamples are the translation of models generated by SMT solvers, when a
formula can be shown to be satisfiable, into an execution trace at the SPARK
level for the user.
Such models are checked and classified before being displayed to the user,
using runtime assertion checking (RAC) split into 2 parts: (a) a giant-step
RAC done on the Why3 side and (b) a small-step RAC done on the SPARK side.
The results of these two RAC executions allow to classify and validate the
counterexamples given by the SMT solvers.

Roughly, it works in the following way.

- During :ref:`Translation to Why3`, annotations (attributes) are added to the
  generated code together with locations, such as ``model_trace`` attributes.
- Then, during :ref:`Weakest Precondition`, the attributes are kept and still
  associated to appropriate variables.
- Then, a specific transformation for counterexamples introduces new
  constants for all constants appearing inside the verification condition. The
  location of these constants is the one of the VC. This allows to have more
  counterexample values on the VC line (because no variable is defined at an
  overflow check for example).
- Then, while the printer creates the task for the SMT solver, data are
  collected to trace back elements in the output
  of the SMT solver to original elements (in the Why3 file).
- When the SMT solver answers with a model, this model is parsed and processed
  to (a) recover values of original program variables and (b) execute the
  giant-step RAC using these values.
- Then, GNATWhy3 performs some post-processing to cleanup the JSON model
  (mostly removing useless elements) before sending it together with the
  result of the giant-step RAC in JSON format to gnat2why.
- Gnat2why parses the JSON input to import counterexample values and
  giant-step RAC execution results. Then, the small-step RAC is executed by
  gnat2why before (a) classifying the counterexample and (b) if the
  classification is good displaying it.

The following sections give more details of each layer (provers layer, parsing
and processing of models, checking and classifying of counterexamples).
Additional sections then gives information about debugging, testing and
getting statistics on counterexamples.

Provers layer
=============

From Why3, the SMT solvers are called by transforming the task into an SMTv2
file by replacement of syntax and change the goal into its negation. That way,
if the prover is sure that the property is unsatisfiable then we are sure that
indeed the task is valid (meaning it was proved).
So, when provers are able to satisfy the task, they answer sat. In this case,
they found a model for the task (i.e. an instantiation of all constants that is
consistent with the task and does not negate the goal). In this case, we can
ask them for this task using:

.. TODO ??? Add smt2 language in pygments using the .el file availables online
.. code-block:: Ada

    (get-model)

After a sat/unknown answer from a ``(check-sat)`` command, when given a
``(get-model)`` command, the prover produce an assignment for every variable (a
model).
The way things are expressed in SMTv2 is defined in http://smtlib.cs.uiowa.edu/ .
This looks like the following example (S-expr style and prefix operators).
From task:

.. code-block:: Ada

    (declare-const a Int)

    (declare-const b Int)

    (assert (= b 2))

    (assert (= a (- 2 )))

    (assert
      (in_range (* a b)))
    (check-sat)
    (get-model)

yields the following model:

.. code-block:: Ada

     sat
     (model
     (define-fun a () Int (- 2))
     (define-fun b () Int 2)
     )


Parsing and processing of prover models
=======================================

The parsing and processing of prover models is done in 3 steps.

- **First step:** parse the SMT solver model to recover values of original
  program variables. This is done at the Why3 level, the input is the
  SMT solver output (a string) and the output is a ``Model_parser.model``.
- **Second step:** post-process the model to filter out useless elements.
  This is done by GNATWhy3, the input is the ``Model_parser.model`` and the
  output is a ``Model_parser.model`` in JSON format.
- **Third step:** process the JSON output of GNATWhy3 to import values of the
  counterexample model in order to execute the small-step RAC and possibly
  display the counterexample. This is done in gnat2why, the input is a
  ``Model_parser.model`` in JSON format and the output is a list of
  counterexample values (``VC_Kinds.Cntexample_Elt_Lists``).

The following sections describe these 3 steps in more details.

First step: Why3
----------------

The entry points for parsing and processing of prover models in Why3 is in the
module :download:`Model_parser <../../why3/src/core/model_parser.ml>`.
The model is parsed using function ``model_parser`` by first applying a *raw*
model parser, which is implemented in the function ``parse`` of module
:download:`Smtv2_model_parser <../../why3/src/driver/smtv2_model_parser.ml>`
for SMTv2 prover models, and then processing the resulting preliminary
model elements.

First, the prover model is cut from the prover output using a regular
expression (``Smtv2_model_parser.get_model_string``).

The parsing of SMTv2 models is done by 3 consecutive modules.

- ``Smtv2_model_parser.FromStringToSexp`` parses the prover output
  as S-expressions. The definition of the S-expression and the parser are
  implemented in module :download:`Sexp <../../why3/src/driver/sexp.mli>`.
- ``Smtv2_model_parser.FromSexpToModel`` converts S-expressions to a mapping
  of variable names to SMTv2 definitions from module.
  :download:`Smtv2_model_defs <../../why3/src/driver/smtv2_model_defs.mli>`.
- ``Smtv2_model_parser.FromModelToTerm`` converts SMTv2 definitions to a list
  of preliminary ``Model_parser.model_element`` values.

Each ``Model_parser.model_element`` represents a counterexample value for
a single source code element, and contains the following fields:

- ``me_name``, the name of the model element, i.e. either the value of the
  ``model_trace`` attribute, or the name of original logical function symbol
  in the source code if such an attribute is not present;
- ``me_kind``, the kind of the model element defined by
  ``Model_parser.model_element_kind``;
- ``me_value``, the counterexample value of type ``Term.term``;
- ``me_concrete_value``, the same value represented in concrete syntax, used
  for pretty printing and JSON printing, where e.g. epsilon terms representing
  records or projections are recognized as such;
- ``me_location``, the location of the corresponding element in the source code;
- ``me_attrs``, the attributes associated to this element;
- ``me_lsymbol``, the logical symbol corresponding to the original element
  in the source code.

The preliminary model elements obtained from the raw model parser are then
further processed in function ``Model_parser.model_parser``. The bulk of the
processing is done in function ``Model_parser.build_model_rec``, which builds
a ``Model_parser.model_files``, i.e. a list of model elements sorted by
filename and location, by adding the elements at all relevant locations.

Second step: GNATWhy3
---------------------

In SPARK, the model is additionally processed using
``Gnat_counterexample.post_clean``, which prepares the values in the model
elements for display in ``gnatprove``:

- model elements that do not have a name recognized by SPARK are removed
  (i.e. only model elements with names corresponding to a model trace
  containing a GNAT identifier are kept);
- the values of unconstrained arrays and split discriminants are modified.

Finally, model elements are transformed into JSON format using
``Model_parser.json_model``.

Third step: gnat2why
--------------------

First, the counterexample model in JSON format is parsed to typed values:
the entry point is the function ``VC_Kinds.From_JSON``.

Then, these typed counterexample values may be used:

- by the small-step RAC through ``CE_parsing.Get_Counterexample_Value``,
- by the pretty printer through ``CE_parsing.Parse_Counterexample_Line``.

Checking and classifying counterexamples
========================================

GNATprove checks counterexample models and classifies proof failures: the
subprogram from where the VC originates is executed using two types of
runtime assertion checking (RAC), small-step RAC and giant-step RAC.

The giant-step RAC is requested from gnatwhy3. A giant-step RAC corresponds
to a normal RAC execution of the WhyML program, with the difference that
function calls and loops are executed by a single "giant step", where return
values of function calls and values of modified variables are obtained directly
from the model (i.e. the model is used as an *oracle* during the giant-step RAC
execution).
The result of the giant-step RAC is then retrieved by gnat2why
(see ``Gnat2Why.Error_Messages.Handle_Result``).

The small-step RAC is implemented in ``CE_RAC.RAC_Execute`` and corresponds to
the execution of a binary compiled with assertions enabled.
In the small-step RAC execution, the counterexample provides the values for the
arguments of the subprogram and for values of global variables.

Note that the small-step RAC is implemented in gnat2why, based on the original
Ada program, because the generated Why3 program is not executable. Only
the giant-step RAC execution is based on the WhyML program generated by
gnat2why.

Both RAC executions result in a ``CE_RAC.Result``, which contains the kind of
the result (normal termination, assertion failure, incomplete execution, failed
assumption, or not executed) and additionally information about the relevant VC
for assertion failures.

The results of the small- and giant-step RAC executions are combined into a
*verdict* on the counterexample, using the following table.

===================== ============================== ====================== ================== ==========
\                     Giant-step RAC result
--------------------- -----------------------------------------------------------------------------------
Small-step RAC result Failure                        Normal                 Stuck              Incomplete
===================== ============================== ====================== ================== ==========
Failure at VC         Non-conformity
--------------------- -----------------------------------------------------------------------------------
Failure elsewhere     Bad counterexample
--------------------- -----------------------------------------------------------------------------------
Stuck                 Invalid assumption
--------------------- -----------------------------------------------------------------------------------
Normal                Subcontract-weakness           Bad counterexample     Bad counterexample Incomplete
Incomplete            Non-conformity or sub-weakness Incomplete             Bad counterexample Incomplete
===================== ============================== ====================== ================== ==========

This approach to checking and classifying counterexamples is described in the
article *Explaining Counterexamples with Giant-Step Assertion Checking*
(https://eptcs.web.cse.unsw.edu.au/paper.cgi?FIDE2021.10), and more details are
available in the technical report *Giant-step Semantics for the Categorisation
of Counterexamples* (https://hal.inria.fr/hal-03213438).

The verdict of the counterexample checking from the table above is represented
in the SPARK message about the proof failure as follows:

============================== ============== ======== ===========================================================
Verdict                        Counterexample Priority Fix
============================== ============== ======== ===========================================================
Incomplete                     discard        medium
------------------------------ -------------- -------- -----------------------------------------------------------
Bad CE                         discard        medium
------------------------------ -------------- -------- -----------------------------------------------------------
Non-conformity                 show           high
------------------------------ -------------- -------- -----------------------------------------------------------
Subcontract weakness           show           medium   "add or complete related loop invariants or postconditions"
------------------------------ -------------- -------- -----------------------------------------------------------
Non-conformity or sub-weakness show           medium
============================== ============== ======== ===========================================================

Testing
=======

The normal RAC engine is tested in ``TC02-027__rac``, which compares the
behaviour of the RAC engine on the main procedure of a project with the
behaviour of a program compiled with assertions enabled:

1. Compile the project using options ``-gnata -gnatw.A -gnat2022``

2. Execute the compiled program and capture the output

3. Execute the small-step RAC engine with ``gnatprove`` using option
   ``--debug-exec-rac`` (which runs the RAC engine on the main procedure and
   exits), and capture the output.

4. Compare the output of the compiled program and of the RAC engine

This is implemented by the ``RACTestDriver`` in
:download:`run-tests <../../testsuite/gnatprove/run-tests>`.

Debugging
=========

See :ref:`gnatwhy3_debug` for general debugging tips with GNATprove and
GNATWhy3.

Below are presented some debugging tips when working specifically with
counterexamples.

The results of the two RAC executions are printed by GNATprove when using
option ``--debug``. When setting the environment variable
``GNAT2WHY_RAC_TRACE=on`` the program nodes are printed as they are executed
during the small-step RAC.

To collect debugging information about counterexample models parsing
and RAC executions in GNATWhy3, the following Why3 debug flags may be
useful:
``smtv2_parser,model_parser, model_parser,
rac-check-term-result, rac-check-term-sat, rac-values,
check-ce-rac-results, check-ce-categorization``.
They can be activated either using the GNATWhy3's option ``--debug-why3``
or the ``WHY3DEBUG=<flags>`` environment variable, where
``flags`` is a comma-separated list of Why3 debug flags.

It is sometimes useful to launch Why3 (without the overcoat of GNATWhy3)
on a generated .mlw file.

- For a given test of the testsuite, first use the function ``expandtest``
  given in the SPARK wiki:
  https://gitlab.adacore-it.com/eng/spark/spark2014/-/wikis/Testing#running-tests-by-hand
- Then, go in the ``gnatprove`` directory that has been created, where
  generated .mlw files are present.
  In order to launch Why3, it is necessary to use the generated configuration
  file ``why3.conf`` (at the root of the ``gnatprove`` directory) and to
  use the SPARK libraries instead of the standard libraries.
  This can be achieved with the following Why3 command.

.. code-block::

  why3 prove -C why3.conf -L ~/spark2014/install/share/spark/theories -L ~/spark2014/install/libexec/spark/share/why3/theories/ --no-stdlib -P cvc5_ce -a split_vc --check-ce --rac-prover=cvc5_ce --debug=<debug_flags> <file.mlw>


Computing statistics about counterexamples
==========================================

Using the ``--debug`` flag of GNATprove, additional information (the
final verdict and its optional reason, the results of the small- and
giant-step RAC execution and their optional reasons and the location)
is printed about the checking of each counterexample with the prefix
``VERDICT``:

.. code-block::

  VERDICT: <final-verdict>, Reason: <final-verdict-reason> | Small-step: <small-step-result>, Reason: <small-step-reason> | Giant-step: <giant-step-result>, Reason: <giant-step-reason>

When running the testsuite on a mailserver, it is possible to force
the printing of such verdicts by patching the function
``Gnat2Why.Error_Messages.Handle_Result`` as following.

.. code-block:: Ada

  if not Rec.Result
    and then True --  CE_RAC.Do_RAC_Info
  then
    Write_Str
      ("----------"
        & LF & "VERDICT: "

Then, the Python script
:download:`compute_ce_stats.py <../../testsuite/gnatprove/compute_ce_stats.py>`
can be used to parse the mailserver's ``.out`` files and extract statistics
about counterexample classification.
This script generates ``.csv`` files, which can be further processed using
for example the spreadsheet in the ProofInUse GitLab repository
for collaboration with AdaCore
(https://gitlab.inria.fr/proofinuse/adacore/-/blob/master/ce-check/stats_CE_SPARK_testsuite.ods).
