
= Highlight Languages Test Suite
Tristano Ajmone
2019-10-31
:lang: en
// Rev Info
:revremark: Highlight 3.53
:revnumber!:
// TOC Settings:
:toclevels: 5
// GitLab setting to show TOC after Preamble
:toc: macro
// TOC ... HTML Backend Hack to show TOC on the Left
ifdef::backend-html5[]
:toc: left
endif::[]
// TOC ... GitHub Hack to show TOC after Preamble (required)
ifdef::env-github[]
:toc: macro
endif::[]
// Sections Numbering:
:sectnums!:
// Cross References:
:idprefix:
:xrefstyle: short
:section-refsig: Sect.
// Misc Settings:
:experimental:
:icons: font
:linkattrs:
:reproducible:
:sectanchors:
// GitHub Settings for Admonitions Icons:
ifdef::env-github[]
:caution-caption: :fire:
:important-caption: :heavy_exclamation_mark:
:note-caption: :information_source:
:tip-caption: :bulb:
:warning-caption: :warning:
endif::[]

// =====================================
// Custom Attributes for Reference Links
// =====================================
// Project Files:
:build-theme_sh: pass:q[link:./build-theme.sh[`build-theme.sh`^,title="View source file"]]
:LICENSE: pass:q[link:LICENSE[`LICENSE`^]]
:MIT_license: pass:q[link:LICENSE[MIT license^]]
:test-theme_scss: pass:q[link:./test-theme.scss[`test-theme.scss`^,title="View source file"]]
:test-theme_htm: pass:q[link:./test-theme.htm[`test-theme.htm`^,title="Read document"]]
// External References:
:README_TESTCASES: pass:q[link:https://gitlab.com/saalen/highlight/blob/master/README_TESTCASES.adoc[`README_TESTCASES.adoc`^]]
// :xxx: pass:q[link:xxx[`xxx`^]]

// *****************************************************************************
// *                                                                           *
// *                            Document Preamble                              *
// *                                                                           *
// *****************************************************************************

// =============================================================================
//                              repository Badges
// =============================================================================
// Badge: MIT License
link:#license[image:https://img.shields.io/badge/license-MIT-00b5da[MIT License]]
// Badge: Travis CI (GitHub clone only)
ifdef::env-github[]
{nbsp}
image:https://travis-ci.org/tajmone/highlight-test-suite.svg?branch=master[Travis CI^,title="Travis CI build status"]
endif::[]

A collection of syntax tests for Highlight's languages definitions.

* https://gitlab.com/tajmone/highlight-test-suite[gitlab.com/tajmone/highlight-test-suite,title="Visit main project repository on GitLab"] (main repo)
* https://github.com/tajmone/highlight-test-suite[github.com/tajmone/highlight-test-suite,title="Visit cloned project repository on GitHub"] (clone)

// >>> GitLab/GitHub hacks to ensure TOC is shown after Preamble: >>>>>>>>>>>>>>
ifndef::backend-html5[]
'''
toc::[]
'''
endif::[]
ifdef::env-github[]
'''
toc::[]
'''
endif::[]
// <<< GitHub/GitLab hacks <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<



== About This Project

Highlight v3.45 introduced a new feature that allows to test if a source file is being syntax highlighted as expected.
The goal of this project is to create a collection of tests for all the languages supported by Highlight, in order to:

1. Uncover and annotate edge cases that are not handled correctly by a landDef.
2. Detected breaking changes while editing a landDef.
3. Test a langDef against new features of a language when the latter is updated.

While the repository is conceived to test only against the syntax definitions that ship with the official Highlight package, developers can also use this project to test syntax developement in local branches before submitting new/updated langDefs to Highlight repository.
This offers a convenient work approach, especially for language definitions which fail some tests and need updating, where the developer can run the existing test files against his/her tweaked langDef (using the `--config-file=` option inside a customized test script).
Check the project branches to see some examples of how the repository is being used to update existing langDefs and work on upcoming new syntaxes.

== Project Structure

Each language should be tested in a folder of its own, baring the same name as its langDef file.

How tests should be conducted for any given language might depend on the nature of the language at hand, especially if the language supports nested languages.

As a general rule, splitting tests into individual files should simplify tracking failures; so it's preferable to create many small individual test files targetting specific features of the language, and give each file a self-explanatory filename.

Each language folder should contain a `test.sh` script that executes all the tests for that language and redirects failed tests errors (STDERR) to the `errors.log` file.
Here is an example `test.sh` script from the `purebasic/` folder:

[source,bash]
---------------------------
#!/bin/bash
rm ./errors.log >/dev/null 2>&1
highlight\
  --isolate\ <1>
  --batch-recursive=*.pb\ <2>
  --validate-input\
  --replace-tabs=2\ <3>
  --config-file=../vanilla.theme\   <4>
  --style-infile=../test-theme.css\ <5>
  --font=consolas\ <6>
  --font-size=12\  <7>
  2>>errors.log
---------------------------

<1> Use `--isolate` (optional) to be able to test whitespace between contiguos tokens of same type. Introduced in Highlight v3.51.
<2> Use the base file extension of your language in place of `.pb`.
<3> Set tabs width according to your language requirements.
<4> Since Highlight demands a base theme to build the CSS, we provide a plain vanilla theme, with all elements set to white on a black canvas, no italics nor bold.
Its CSS will be overriden by the stylesheet imported on the following line.
<5> This is a custom CSS theme providing a color scheme that simplifies tracking the highlighted elements in the HTML files generated by the tests scripts -- see {test-theme_htm} for more info. +
(CSS built from the Sass source {test-theme_scss}, via {build-theme_sh}.)
<6> Control over font type and size are left to user discretion, via script options (the CSS theme doesn't enforce any font).
<7> Font size is specified in _pt_ units (12pt -> 16px).

Using a standard convention in tests script will allow them to be used for automatic testing via Continuos Integration at some point in the future, when the project is mature.
Bash scripts are usable on all platforms (Git for Windows ships with a Bash).

A `README` file in the folder should provide some information on the language being tested, the tests being carried out, and annotate tests failures by providing meaningful notes and examples that might help improving the langDef to cover the language features which it failed.
The `README` file can be in any of the markup formats supported by GitLab's documents preview (AsciiDoc, Markdown, etc.).


== Contributing


LangDefs creators and/or maintainers are invited to add to this project the tests for their syntaxes.

If you're aware of any problems with a language definition, feel free to add a test file to demonstrate how the language fails to highlight the syntax as expected, adding some notes on the test and the expected results.
If that language is not present in the project, you can create a new folder for it, even if it's only to provide a single failing test example.

Adding failure tests is an important remainder of the fixes that should be made to a langDef, so anyone is invited to submit tests that draw attention to uncovered edge cases, regardless of wether he/she is intending to fix the langDef or not.

[IMPORTANT]
================================================================================
This project is MIT licensed.
Any contents submitted to the project will implicitly fall under the same MIT license, unless otherwise specified.
Don't submit any contents that are incompatible with the {MIT_license}.
================================================================================


== Highlight Syntax Tests

Highlight will automatically carry out syntax tests when the input filename starts with `syntax_test_`.

The test system is very simple to use as it exploits columns markers and state indicators in comment lines within the source file to indicate the expected highlight state for elements in the previous non-comment line of code:


[source,C]
--------------------------------------------------------------------------------
#include "myheader"
//< ppc  ^^^^^^^^^^ pps <1> <2>
--------------------------------------------------------------------------------

In the example above, the comment line carries out two tests on the previous line of code:

<1> that `#include` is being highlighted as  _preprocessor_ (ppc), and
<2> `"myheader"` as _preprocessor string_ (pps).

The `^` indicators point at the tested string sections of the previous line, `<` points at the character directly above the comment delimiter's first character (`#` in the example above).

Syntax testing is fully documented in the {README_TESTCASES} file that ships with Highlight.


== License

This project is released under {MIT_license}.

--------------------------------------------------------------------------------
MIT License

Copyright (c) 2018 Tristano Ajmone <tajmone@gmail.com>
https://gitlab.com/tajmone/highlight-test-suite

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
--------------------------------------------------------------------------------


// EOF //
