DFASYN(5)

NAME
dfasyn

SYNOPSYS
This page describes the format of the input-file for the dfasyn deterministic finite automaton generator.

DESCRIPTION

  Overview
Reserved words may be given in all-lowercase, all-uppercase, initial capitals, or 'WikiWord' format (e.g. endblock may be given as endblock, Endblock, EndBlock or ENDBLOCK.

  Block declaration
A block declaration is used to group together a set of state declarations. Blocks are useful if there are blocks of states and their interconnections that occur more than once in the NFA. In this case it is useful to declare a block, allowing that block to be instantiated more than once elsewhere in the input file. Since state declarations are only allowed inside blocks, there must be at least one block declaration in any useful input file. The syntax of a block declaration is
    block block-name {
        [ instance-declarations ]
        [ state-declarations ]
    }

  State declarations
A state declaration gives rise to a state in the input NFA. The syntax of a state declaration is
    state state-name [ entry entry-name ]
        [ transitions ]
States are implicitly terminated by the beginning of another type of construct. entry entry-name (if present) defines the name of an entry point into the scanner. In the resulting C-code, a symbol called entry-name will be declared. Its value will be the DFA state number of the state containing just this NFA state (plus its epsilon closure.) This allows for multiple scanners to be generated from the same input file. For example, if one scanner is the same as another but with some extra text that must match at the beginning, two different entry states can be declared to represent this. dfasyn will be able to common-up all of the common part of the DFA's transition tables. If there are no entry directives anywhere in the input file, dfasyn defaults to the last mentioned state in the last block being the entry state. transitions is a whitespace-separated sequence of zero or more transitions. These define which of the automaton's input symbols cause a transition from this state to which other states. The same state may be declared more than once inside its block. In this case, the transitions given in the second declaration will be merged with those given in the first, as though all the transitions had been given in the first place.

  Instance declarations
A block may be instantiated inside another block. This is useful if there is a block of states with their transitions that occurs in more than once place within the NFA. The syntax for an instance declaration is
    instance-name : block-name
where instance-name is the name of the new instance, and block-name is the name of the block that is being instantiated. This block must have been declared earlier in the input file. For one thing, this prevents mutually recursive definitions. When such an instance has been created, the states inside it may be referred to within the enclosing block by prefixing their names with the instance-name followed by a period.

  Transitions
A state-to-state transition is specified as follows.
    transition -> destinations
destinations is a comma-separated list of one or more fully-qualified state names. These are the states to which the NFA moves if the transition is matched next in the input. The destination state names are allowed to be forward-references; just the name is stored during parsing, and a second pass later is used to resolve all the names. There is no need for a named destination to actually be declared with another state definition; a state just comes into being if it is named at all. A transition defines the inputs that are required to cause the scanner to move from one state to another. A transition is a semicolon-separated list of one or more stimuli. (If there is only one stimulus, no semicolon is required.) The transition matches as a whole if the stimuli are matched individually in sequential order from left to right.

  Transitions to a tag
Where a transition leads to a tagged exit state, the following syntax is used:
    transition = tags
where tags is a comma-separated list of one or more tag names. Thus a construction like
    state foo XXX = TAG1
indicates that matching the token XXX leads to a state in which TAG1 applies.

  Stimuli
A stimulus is a pipe-separated list of alternatives. Each alternative may be one of the following:

*   the name of a token

*   a character class

*   the name of an abbreviation

*   an empty string (which gives rise to an epsilon transition )

*   an inline block instance

  Input symbols
Input symbols can be defined in two ways. The first is to use ASCII characters directly. The second is to define a set of tokens and use a front-end module to generate these based on the actual input. You can actually mix both types of input symbol. For example, you might wish to use ASCII characters mostly, but detect "end-of-file" as an explicit symbol.

  ASCII input and character classes.
Single ASCII characters can be given in double-quotes. Sets of ASCII characters can be given in square brackets, similar to shell globbing. Character classes can be negated and differenced.

[a] The character "a".

[abe-h]
    Any of the characters "a", "b", "e", "f", "g", "h".

~[abc]
    Any of the 253 characters excluding "a", "b" and "c"; a negated character class.

[^abc]
    Ditto - another way of expressing a negated character class.

[a-z]~[c]
    Equivalent to [abd-z].

The following special cases are available within the square brackets:

\-  A hyphen. Normally the hyphen is used as a range separator. To get a literal hyphen, it must be escaped by a back-slash.

\]  A closing square bracket. The escaping is required to prevent it being handled as the end of the character class.

\\  A literal backslash.

\^  A literal "^".

\n  The same character as "\n" in C.

\r  The same character as "\r" in C.

\f  The same character as "\f" in C.

\t  The same character as "\t" in C.

^A  Generate a control character, in this case ASCII character 1. Defined for ^@ through to ^Z.

\xa9
    The ASCII character with hex value 0xa9. Upper or lower case hex may be used.

\234
    The ASCII character with octal value 0234.

  Tokens
To define non-ASCII inputs, at least one tokens directive must be used. The syntax is

tokens list-of-tokens

where list-of-tokens is a space-separated list of token names. Each token name is a string that will be acceptable as a C macro name when prefixed by the current prefix string plus an underscore. If more than one tokens line appears in the input file, the 2nd and subsequent lines are treated as though their entries were concatenated with the 1st line.

  Abbreviations
An abbreviation provides a convenient way to define a shorthand name for a frequently used stimulus. The syntax is
    abbrev abbrev-name = stimulus
For example:
    abbrev FOO = [aeiouAEIOU] | A_TOKEN | <xyzzy:in->out>

  Inline block instances
A stimulus may take the form of a block instance. This is a convenient shorthand when a complex sequence of input tokens needs to be matched as part of a transition. The syntax of an inline block instance is
    <block_name:entry_state->exit_state>
As an example, given a block double_a defined like this
    block double_a
      state in A -> out
    endblock
the following construction
    block x
      state foo <double_a:in->out> ; B ; <double_a:in->out> -> bar
    endblock
is equivalent to
    block x
      aa1 : double_a
      aa2 : double_a
      state foo -> aa1.in
      state aa1.out
        B -> aa2.in
      state aa2.out -> bar
    endblock
Note that in the second example, where explicit instances have been created, they must have unique names. In the first case, dfasyn will create the two anonymous instances automatically and handle all the plumbing to connect up the in and out states. Note there is no requirement for the states to be named 'in' and 'out'; that is merely a convention. An instanced block may have multiple inputs, with different inputs being used in different instantiations of the block, for example.

  Tags and attributes
Tags are associated with the NFA states in the input. An NFA state may have an arbitrary number of tags associated with it, through what amounts to a list of strings. Attributes are attached to the DFA states in the output. In the generated C-file, the attributes are expressed in terms of an array which is indexed by the DFA state number and whose elements are the attribute values applying to the states. Once the DFA has been generated, dfasyn knows the NFA states that apply in each DFA state. From this, the tags associated with a DFA state are given by the union of all the tags appylying in all the NFA states that apply in that DFA state. The input file defines how a set of tags applying in a DFA state is to be reduced to a single attribute value. A boolean expression language is provided for this purpose. Although the default is to generate a single attribute table, dfasyn can generate arbitrarily many tables if required. This is achieved by using attribute groups. The NFA tag namespace is shared across all such groups. The group syntax is as follows:
    group groupname { declaration [ , declaration ... ] }
where each declaration is one of the following:
    attr attribute-name [ , attribute-name ... ]
    attr attribute-name : expression
    early attr attribute-name [ , attribute-name ... ]
    early attr attribute-name : expression
In the form with no expression, each attribute-name has an implicit expression consisting of just the tag with the same name as itself. expression is defined in the section Expressions later. The short form
    attr foo
is short for
    attr foo : foo
i.e. it allows an attribute to be defined which has the same name as a tag and which is active in the cases where precisely that tag is active. If an attribute is prefixed by early, it means that the C-code you provide to drive the DFA is going to stop scanning once this state attribute is detected. For example, this would apply if you were coding a "shortest match" scanner. dfasyn will prune all the transitions away from any DFA state having such an attribute. This may lead to greater opportunities for dfasyn to compress the DFA. A default attribute must be declared. This is used to fill all the entries in the attribute array for DFA states that end up with no explicit attribute defined. (It is also used in determining where the DFA may be optimised to remove "dead states".) The syntax is
    defattr default-attribute-string
Finally, the C-type of the attribute must be declared. This becomes the base type of the array indexed by the DFA state number. The syntax is
    type attribute-type-name
It is illegal for more than one attribute in a particular attribute group to be active in a DFA state. If this situation occurs, it indicates that the expression logic for that group is defective.

  Expressions
An expression defines an attribute in terms of a boolean relationship between one more more tags. An expression may be any one of the following:
    expression & expression
    expression | expression
    expression ^ expression
    expression ? expression : expression
    (expression)
    ~expression
    !expression
    tag-name
Note that ~expression and !expression both mean the negation of expression. The operator precedence is what would be expected for a C-programmer.

Prefix specification
The prefix used in the generated C-file can optionally be set in the input file using the following syntax:
    prefix prefix-string
where prefix-string_ (i.e. the specific string followed by an underscore) will occur at the start of each symbol name in the generated C-file. If the prefix has been set via the command line using -p, the prefix line in the input file will be ignored and a warning given.

THE GENERATED C-FILE
The generated file exports the following symbols that can be used by the calling program:

short
    prefix_char2tok [256];
    If character classes have been used, this table maps from ASCII values to the internal tokens numbers used by the generated DFA. This array will be defined in the generated C-file. If a header file is being generated, it will be declared in there also.

#define
    prefix_TOKEN numeric_value
    If a directive has been used, each such token will be assigned a number. These assignments are emitted by as a series of #define lines. Each token name from the input file will have the prefix and an underscore prepended to form the name of the symbol in the #define. If a header file is being generated (-ho), these definitions are placed in the header file. Otherwise, they are placed in the main output C-file.

int prefix_next_state (int current_state, int next_state);
    This is the prototype for the next state function which the calling program must invoke. If no -I option has been used, this function will be defined in the generated C-file. If a header file is being generated, it will be prototyped in there also. If -I has been used, the function will be defined in the header file.

int prefix_entry-name
    If the entrystruct directive has not been used, this format is used to define the DFA state numbers for the defined entry points. The calling program uses these values to set the current_state at the start of the scanning process, depending on which entry point is being used. If there is more than one entry, there will be more than one such line.

struct
    entrystruct-type { ... } entrystruct-var
    If the entrystruct directive has been used, the DFA state numbers for the entry points are declared as elements of a struct. The struct member names are identical to the entry names used in the dfasyn input file. The declaration of the struct variable containing the state numbers will be in the generated C-file. If a header file is being generated (-ho), the definition of the struct type will be in there. Otherwise, it will be in the C-file also.

attr-type
    prefix_attr [#DFA-states]
    This defines the attributes for each of the DFA states in the default attribute group. If no type attr-type declaration was in the input file, the default of short will be used. If other attribute groups are defined, there will be a similar array for each one:

group-attr-type
    prefix_group-name [#DFA-states]
    For the attribute group declared with group group-name in the input file, this defines the attribute of each of the DFA states in that group.

TEXT PASSTHROUGH
To pass a block of literal text through to the output file without interpretation, enclose it in %{ ... %} like this:
    %{
    #include "foo.h"
    %}
The opening and closing patterns must be on lines on their own (trailing whitespace is allowed).

SEE ALSO
dfasyn(1)