Next: Introduction, Previous: (dir), Up: (dir)
• Introduction: | ||
• Parsing: | ||
• Translation: | ||
• Administrative: | ||
• Todos: | ||
• References: |
WARNING: This manual is currently in a very immature state.
A LALR(1) parser is a pushdown automata for parsing computer languages. In this tool the automata, along with its auxiliary parameters (e.g., actions), is called a machine. The grammar is called the specification. The program that processes, driven by the machine, input token to generate a final output, or error, is the parser.
A simplest way to introduce working with nyacc
is to work through
an example. Consider the following contents of the file calc.scm.
(use-modules (nyacc lalr)) (use-modules (nyacc lex)) (define calc-spec (lalr-spec (prec< (left "+" "-") (left "*" "/")) (start expr) (grammar (expr (expr "+" expr ($$ (+ $1 $3))) (expr "-" expr ($$ (- $1 $3))) (expr "*" expr ($$ (* $1 $3))) (expr "/" expr ($$ (/ $1 $3))) ('$fx ($$ (string->number $1))))))) (define calc-mach (make-lalr-machine calc-spec)) (define parse-expr (let ((gen-lexer (make-lexer-generator (assq-ref calc-mach 'mtab))) (calc-parser (make-lalr-parser calc-mach))) (lambda () (calc-parser (gen-lexer))))) (define res (with-input-from-string "1 + 4 / 2 * 3 - 5" parse-expr)) (simple-format #t "expect 2; get ~S\n" res) ;; expect: 2
Here is an explanation of the code:
use-modules
syntax.
lalr-spec
syntax is used to generate a (canonical)
specification from the grammar and options. The syntax is imported
from the module (nyacc lalr)
.
prec<
directive indicates that
the tokens appearing in the sequence of associativity directives
should be interpreted in increasing order of precedence. The
associativity statements left
indicate that the tokens have left
associativity. So, in this grammar +
, -
, *
, and
/
are left associative, *
and /
have equal
precedence, +
and -
have equal precedence, but *
and /
have higher precedence than +
and -
.
(Note: this syntax may change in the future.)
start
directive indicates which left-hand symbol in the
grammar is the starting symbol for the grammar.
grammar
directive is used to specify the production rules.
In the example above one left-hand side is associated with multiple
right hand sides. But this is not required.
"+"
), scheme character syntax (e.g., #\+
), or
quoted identifiers (e.g., '+
). There is no syntax to declare
tokens.
'$fx
indicates an unsigned integer. The
lexical analyzer tools will emit this token when an integer is
detected in the input.
function
to indicate a non-terminal
and "function"
to indicate a terminal. The reader will signal
an error when this condition is detected.
$$
form is used to
specify an action associated with the rule. Ordinarily, the action
appears as the last element of a right-hand side, but mid-rule
actions are possible (see Section TBD).
lalr-spec
is an associative array so you can
peek at the internals using standard Scheme procedures.
make-lalr-machine
.
This routine does the bulk of the processing to produce an LALR(1)
automata.
(gen-lexer (make-lexer-generator (assq-ref calc-mach 'mtab)))
We build a generator because a lexical analyzer may require state
(e.g., line number, mode). The generator is constructed from the
match table provided by the machine. The procedure
make-lexer-generator
is imported from the module (nyacc
lex)
. Optional arguments to make-lexer-generator
allow the
user to specify how identifiers, comments, numbers, etc are read in.
(calc-parser (make-lalr-parser calc-mach)))
This code generates a parser (procedure) from the machine and the match table. The match table is the handshake between the lexical analyzer and the parser for encoding tokens. In this example the match table is symbol based, but there is an option to hash these symbols into integers. See Section TBD.
(lambda () (calc-parser (gen-lexer)))))
Note that parse-expr
is a thunk: a procedure of no arguments.
(current-input-port)
so we set up the environment
using with-input-from-string
. See the Input/Ouput section of
the Guile Reference Manual for more information.
(define res (with-input-from-string "1 + 4 / 2 * 3 - 5" parse-expr))
If we execute the example file above we should get the following:
$ guile calc.scm expect 2; get 2 $
In some parser generators one declares terminals in the grammar file
and the generator will provide an include file providing the list of
terminals along with the associated “hash codes”. In NYACC the
terminals are detected in the grammar as non-identifiers: strings
(e.g., "for"
), symbols (e.g., '$ident
) or characters
(e.g., #\+
). The machine generation phase of the parser
generates a match table which is an a-list of these objects along with
the token code. These codes are what the lexical analyzer should return.
BLA Bla bla. So in the end we have
"for"
).
make-lexer-generator
).
for
), whereas in the case of special items,
processed in the lexical analyzer by readers (e.g., read-num
), the
keys will be symbols (e.g., '$fl
).
Now one item need to be dealt with and that is the token value for the
default. It should be -1
or '$default
. WORK ON THIS.
Next: Translation, Previous: Introduction, Up: Top
nyacc provides several modules:
This is a module providing macros for generating specifications, machines and parsers.
This is a module providing procedures for generating lexical analyzers.
This is a module providing utilities used by the other modules.
lalr
ModuleWARNING: This section is quite crufty.
The lalr1
module provides syntax and procedures for building LALR
parsers. The following syntax and procedures are exported:
lalr-spec
syntax
make-lalr-machine
procedure
We have (experimental) convenience macros:
($? foo bar baz) => ``foo bar baz'' occurs never or once ($* foo bar baz) => ``foo bar baz'' occurs zero or more times ($+ foo bar baz) => ``foo bar baz'' occurs one or more times
However, these have hardcoded actions and are considered to be, in current form, unattractive for practical use.
Todo: discuss
'$fx
, '$ident
)
(pp-lalr-grammar calc-spec)
(pp-lalr-machine calc-mach)
(define calc-mach (compact-mach calc-mach))
(define calc-mach (hashify-machine calc-mach))
expr
could have been expressed using
(expr (expr "+" expr ($$ (+ $1 $3)))) (expr (expr "-" expr ($$ (- $1 $3)))) (expr (expr "*" expr ($$ (* $1 $3)))) (expr (expr #\/ expr ($$ (/ $1 $3)))) (expr ('$fx ($$ (string->number $1))))
(prec< "then" "else") (prec< "t1" "t2" "t3" "t4" "t5") => ((t1 . t2) (t2 . t3) (t3 . t4) (t4 . t5) (then . else))
lex
ModuleThe NYACC lex
module provide routines for constructing
lexical analyzers. The intension is to provide routines to make
construction easy, not necessarily the most efficient.
export
ModuleNYACC provides routines for exporting NYACC grammar specifications to other LALR parser generators.
The Bison exporter uses the following rules:
"for"
is
converted to FOR
.
"+"
is converted to '+'
.
#\!
is converted to '!'
.
ChSeq_i_j_k
where
i, j and k are decimal representations of the character
code. For example "+="
is converted to ChSeq_43_61
.
$
and -
are replaced with _
.
TODO: Export to Bison xml format.
The Guile exporter uses the following rules: TBD.
Next: Administrative, Previous: Parsing, Up: Top
Under ‘examples/nyacc’ are utilities for translating languages along with some samples. The approach that is used here is to parse languages into a SXML based parse tree and use the SXML modules in Guile to translate. We have built a javascript to tree-il translater which means that one can execute javascript at the Guile command line:
scheme@(guile-user)> ,L javascript need to complete
In actions in nyacc can use our tagged-lists to build the trees. For example, building a statement list for a program might go like this:
(program (stmt-list ($$ `(program ,(tl->list $1)))) (...)) (stmt-list (stmt ($$ (make-tl 'stmt-list $1))) (stmt-list stmt ($$ (tl-append $1 $2))))
To work with the trees described in the last section use
(sx-ref tree 1) (sx-attr tree) (sx-attr-ref tree 'item) (sx-tail tree 2)
This illustrates translation with foldts*-values
and
sxml-match
.
Next: Todos, Previous: Translation, Up: Top
Installation instructions are included in the top-level file README.nyacc of the source distribution.
Bug reporting will be dealt with once the package is place on a publically accessible source repository.
The Free Documentation License is included in the Guile Reference Manual. It is included with the NYACC source as the file COPYING.DOC.
Next: References, Previous: Administrative, Up: Top
Todo/Notes/Ideas:
add error handling (lalr-spec will now return #f for fatal error)
support other target languages: (write-lalr-parser pgen "foo.py" #:lang ’python)
export functions to allow user to control the flow i.e., something like: (parse-1 state) => state
macros - gotta be scheme macros but how to deal with other stuff (macro ($? val ...) () (val ...)) (macro ($* val ...) () (_ val ...)) (macro ($+ val ...) (val ...) (_ val ...)) idea: use $0 for LHS
support semantic forms: (1) attribute grammars, (2) translational semantics, (3) operational semantics, (4) denotational semantics
add ($abort) and ($accept)
keep resolved shift/reduce conflicts for pp-lalr-machine (now have rat-v – removed action table – in mach, need to add to pp)
add a location stack to the parser/lexer
write parser file generator (working prototype)
think
Aho, A.V., Sethi, R., and Ullman, J. D., “Compilers: Principles, Techniques and Tools,” Addison-Wesley, 1985 (aka the Dragon Book)
DeRemer, F., and Pennello, T., “Efficient Computation of LALR(1) Look-Ahead Sets.” ACM Trans. Prog. Lang. and Systems, Vol. 4, No. 4., Oct. 1982, pp. 615-649.
R. P. Corbett, “Static Semantics and Compiler Error Recovery,” Ph.D. Thesis, UC Berkeley, 1985.