Overview   Project   Class   Tree   Deprecated   Index 
Grammatica 1.3 Documentation
FRAMES    NO FRAMES
SUMMARY:  INNER | FIELD | CONSTR | METHOD

PerCederberg.Grammatica.Parser
Class Tokenizer

System.Object
   |
   +--Tokenizer

   in Tokenizer.cs

class Tokenizer
extends System.Object

A character stream tokenizer. This class groups the characters read from the stream together into tokens ("words"). The grouping is controlled by token patterns that contain either a fixed string to search for, or a regular expression. If the stream of characters don't match any of the token patterns, a parse exception is thrown.


Constructor Summary
Tokenizer( TextReader input )
          Creates a new tokenizer for the specified input stream.
 
Method Summary
 void AddPattern( TokenPattern pattern )
          Adds a new token pattern to the tokenizer.
 int GetCurrentColumn()
          Returns the current column number.
 int GetCurrentLine()
          Returns the current line number.
 string GetPatternDescription( int id )
          Returns a description of the token pattern with the specified id.
 Token Next()
          Finds the next token on the stream.
 override string ToString()
          Returns a string representation of this object.
 

Constructor Detail

Tokenizer

public Tokenizer( TextReader input );
Creates a new tokenizer for the specified input stream.
Parameters:
input - the input stream to read


Method Detail

AddPattern

public void AddPattern( TokenPattern pattern );
Adds a new token pattern to the tokenizer. The pattern will be added last in the list, choosing a previous token pattern in case two matches the same string.
Parameters:
pattern - the pattern to add
Throws:
ParserCreationException - if the pattern couldn't be added to the tokenizer

GetCurrentColumn

public int GetCurrentColumn();
Returns the current column number. This number will be the column number of the next token returned.
Returns:
the current column number

GetCurrentLine

public int GetCurrentLine();
Returns the current line number. This number will be the line number of the next token returned.
Returns:
the current line number

GetPatternDescription

public string GetPatternDescription( int id );
Returns a description of the token pattern with the specified id.
Parameters:
id - the token pattern id
Returns:
the token pattern description, or null if not present

Next

public Token Next();
Finds the next token on the stream. This method will return null when end of file has been reached. It will return a parse exception if no token matched the input stream, or if a token pattern with the error flag set matched. Any tokens matching a token pattern with the ignore flag set will be silently ignored and the next token will be returned.
Returns:
the next token found, or null if end of file was encountered
Throws:
ParseException - if the input stream couldn't be read or parsed correctly

ToString

public override string ToString();
Returns a string representation of this object. The returned string will contain the details of all the token patterns contained in this tokenizer.
Returns:
a detailed string representation

 Overview   Project   Class   Tree   Deprecated   Index 
Grammatica 1.3 Documentation
FRAMES    NO FRAMES
SUMMARY:  INNER | FIELD | CONSTR | METHOD