|
Grammatica 1.3 Documentation | |||||||
FRAMES NO FRAMES | ||||||||
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
System.Object
|
+--Tokenizer
in Tokenizer.csA character stream tokenizer. This class groups the characters read from the stream together into tokens ("words"). The grouping is controlled by token patterns that contain either a fixed string to search for, or a regular expression. If the stream of characters don't match any of the token patterns, a parse exception is thrown.
Constructor Summary | |
Tokenizer( TextReader input )
Creates a new tokenizer for the specified input stream. |
Method Summary | |
void |
AddPattern( TokenPattern pattern )
Adds a new token pattern to the tokenizer. |
int |
GetCurrentColumn()
Returns the current column number. |
int |
GetCurrentLine()
Returns the current line number. |
string |
GetPatternDescription( int id )
Returns a description of the token pattern with the specified id. |
Token |
Next()
Finds the next token on the stream. |
override string |
ToString()
Returns a string representation of this object. |
Constructor Detail |
public Tokenizer( TextReader input );
input
- the input stream to readMethod Detail |
public void AddPattern( TokenPattern pattern );
pattern
- the pattern to addParserCreationException
- if the pattern couldn't be
added to the tokenizerpublic int GetCurrentColumn();
public int GetCurrentLine();
public string GetPatternDescription( int id );
id
- the token pattern idpublic Token Next();
ParseException
- if the input stream couldn't be read or
parsed correctlypublic override string ToString();
|
Grammatica 1.3 Documentation | |||||||
FRAMES NO FRAMES | ||||||||
SUMMARY: INNER | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |