File mu/Lexer

class mu.Lexer

Inherits

State supers

gps.StreamLexer

variables

tom.MutableDictionary tokens;

The set of identifier-like tokens known by this lexer class.

methods

redefine instance (id)
  alloc;

A redefinition of alloc to provide some of Objective-C's +initialize functionality.

void
  addToken (tom.ByteString, int) (name, number);

Add the identifier name to the identifier-like tokens as the Token value number.

deferred void
  fillTokens;

Fill the tokens by repeatedly invoking addToken.

deferred class (gps.Parser)
  parserClass;

Undocumented.

instance mu.Lexer

variables

tom.MutableByteString buf;

The buffer for what we read.

int size;

The (cached) size of the buffer.

int start;

The start position of the current token in the buffer, the next position in the buffer, and the end position (one beyond last).

int next;

int beyond;

int intvalue;

The semantics of an integer kind.

tom.ByteSubstring doc_tag;

A substring used to refer to the documentation tag in the buf.

public mutable boolean docp;

Iff TRUE, we are supposedly interested in documentation.

public mutable boolean skips;

Iff TRUE, we do not collect semantics with the tokens.

methods

redefine id
  initWithStream tom.InputStream s
           named tom.String n;

Undocumented.

boolean
  docp tom.String tag;

Return TRUE iff we are interested in documentation that starts with the tag. The default implementation simply asks self whether docp (plain).

int
  nextToken;

Undocumented.

int
  semantics;

Undocumented.

deferred int
  character;

Various methods to return the correct token number for a specific parser.

deferred int
  identifier;

Undocumented.

deferred int
  number;

Undocumented.

deferred int
  string;

Undocumented.

int
  documentation tom.String body
            tag tom.String tag
             at int line;

The default action for returning documentation is to skip it. This is necessary if docp is set while lexer thinks it should not. A lexer genuinly interested in documentation, for the time being, should return the correct documentation token to the caller, or it could inform the parser of the documentation, which avoids having to flood the grammar with DOCUMENTATION tokens.

int
  documentation tom.String body
            tag tom.String tag;

Undocumented.