Class ASourceScanner

Refers to the language used to tokenize the source

Construct a new instance of ASourceScanner that will iterate over the source in ThisSource using ThisLanguage to tokenize the source.

This method differs from the constructor for instances of AScanner in that it expects the source stream to be an instance of ATextInputStream rather than an intermediate code stream, and it requires a language definition to use in order to tokenize the source.

Initializer

Retrieve the next token from the source.

This method overrides the behavior inherited from AScanner: where that method constructs and returns instances of AToken exclusively, this method will construct and return instances of ATokenFromSource for the most part.

This method always succeeds; however, if the end of the source has been reached, the token created will be an instance of AStreamEndingToken. If AScanner.CurrentToken already refers to such an instance, then this routine does nothing.

Peek at the next token from the source without advancing the scanner.

This method reads a token from the source and returns it, then rewinds the source position so that the token will be read again.

The caller should free the token reference returned by this routine when it is no longer needed. The scanner does NOT free the reference.

Retrieve a reference to the language definition used to tokenize the source.

The caller should NOT free this reference without first freeing the scanner; the reference is required in order to properly tokenize the source (it is passed to ATokenFromSource.using.)