causerie

Introduction

Units

Class Hierarchy

Classes, Interfaces, Objects and Records

Types

Variables

Constants

Functions and Procedures

Identifiers

Classes hierarchy graph

Class AScanner

DescriptionHierarchyFieldsMethodsProperties

Unit

parsing

Declaration

type AScanner = class(AnIterator)

Description

This class represents a scanner that is used to return tokens from a stream.

The base class, AScanner, is meant to be used with files that contained parsed intermediate code. The first descendant of this class, ASourceScanner, is designed to be used with files that contain raw source.

Scanners are constructed as iterators that iterate over the tokens in an arbitrary stream. Thus, they can be used by a parser within an iterator loop: 

    type MyParser = ASourceParser
      public
        function parse: integer; override;
    end;

    // ...

    // Parse the source
    function MyParser.parse: integer;

    begin
      // MySource and MyScanner are both protected properties of
      // AParser and its descendants.  MyLanguage is a protected property of
      // ASourceParser and its descendants.

      // Assuming, then, that the above properties have been been properly set
      // up and initialized elsewhere before this method is called...

      result := -1;

      // We can't parse a nonexistent source
      if MySource = nil then
        exit;

      // Establish the scanner
      // MySource is an instance of ASourceInputStream; the 'iterator' method
      // actually constructs an instance of ASourceScanner for us.
      MyScanner := ASourceScanner(MySource.iterator);

      if MyScanner = nil then
        exit;

      // Zero syntax errors thus far
      result := 0;

      // Iterator loop
      while MyScanner.continues do
      begin
        case MyScanner.CurrentToken.opcode of:
          // Parser code here
        end;

        // Get the next token
        MyScanner.next;
      end;

      // Dispose of the scanner
      MyScanner.free;

      // Report the number of syntax errors (MyLog is a protected property of
      // AParser and its descendants)
      result := MyLog.Counter('errors').value;
    end;

Hierarchy

Overview

Fields

Protected MySource: AStream;
Protected myLineNumber: longword;
Protected MyCurrentToken: AToken;
Protected myTokenBehaviors: TScannerTokenBehaviors;

Methods

Public constructor forSource(const ThisSource: AStream); virtual; overload;
Public function init: boolean; override;
Public procedure returnToken(const ThisToken: AToken); virtual;
Public procedure rewind; virtual;
Public procedure next; override;
Public function Peek: AToken; virtual;
Public function continues: boolean; override;
Public function Source: AStream;
Public function lineNumber: longword;
Public function CurrentToken: AToken;
Public function tokenBehaviors: TScannerTokenBehaviors; virtual;
Public function setTokenBehaviors(const newBehaviors: TScannerTokenBehaviors): TScannerTokenBehaviors; virtual;

Description

Fields

Protected MySource: AStream;

Refers to the stream being scanned for tokens.

Protected myLineNumber: longword;

Stores the current line number in the source.

Protected MyCurrentToken: AToken;

Refers to the most recent token retrieved from the source stream

Protected myTokenBehaviors: TScannerTokenBehaviors;

Flags that indicate how the scanner handles certain tokens

Methods

Public constructor forSource(const ThisSource: AStream); virtual; overload;

Construct a scanner that will iterate over the given source stream.

Scanning begins at the current position within ThisSource.

Public function init: boolean; override;

Initializer

Public procedure returnToken(const ThisToken: AToken); virtual;

Return the specified token to the source stream.

This method can be used by parsers to "peek" at a token and then return it to the source for further processing.

ThisToken is freed by this routine and should not be used after the routine returns.

This method calls AToken.returnTo.

Public procedure rewind; virtual;

Return the token most recently read from the source back to the source.

This method can be used by parsers to "peek" at a token and then return it to the source for further processing. The current token is freed by this routine and should not be used after the routine returns.

This method calls AToken.returnTo.

Public procedure next; override;

Retrieve the next token from the source.

This method always succeeds; however, if the end of the source has been reached, the token created will be an instance of AStreamEndingToken. If AScanner.CurrentToken already refers to such an instance, then this routine does nothing.

Public function Peek: AToken; virtual;

Peek at the next token from the source without advancing the scanner.

This method reads a token from the source and returns it, then rewinds the source position so that the token will be read again.

The caller should free the token reference returned by this routine when it is no longer needed. The scanner does NOT free the reference.

Public function continues: boolean; override;

Determine whether or not there are more tokens to retrieve from the source.

If the source indicates that its end has been reached, this method will return False; otherwise, it will return True, because the last token to retrieve from the source will always be an instance of AStreamEndingToken.

Public function Source: AStream;

Retrieve a reference to the source stream being scanned for tokens.

This reference should NOT be freed before the scanner itself is freed, otherwise, undefined things (program crashes) will happen.

Public function lineNumber: longword;

Retrieve the current line number from the source.

This function is primarily used by the parser to output error and status messages. The line number is updated whenever an instance of ALineEndingToken is encountered by the scanner.

Public function CurrentToken: AToken;

Retrieve a reference to the most recent token read from the stream.

The caller should free this reference. This is NOT done by the scanner, not even when a new token is read from the source. This behavior is by design, since AParser and its descendants organize tokens into instances of ATokenList, which will free all of its token references when the list itself is destroyed.

Public function tokenBehaviors: TScannerTokenBehaviors; virtual;

Retrieve the behaviors defined for the scanner when certain tokens are encountered. For more information on the available behaviors, see TScannerTokenBehavior.

Public function setTokenBehaviors(const newBehaviors: TScannerTokenBehaviors): TScannerTokenBehaviors; virtual;

Set the behaviors for the scanner when certain tokens are encountered. For more information on the available behaviors, see TScannerTokenBehavior.

Returns

The previous value of AScanner.tokenBehaviors.

Generated by PasDoc 0.13.0 on 2015-01-10 17:13:18