causerie

Introduction

Units

Class Hierarchy

Classes, Interfaces, Objects and Records

Types

Variables

Constants

Functions and Procedures

Identifiers

Classes hierarchy graph

Class AToken

DescriptionHierarchyFieldsMethodsProperties

Unit

parsing

Declaration

type AToken = class(ANode)

Description

This class represents a basic token which is parsed from a source stream or retrieved from an intermediate code stream. A token is used to match text from the source to an internal representation (TOpcode) so that it can be processed and acted upon by a parser.

Tokens support streaming to and from arbitrary streams and can be chained together in lists – this, in fact, is how the default implementations of AParser and its descendants manage tokens.

Hierarchy

Overview

Fields

Protected myOpcode: TOpcode;
Protected mySilence: boolean;

Methods

Public class function from(const Source: AStream): AToken; virtual;
Public function init: boolean; override;
Public function category: TOpcode;
Public procedure returnTo(const Source: AStream); virtual;
Public function isOneOf(const theseOpcodes: array of TOpcode): boolean; virtual;
Public function isPartOf(const Rule: ASyntaxRule): boolean; overload; virtual;
Public function shallowCopyFrom(const Other: AnObject): boolean; override;
Public function selfStreamingLength: TStreamIOSize; override;
Public function writeSelfTo(const Dest: AStream): TStreamIOSize; override;
Public function writeTo(const Dest: AStream): TStreamIOSize; override;
Public function readFrom(const Source: AStream): TStreamIOSize; override;
Public function toString: AnsiString; override;
Public function opcode: TOpcode;
Public function silenced: boolean; virtual;
Public function setSilenced(const flag: boolean): boolean; virtual;

Description

Fields

Protected myOpcode: TOpcode;

Stores the internal representation of the token – its opcode

Protected mySilence: boolean;

Indicates whether or not the token should be appended to parser output

Methods

Public class function from(const Source: AStream): AToken; virtual;

Construct and return the appropriate instance of AToken or ASymbolicToken.

This method is designed to be used on streams that contain intermediate code. It is a factory method that will read an opcode from Source and construct the appropriate token instance depending on that opcode. If the opcode is TOKCAT_IDENTIFIER, TOKCAT_NUMBER, or TOKCAT_STRING, an instance of ASymbolicToken is read and returned; otherwise an instance of AToken is returned.

Public function init: boolean; override;

Initializer

Public function category: TOpcode;

Retrieve the category to which the token belongs.

Token category constants are defined such that the category can be specified in the high word of a TOpcode while a unique code for the token is specified in the low word. This is particularly useful for keywords, operators, and special characters that are recognized by a language.

This method simply masks out all but the high word and returns that value, which should be one of the TOKCAT_* constants, such as TOKCAT_KEYWORD. For tokens that do not represent a keyword, operator, or special character, this value will likely be the same as that returned by AToken.opcode.

Public procedure returnTo(const Source: AStream); virtual;

Return the token to its source. The token will be read again the next time the source is queried for a token.

As defined in the base implementation of AToken, this method merely calls AStream.rewindBy on Source, passing the size of its opcode as the number of bytes by which the stream position should be rewound.

This method is defined to provide a way for a scanner to "peek" at the next token in the source stream without actually having to process it.

Public function isOneOf(const theseOpcodes: array of TOpcode): boolean; virtual;

Determine whether or not the token opcode is one of those specified by a list of opcodes.

This method checks the token opcode against those specified in theseOpcodes. If a match is found, then this method returns True. Otherwise it returns False.

Public function isPartOf(const Rule: ASyntaxRule): boolean; overload; virtual;

Determine whether or not the token is specified in the given rule.

This method checks the token opcode against those specified by Rule. If the token appears anywhere at all in the rule, this method returns True; otherwise it returns False.

This method is defined to allow a parser to determine whether a given token may appear at the current source location; in other words, it allows a parser to check syntax.

Public function shallowCopyFrom(const Other: AnObject): boolean; override;

Construct a shallow copy of the other object.

This method extends the behavior inherited from AnObject.shallowCopyFrom: it calls that method, then checks to see whether Other is an instance of AToken. If so, it copies the values of

from Other to Self, overwriting the values in Self.

Note that this method does NOT copy any sibling or child nodes and so cannot be used to create a full copy of any descendant of ANode. The copy will NOT be automatically placed in the list to which Other belongs, if any, but the caller is free to do so.

Public function selfStreamingLength: TStreamIOSize; override;

Calculate the number of bytes required to stream the token.

This method overrides the behavior inherited from ANode.selfStreamingLength. It returns the number of bytes required to stream the value of AToken.opcode.

Public function writeSelfTo(const Dest: AStream): TStreamIOSize; override;

Write the token, and just the token, to the specified intermediate code stream.

This method overrides the behavior inherited from ANode. In the base implementation provided in AToken, it simply writes the value of AToken.opcode to Dest. Descendants of AToken may override this method to write additional data to the stream.

Returns

The total number of bytes written to Dest.

Public function writeTo(const Dest: AStream): TStreamIOSize; override;

Write the token to the specified intermediate code stream.

This method overrides the behavior inherited from ANode. It does not seek to preserve the links between tokens even if they are part of ATokenList; this is because tokens are parsed serially and do not need to be loaded all at once into memory. The method as defined here simply calls AToken.writeSelfTo and returns.

Returns

The total number of bytes written to Dest.

Public function readFrom(const Source: AStream): TStreamIOSize; override;

Read the token from the specified intermediate code stream.

This method overrides the behavior inherited from ANode. In the base implementation provided in AToken, it simply reads the value of AToken.opcode from Source. Descendants of AToken may override this method to read additional data from the stream.

Returns

The total number of bytes read from Source.

Public function toString: AnsiString; override;

Produce a string representation of the token, suitable for output to a text-based device such as a console.

This method overrides the behavior inherited from ANode. It returns the display name of the token and a hex-formatted representation of AToken.opcode. The format of the returned string is governed by toknStringRepresentation.

Public function opcode: TOpcode;

Retrieve the opcode associated with the token.

The value returned by this function will vary depending on the type of token parsed. At the very least, it will contain one of the TOKCAT_* constants, such as TOKCAT_IDENTIFIER. This value is set by the token when it is parsed from the source and cannot be changed.

Public function silenced: boolean; virtual;

Determine whether or not the token is silenced.

A token that is silenced should not be output by a parser to its intermediate code stream. This flag will most often be used to mark tokens that represent whitespace, as most parsers choose to ignore whitespace.

Public function setSilenced(const flag: boolean): boolean; virtual;

Set whether or not the token is silenced.

A token that is silenced should not be output by a parser to its intermediate code stream. This flag will most often be used to mark tokens that represent whitespace, as most parsers are written to ignore whitespace.

Returns

Whether or not the token was previously silenced.

Generated by PasDoc 0.13.0 on 2015-01-10 17:13:18