Unit parsing

This type defines the character types that are recognized by a scanner for a parsed language.

Values

• CHARCAT_DUMMY = 0: An unrecognized or erroneous character
• CHARCAT_SPECIAL: A character that is not alphanumeric, but which is recognized by the language
• CHARCAT_LETTER: A character that may begin an identifier or reserved word
• CHARCAT_DIGIT: A character that may begin a numeric constant
• CHARCAT_WORD: A character that may be part of an identifier or reserved word, but which may not begin it
• CHARCAT_NUMERIC: A character that may be part of a numeric constant, but which may not begin it
• CHARCAT_SPACE: A character that is recognized as white space, but which is not an indicator that the line or the stream has ended
• CHARCAT_EOL: A character that may delimit the end of a source line
• CHARCAT_EOS: A character that may delimit the end of a source stream
• CHARCAT_ERROR: A character that is unrecognized

This type refers to the class definition for all instances of AParsedLanguage and its descendants. It allows generic parsers to be defined which can accept any valid instance of AParsedLanguage. It also allows parsers for a given language to construct their own language definition instances, simplifying the steps required to parse a given language.

This type represents an opcode, which is a way of constructing a numeric representation of a token string that has special meaning to a parser. The internal representation of an opcode is faster for a computer to manage than the associated token string; as a result, almost all parsers use some kind of logic to match a token string to an internal representation.

This type is defined here to make parser code more flexible.

This type represents a dynamic array of opcodes. It is primarly used by ASyntaxRule.

This type defines the ways in which instances of AScanner handle instances of certain tokens:

• SCAN_NO_WHITESPACE: Discard whitespace tokens when they are encountered. When this behavior is enabled, AScanner.CurrentToken will never refer to an instance of ASpaceToken, and AScanner.next will read tokens from the source until one is encountered that is not determined to be whitespace.

• SCAN_CONSOLIDATE_WHITESPACE: Consolidate consecutive instances of the same whitespace character into a single instance of ASpaceToken. When this behavior is enabled, consecutive instances of the same whitespace character in the source will be collected into one instance of ASpaceToken; otherwise, multiple instances of ASpaceToken will be returned.

  Obviously, this flag has no effect if SCAN_NO_WHITESPACE is enabled.

• SCAN_CONSOLIDATE_LINE_ENDINGS: Consolidate consecutive instances of the same line ending character into a single instance of ALineEndingToken. When this behavior is enabled, consecutive instances of the same line ending character in the source will be collected into one instance of ALineEndingToken; otherwise, multiple instances of ALineEndingToken will be returned.

Values

• SCAN_NO_WHITESPACE: Discard whitespace tokens
• SCAN_CONSOLIDATE_WHITESPACE: Consolidate whitespace into a single token
• SCAN_CONSOLIDATE_LINE_ENDINGS: Consolidate line endings into a single token

This type defines a set of one or more token behaviors. For more information on the behaviors available, see TScannerTokenBehavior.

This type represents the scope of a symbol, which is a way of indicating the symbol table to which it belongs. Generally speaking, each program block has its own scope.

This type is defined to make parser code more flexible.

This constant defines how a string representation of ALineEndingToken is constructed when ALineEndingToken.toString is called.

The integer placeholder is filled with the number of line endings represented by the token, as retrieved by a call to ALineEndingToken.lineCount.

This constant defines the character which makes up a typical Apple-style line ending. It can be used by language definitions to specify those characters which are considered line endings.

This constant the defines the character which makes up a typical Unix-style line ending. It can be used by language definitions to specify those characters which are considered line endings.

This constant defines the sequence of characters which makes up typical Windows-style line endings. It is used in the tokenizer source and can also be used by language definitions to specify those characters which are considered line endings.

This string defines the error message and format used when the end of a stream is encountered unexpectedly.

As defined here, the error message requires no additional parameters.

This constant defines the format of the error message logged by instances of AParser when AParser.resyncToToken and AParser.resyncTo encounter an unexpected token in the source.

The string placeholder is filled with a string representation of the unexpected token, as returned by a call to AToken.toString.

This string defines the characters which are typically allowed to begin numeric constants in various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically counted as line ending characters by various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically used to mark the end of a stream by various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically allowed to begin identifiers in various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically allowed to be part of a numeric constant, though they may not begin it, by various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically counted as whitespace by various languages. It is defined here for convenience, so that other language specifications may reference it.

This string defines the characters which are typically allowed to be part of an identifier, though they may not begin it, in various languages. It is defined here for convenience, so that other language specifications may reference it.

This constant determines the format of the string returned by calls to AParserNote.toString, AParserHint.toString, AParserWarning.toString, AParserSyntaxError.toString, and AParserFatalError.toString when there is a valid (named) source.

The first string placeholder is the name of the source stream being parsed, as determined by a call to AStream.name. If the source stream represents a file stream, then this will contain the name of the file being parsed. The integer placeholder will be filled with the current line number in the source stream, as determined by a call to AScanner.lineNumber. The second string placeholder is filled with the error or status message, which is constructed by calling the inherited toString message for each of the above classes: ALoggedNote.toString, ALoggedHint.toString, ALoggedWarning.toString, ALoggedError.toString, or ALoggedFatalError.toString.

Rules in ASyntaxRuleset can be retrieved by an arbitrary name or value; however, it is common to define a symbolic constant for each rule so that the rule name or value does not have to be hard-coded. In addition, there are some basic rules that will probably be common across all types of parsers: a rules defining basic token types and a rule to indicate which tokens are allowed to begin and end a statement.

This constant is used to define the tokens that may begin a statement. It is not automatically used by the base instance of AParsedLanguage, but is provided here as a way to standardize the way in which one of the more fundamental rules of a parsed language is defined.

Rules in ASyntaxRuleset can be retrieved by an arbitrary name or value; however, it is common to define a symbolic constant for each rule so that the rule name or value does not have to be hard-coded. In addition, there are some basic rules that will probably be common across all types of parsers: a rules defining basic token types and a rule to indicate which tokens are allowed to begin and end a statement.

This constant is used to define the tokens that may end a statement. It is not automatically used by the base instance of AParsedLanguage, but is provided here as a way to standardize the way in which one of the more fundamental rules of a parsed language is defined.

Rules in ASyntaxRuleset can be retrieved by an arbitrary name or value; however, it is common to define a symbolic constant for each rule so that the rule name or value does not have to be hard-coded. In addition, there are some basic rules that will probably be common across all types of parsers: rules defining basic token types and indicating which tokens are allowed to begin and end a statement.

This constant is used to define an invalid rule (no rule).

This constant defines the default handling flags used by instances of AScanner when whitespace and line ending tokens are encountered.

The default behavior is to ignore whitespace and to consolidate it into a single token if it is not ignored. To set the behavior of a scanner, call AScanner.setTokenBehaviors.

This constant defines how a string representation of AStreamEndingToken is constructed when AStreamEndingToken.toString is called.

As defined here, the format requires no additional parameters.

This constant defines how a string representation of ASymbolicToken is constructed when ASymbolicToken.toString is called.

The string placeholder will be filled with the display name of the class, as returned by a call to APrintingObject.displayName. The first and second integer placeholders are filled with the scope and index of the symbol represented by the token, as returned by a call to ASymbolicToken.symbol.

This flag indicates that a symbol has been declared as external to the source. It is designed to be combined with another symbol category code, such as SYMCAT_SUBROUTINE (to indicate an external function) or SYMCAT_VARIABLE (to indicate an external variable).

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a constant value: either a numeric constant or a string literal.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a parameter accepted by a subroutine: one of the variables that may be passed to a subroutine.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a member of a structured type: a record, class, or other memory structure.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a subroutine: a function, procedure, or other labeled block of code.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a data type.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that was encountered in the source stream, but which has no definition (an undeclared identifier of some kind). This is the default value for ASymbol.category that is set by ASymbol.init, although descendant classes may modify this value.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant is a placeholder for user-defined symbol categories. User- defined symbol categories should begin with this value. Values lower than this constant are reserved in case there is a need for future expansion of the base parsing library.

This constant enumerates one of the ways in which a symbol may be defined in a source stream. It is not directly used by the base implementations of ASymbol, ASymbolFromSource, or ASymbolRecalled – except when they read and write themselves using a binary stream – but they are provided to help standardize the ways in which symbols are represented across various parser implementations.

This constant represents a symbol that is declared as a variable: an instance of a known type.

This string controls the format of the error message output when there is an attempt to define a symbol that has the same name as one which has already been defined in the current scope. This message is output as a syntax error by ASymbolParser.EnterSymbolInto.

The first string placeholder is filled with the name of the symbol, as returned by a call to ASymbol.toString. The second string placeholder is filled with the name of the source stream in which the symbol was declared, as returned by a call to ASymbolFromSource.sourceName. The integer placeholder is filled with the line number at which the symbol was declared, as returned by a call to ASymbolFromSource.sourceLine.

This constant enumerates one of the more common symbol table scopes.

This constant represents a symbol that belongs to the global symbol table (i.e., which is visible from everywhere in the source program).

This constant enumerates one of the more common symbol table scopes.

This constant represents a symbol that has no scope (belongs to no symbol table). This is the default value for ASymbol.scope that is set by ASymbol.init, although descendant classes may modify this value.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to mask out all but the bits which identify the category into which the opcode falls. One simply ANDs the TOpcode with this value; the resulting value will be one of the category constants.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define an invalid, "null" token – one that has not been properly initialized. This is the initial state of all base instances of AToken.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a sequence of at least one character that represents the end of a line in the source being parsed. If the token has been processed by the base instance of ALineEndingTokenFromSource, then the token text may contain one or two characters, depending upon whether the source file has Windows, Apple, or Unix/Linux-based line endings.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a token that represents the end of the source stream. Depending on the language being parsed, there may be an actual sequence of characters that denotes the end of the source; it is more likely, however, that a scanner will return this type of token when the stream being scanned indicates that the end has been reached.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define an unrecognized or erroneous token. The base instances of ANumericConstantToken will place themselves in this category if the number failed to meet basic checks of validity; this allows the parser to complain that something in the source looks like a number or string constant, but is not actually one.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define an identifier, which is a series of alphanumeric characters that does not have a pre-defined meaning in the syntax of the language being parsed. Identifiers usually represent variables or routines within the source being parsed.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a keyword, which is a series of alphanumeric characters that has a pre-defined meaning in the syntax of the language being parsed. Keywords can represent language constructs, such as a conditional statement, or reserved method names.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a valid numeric constant. If the token has been parsed by the base instance of ANumericConstantToken, and the token opcode contains this category, then it means that the sequence of characters contained by the token evaluates to a valid number.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in ordr to quickly determine the type of token represented by a specific opcode.

This constant is used to define a rule, thus allowing instances of ASyntaxRule to contain specific tokens as well as references to other rules.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a token that is recognized at whitespace. Note that end-of-line and end-of-stream markers are not included in this category; these have their own token categories: TOKCAT_EOL and TOKCAT_EOS, respectively. It is up to the scanner or parser to determine whether or not a whitespace token will be processed or ignored; some languages, like Python, find whitespace significant.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a special token, which may consist of one or more characters with pre-defined meaning in the syntax of the language. Special tokens often represent operators that are represented with a symbol, or delimiters that separate identifiers or which indicate the continuation (or end) of a line.

Each of these constants defines a category of token that is recognized by a scanner for a parsed language. The category code resides in the high word of a TOpcode, while the specific internal representation of that token occupies the low word. It is possible to mask out the internal representation using TOKCAT_CATMASK, in order to quickly determine the type of token represented by a specific opcode.

This constant is used to define a valid string literal; however, strings are NOT parsed by the default scanner and tokenizer methods defined within parsing. This is because there are a variety of ways to define a string (such as in Python, where there are strings and docstrings), so it is left to the parser to process the string. However, the token code remains valid and can be output into an intermediate code stream.

This constant defines how a string representation of AToken is constructed when AToken.toString is called.

The string placeholder will be filled with the display name of the class, as returned by a call to APrintingObject.displayName. The integer placeholder will be filled with the value of the token opcode, as returned by a call to AToken.opcode.