Template Lexer

The implementation of the lexer is contained within this mixin template.

template Lexer(Token, alias defaultTokenFunction, alias tokenSeparatingFunction, alias staticTokens, alias dynamicTokens, alias possibleDefaultTokens, alias tokenHandlers) ;

To use it, this template should be mixed in to a struct that represents the lexer for your language. This struct should implement the following methods:

  • popFront, which should call this mixin's popFront() and additionally perform any token filtering or shuffling you deem necessary. For example, you can implement popFront to skip comment or tokens.
  • A function that serves as the default token lexing function. For most languages this will be the identifier lexing function. This should then be passed to the Lexer template mixin as the template parameter.
  • A function that is able to determine if an identifier/keyword has come to an end. This function must return bool and take a single size_t argument representing the number of bytes to skip over before looking for a separating character.
  • Any functions referred to in the tokenHandlers template paramater. These functions must be marked pure nothrow, take no arguments, and return a token
  • A constructor that initializes the range field as well as calls popFront() exactly once (to initialize the front field).

Contained Variables

NameTypeDescription
_front TokenThe token that is currently at the front of the range.
range LexerRangeThe lexer input.

Contained Functions

NameDescription
_popFront Advances the lexer to the next token and stores the new current token in the front variable.
advance
calcSplitCount
empty Implements the range primitive empty.
escape
front Implements the range primitive front.
generateByteMask
generateMask
generateStatements
generateStatementsStep
getBeginningChars
getFront
printCase

Contained Aliases

NameDescription
_IDType

Parameters

NameDescription
Token TokenStructure
defaultTokenFunction defaultTokenFunction
tokenSeparatingFunction tokenSeparatingFunction
staticTokens staticTokens
dynamicTokens dynamicTokens
possibleDefaultTokens possibleDefaultTokens
tokenHandlers tokenHandlers

Examples

struct CalculatorLexer
{
    mixin Lexer!(IdType, Token, defaultTokenFunction, isSeparating,
        staticTokens, dynamicTokens, possibleDefaultTokens, tokenHandlers);

    this (ubyte[] bytes)
    {
        this.range = LexerRange(bytes);
        popFront();
    }

    void popFront() pure
    {
        _popFront();
    }

    Token lexNumber() pure nothrow @safe
    {
        // implementation goes here
    }

    Token lexWhitespace() pure nothrow @safe
    {
        // implementation goes here
    }

    Token defaultTokenFunction() pure nothrow @safe
    {
        // There is no default token in the example calculator language, so
        // this is always an error.
        range.popFront();
        return Token(tok!"");
    }

    bool isSeparating(size_t offset) pure nothrow @safe
    {
        // For this example language, always return true.
        return true;
    }
}