AbstractLexer
in package
Base class for writing simple lexers, i.e. for creating small DSLs.
Table of Contents
- $lookahead : array<string|int, mixed>|null
- The next token in the input.
- $token : array<string|int, mixed>|null
- The last matched/seen token.
- $input : string
- Lexer original input string.
- $peek : int
- Current peek of current lexer position.
- $position : int
- Current lexer position in input string.
- $regex : string
- Composed regex for input parsing.
- $tokens : array<string|int, mixed>
- Array of scanned tokens.
- getInputUntilPosition() : string
- Retrieve the original lexer's input until a given position.
- getLiteral() : int|string
- Gets the literal for a given token.
- glimpse() : array<string|int, mixed>|null
- Peeks at the next token, returns it and immediately resets the peek.
- isA() : bool
- Checks if given value is identical to the given token.
- isNextToken() : bool
- Checks whether a given token matches the current lookahead.
- isNextTokenAny() : bool
- Checks whether any of the given tokens matches the current lookahead.
- moveNext() : bool
- Moves to the next token in the input string.
- peek() : array<string|int, mixed>|null
- Moves the lookahead token forward.
- reset() : void
- Resets the lexer.
- resetPeek() : void
- Resets the peek pointer to 0.
- resetPosition() : void
- Resets the lexer position on the input to the given position.
- setInput() : void
- Sets the input data to be tokenized.
- skipUntil() : void
- Tells the lexer to skip input tokens until it sees a token with the given value.
- getCatchablePatterns() : array<string|int, mixed>
- Lexical catchable patterns.
- getModifiers() : string
- Regex modifiers
- getNonCatchablePatterns() : array<string|int, mixed>
- Lexical non-catchable patterns.
- getType() : int|string|null
- Retrieve token type. Also processes the token value if necessary.
- scan() : void
- Scans the input string for tokens.
Properties
$lookahead
The next token in the input.
public
array<string|int, mixed>|null
$lookahead
$token
The last matched/seen token.
public
array<string|int, mixed>|null
$token
$input
Lexer original input string.
private
string
$input
$peek
Current peek of current lexer position.
private
int
$peek
= 0
$position
Current lexer position in input string.
private
int
$position
= 0
$regex
Composed regex for input parsing.
private
string
$regex
$tokens
Array of scanned tokens.
private
array<string|int, mixed>
$tokens
= []
Each token is an associative array containing three items:
- 'value' : the string value of the token in the input string
- 'type' : the type of the token (identifier, numeric, string, input parameter, none)
- 'position' : the position of the token in the input string
Methods
getInputUntilPosition()
Retrieve the original lexer's input until a given position.
public
getInputUntilPosition(int $position) : string
Parameters
- $position : int
Return values
string —getLiteral()
Gets the literal for a given token.
public
getLiteral(int|string $token) : int|string
Parameters
- $token : int|string
Return values
int|string —glimpse()
Peeks at the next token, returns it and immediately resets the peek.
public
glimpse() : array<string|int, mixed>|null
Return values
array<string|int, mixed>|null —The next token or NULL if there are no more tokens ahead.
isA()
Checks if given value is identical to the given token.
public
isA(mixed $value, int|string $token) : bool
Parameters
- $value : mixed
- $token : int|string
Return values
bool —isNextToken()
Checks whether a given token matches the current lookahead.
public
isNextToken(int|string $token) : bool
Parameters
- $token : int|string
Return values
bool —isNextTokenAny()
Checks whether any of the given tokens matches the current lookahead.
public
isNextTokenAny(array<string|int, mixed> $tokens) : bool
Parameters
- $tokens : array<string|int, mixed>
Return values
bool —moveNext()
Moves to the next token in the input string.
public
moveNext() : bool
Return values
bool —peek()
Moves the lookahead token forward.
public
peek() : array<string|int, mixed>|null
Return values
array<string|int, mixed>|null —The next token or NULL if there are no more tokens ahead.
reset()
Resets the lexer.
public
reset() : void
Return values
void —resetPeek()
Resets the peek pointer to 0.
public
resetPeek() : void
Return values
void —resetPosition()
Resets the lexer position on the input to the given position.
public
resetPosition(int $position) : void
Parameters
- $position : int
-
Position to place the lexical scanner.
Return values
void —setInput()
Sets the input data to be tokenized.
public
setInput(string $input) : void
The Lexer is immediately reset and the new input tokenized. Any unprocessed tokens from any previous input are lost.
Parameters
- $input : string
-
The input to be tokenized.
Return values
void —skipUntil()
Tells the lexer to skip input tokens until it sees a token with the given value.
public
skipUntil(string $type) : void
Parameters
- $type : string
-
The token type to skip until.
Return values
void —getCatchablePatterns()
Lexical catchable patterns.
protected
abstract getCatchablePatterns() : array<string|int, mixed>
Return values
array<string|int, mixed> —getModifiers()
Regex modifiers
protected
getModifiers() : string
Return values
string —getNonCatchablePatterns()
Lexical non-catchable patterns.
protected
abstract getNonCatchablePatterns() : array<string|int, mixed>
Return values
array<string|int, mixed> —getType()
Retrieve token type. Also processes the token value if necessary.
protected
abstract getType(string &$value) : int|string|null
Parameters
- $value : string
Return values
int|string|null —scan()
Scans the input string for tokens.
protected
scan(string $input) : void
Parameters
- $input : string
-
A query string.