pip_services3_expressions.tokenizers.ITokenizerState module

class pip_services3_expressions.tokenizers.ITokenizerState.ITokenizerState

Bases: abc.ABC

A tokenizerState returns a token, given a scanner, an initial character read from the scanner, and a tokenizer that is conducting an overall tokenization of the scanner. The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the next_token() method has a tokenizer argument.

next_token(scanner: pip_services3_expressions.io.IScanner.IScanner, tokenizer: ITokenizer) → pip_services3_expressions.tokenizers.Token.Token

Gets the next token from the stream started from the character linked to this state.

Parameters

• scanner – A textual string to be tokenized.

• tokenizer – A tokenizer class that controls the process.

Returns

The next token from the top of the stream.