pip_services3_expressions.tokenizers.AbstractTokenizer module
-
class
pip_services3_expressions.tokenizers.AbstractTokenizer.AbstractTokenizer Bases:
pip_services3_expressions.tokenizers.ITokenizer.ITokenizer,abc.ABCImplements an abstract tokenizer class.
-
clear_character_states()
-
get_character_state(symbol: int) → pip_services3_expressions.tokenizers.ITokenizerState.ITokenizerState
-
has_next_token() → bool Checks if there is the next token exist.
- Returns
True if scanner has the next token.
-
next_token() → pip_services3_expressions.tokenizers.Token.Token Gets the next token from the scanner.
- Returns
Next token of null if there are no more tokens left.
-
property
scanner
-
set_character_state(from_symbol: int, to_symbol: int, state: pip_services3_expressions.tokenizers.ITokenizerState.ITokenizerState)
-
tokenize_buffer(buffer: str) → List[pip_services3_expressions.tokenizers.Token.Token] Tokenizes a string buffer into a list of tokens structures.
- Parameters
buffer – A string buffer to be tokenized.
- Returns
A list of token structures.
-
tokenize_buffer_to_strings(buffer: str) → List[str] Tokenizes a string buffer into a list of strings.
- Parameters
buffer – A string buffer to be tokenized.
- Returns
A list of token strings.
-
tokenize_stream(scanner: pip_services3_expressions.io.IScanner.IScanner) → List[pip_services3_expressions.tokenizers.Token.Token] Tokenizes a textual stream into a list of token structures.
- Parameters
scanner – A textual stream to be tokenized.
- Returns
A list of token structures.
-
tokenize_stream_to_string(scanner: pip_services3_expressions.io.IScanner.IScanner) → List[str] Tokenizes a textual stream into a list of strings.
- Parameters
scanner – A textual stream to be tokenized.
- Returns
A list of token strings.
-