pip_services3_expressions.tokenizers.generic.GenericWhitespaceState module
-
class
pip_services3_expressions.tokenizers.generic.GenericWhitespaceState.GenericWhitespaceState Bases:
pip_services3_expressions.tokenizers.IWhitespaceState.IWhitespaceStateA whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer’s next token. By default, all characters from 0 to 32 are whitespace.
-
clear_whitespace_chars() Clears definitions of whitespace characters.
-
next_token(scanner: pip_services3_expressions.io.IScanner.IScanner, tokenizer: pip_services3_expressions.tokenizers.ITokenizer.ITokenizer) → pip_services3_expressions.tokenizers.Token.Token Ignore whitespace (such as blanks and tabs), and return the tokenizer’s next token.
- Parameters
scanner – A textual string to be tokenized.
tokenizer – A tokenizer class that controls the process.
- Returns
The next token from the top of the stream.
-
set_whitespace_chars(from_symbol: int, to_symbol: int, enable: bool) Establish the given characters as whitespace to ignore.
- Parameters
from_symbol – First character index of the interval.
to_symbol – Last character index of the interval.
enable – True if this state should ignore characters in the given range.
-