pip_services3_expressions.tokenizers.IWordState module

class pip_services3_expressions.tokenizers.IWordState.IWordState

Bases: pip_services3_expressions.tokenizers.ITokenizerState.ITokenizerState, abc.ABC

Defines an interface for tokenizer state that processes words, identificators or keywords

clear_word_chars()

Clears definitions of word chars.

set_word_chars(from_symbol: int, to_symbol: int, enable: bool)

Establish characters in the given range as valid characters for part of a word after the first character. Note that the tokenizer must determine which characters are valid as the beginning character of a word.

Parameters

  • from_symbol – First character index of the interval.

  • to_symbol – Last character index of the interval.

  • enable – True if this state should use characters in the given range.