flair.tokenization.SpaceTokenizer#
- class flair.tokenization.SpaceTokenizerView on GitHub#
Bases:
TokenizerTokenizer based on space character only.
- __init__()View on GitHub#
Methods
__init__()from_dict(config)Instantiate the tokenizer from a configuration dictionary.
run_tokenize(text)to_dict()Serialize the tokenizer's configuration to a dictionary.
tokenize(text)Attributes
name- tokenize(text)View on GitHub#
- Return type:
list[str]
- static run_tokenize(text)View on GitHub#
- Return type:
list[str]
- to_dict()View on GitHub#
Serialize the tokenizer’s configuration to a dictionary.
- Return type:
dict
- classmethod from_dict(config)View on GitHub#
Instantiate the tokenizer from a configuration dictionary.
- Return type: