flair.tokenization.SpaceTokenizer#

class flair.tokenization.SpaceTokenizerView on GitHub#

Bases: Tokenizer

Tokenizer based on space character only.

__init__()View on GitHub#

Methods

__init__()

from_dict(config)

Instantiate the tokenizer from a configuration dictionary.

run_tokenize(text)

to_dict()

Serialize the tokenizer's configuration to a dictionary.

tokenize(text)

Attributes

name

tokenize(text)View on GitHub#
Return type:

list[str]

static run_tokenize(text)View on GitHub#
Return type:

list[str]

to_dict()View on GitHub#

Serialize the tokenizer’s configuration to a dictionary.

Return type:

dict

classmethod from_dict(config)View on GitHub#

Instantiate the tokenizer from a configuration dictionary.

Return type:

SpaceTokenizer