flair.tokenization.NoTokenizer

class flair.tokenization.NoTokenizerView on GitHub

Bases: Tokenizer

A dummy tokenizer that performs no tokenization. It returns the original text as a single token in a list, or an empty list if the text is empty or whitespace. Useful when text is pre-tokenized or to disable tokenization.

__init__()View on GitHub

Methods

__init__()
from_dict(config)	Instantiate the tokenizer from a configuration dictionary.
to_dict()	Serialize the tokenizer's configuration to a dictionary.
tokenize(text)	Returns the text as a single token if not empty/whitespace, otherwise returns an empty list.

Attributes

name

tokenize(text)View on GitHub

Returns the text as a single token if not empty/whitespace, otherwise returns an empty list.

Return type:

list[str]

to_dict()View on GitHub

Serialize the tokenizer’s configuration to a dictionary.

Return type:

dict

classmethod from_dict(config)View on GitHub

Instantiate the tokenizer from a configuration dictionary.

Return type:

NoTokenizer