flair.tokenization.Tokenizer

class flair.tokenization.TokenizerView on GitHub

Bases: ABC

An abstract class representing a Tokenizer.

Tokenizers are used to represent algorithms and models to split plain text into individual tokens / words. All subclasses should overwrite tokenize(), which splits the given plain text into tokens. Moreover, subclasses may overwrite name(), returning a unique identifier representing the tokenizer’s configuration.

__init__()

Methods

__init__()
tokenize(text)

Attributes

name

abstract tokenize(text)View on GitHub

Return type:

list[str]

property name: str