flair.tokenization.SegtokTokenizer

class flair.tokenization.SegtokTokenizerView on GitHub

Bases: Tokenizer

Tokenizer using segtok, a third party library dedicated to rules-based Indo-European languages.

For further details see: fnl/segtok

__init__()View on GitHub

Methods

__init__()
run_tokenize(text)
tokenize(text)

Attributes

name

tokenize(text)View on GitHub

Return type:

list[str]

static run_tokenize(text)View on GitHub

Return type:

list[str]