flair.tokenization

JapaneseTokenizer	Tokenizer using konoha to support popular japanese tokenizers.
SciSpacyTokenizer	Tokenizer that uses the en_core_sci_sm Spacy model and some special heuristics.
SegtokTokenizer	Tokenizer using segtok, a third party library dedicated to rules-based Indo-European languages.
SpaceTokenizer	Tokenizer based on space character only.
SpacyTokenizer	Tokenizer using spacy under the hood.
Tokenizer	An abstract class representing a Tokenizer.
TokenizerWrapper	Helper class to wrap tokenizer functions to the class-based tokenizer interface.