flair.tokenization.SpacyTokenizer

class flair.tokenization.SpacyTokenizer(model)View on GitHub

Bases: Tokenizer

Tokenizer using spacy under the hood.

Parameters:

model – a Spacy V2 model or the name of the model to load.

__init__(model)View on GitHub

Methods

__init__(model)
from_dict(config)	Instantiate the tokenizer from a configuration dictionary.
to_dict()	Serialize the tokenizer's configuration to a dictionary.
tokenize(text)

Attributes

name

tokenize(text)View on GitHub

Return type:

list[str]

property name: str

to_dict()View on GitHub

Serialize the tokenizer’s configuration to a dictionary.

Return type:

dict

classmethod from_dict(config)View on GitHub

Instantiate the tokenizer from a configuration dictionary.

Return type:

SpacyTokenizer