flair.datasets.document_classification#

class flair.datasets.document_classification.ClassificationCorpus(data_folder, label_type='class', train_file=None, test_file=None, dev_file=None, truncate_to_max_tokens=-1, truncate_to_max_chars=-1, filter_if_longer_than=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, memory_mode='partial', label_name_map=None, skip_labels=None, allow_examples_without_labels=False, sample_missing_splits=True, encoding='utf-8')View on GitHub #

Bases: Corpus

A classification corpus from FastText-formatted text files.

__init__(data_folder, label_type='class', train_file=None, test_file=None, dev_file=None, truncate_to_max_tokens=-1, truncate_to_max_chars=-1, filter_if_longer_than=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, memory_mode='partial', label_name_map=None, skip_labels=None, allow_examples_without_labels=False, sample_missing_splits=True, encoding='utf-8')View on GitHub #

Instantiates a Corpus from text classification-formatted task data.

Parameters:

data_folder (Union[str, Path]) – base folder with the task data
label_type (str) – name of the label
train_file – the name of the train file
test_file – the name of the test file
dev_file – the name of the dev file, if None, dev data is sampled from train
truncate_to_max_tokens (int) – If set, truncates each Sentence to a maximum number of tokens
truncate_to_max_chars (int) – If set, truncates each Sentence to a maximum number of chars
filter_if_longer_than (int) – If set, filters documents that are longer that the specified number of tokens.
tokenizer (Union[bool, Tokenizer]) – Tokenizer for dataset, default is SegtokTokenizer
memory_mode (str) – Set to what degree to keep corpus in memory (‘full’, ‘partial’ or ‘disk’). Use ‘full’ if full corpus and all embeddings fits into memory for speedups during training. Otherwise use ‘partial’ and if even this is too much for your memory, use ‘disk’.
label_name_map (Optional[dict[str, str]]) – Optionally map label names to different schema.
allow_examples_without_labels – set to True to allow Sentences without label in the corpus.
encoding (str) – Default is ‘utf-8’ but some datasets are in ‘latin-1

class flair.datasets.document_classification.ClassificationDataset(path_to_file, label_type, truncate_to_max_tokens=-1, truncate_to_max_chars=-1, filter_if_longer_than=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, memory_mode='partial', label_name_map=None, skip_labels=None, allow_examples_without_labels=False, encoding='utf-8')View on GitHub #

Bases: FlairDataset

Dataset for classification instantiated from a single FastText-formatted file.

__init__(path_to_file, label_type, truncate_to_max_tokens=-1, truncate_to_max_chars=-1, filter_if_longer_than=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, memory_mode='partial', label_name_map=None, skip_labels=None, allow_examples_without_labels=False, encoding='utf-8')View on GitHub #

Reads a data file for text classification.

The file should contain one document/text per line. The line should have the following format: __label__<class_name> <text> If you have a multi class task, you can have as many labels as you want at the beginning of the line, e.g., __label__<class_name_1> __label__<class_name_2> <text> :type path_to_file: Union[str, Path] :param path_to_file: the path to the data file :type label_type: str :param label_type: name of the label :type truncate_to_max_tokens: :param truncate_to_max_tokens: If set, truncates each Sentence to a maximum number of tokens :type truncate_to_max_chars: :param truncate_to_max_chars: If set, truncates each Sentence to a maximum number of chars :type filter_if_longer_than: int :param filter_if_longer_than: If set, filters documents that are longer that the specified number of tokens. :type tokenizer: Union[bool, Tokenizer] :param tokenizer: Custom tokenizer to use (default is SegtokTokenizer) :type memory_mode: str :param memory_mode: Set to what degree to keep corpus in memory (‘full’, ‘partial’ or ‘disk’). Use ‘full’ if full corpus and all embeddings fits into memory for speedups during training. Otherwise use ‘partial’ and if even this is too much for your memory, use ‘disk’. :type label_name_map: Optional[dict[str, str]] :param label_name_map: Optionally map label names to different schema. :type allow_examples_without_labels: :param allow_examples_without_labels: set to True to allow Sentences without label in the Dataset. :type encoding: str :param encoding: Default is ‘utf-8’ but some datasets are in ‘latin-1 :return: list of sentences

is_in_memory()View on GitHub #

Return type:: bool

class flair.datasets.document_classification.CSVClassificationCorpus(data_folder, column_name_map, label_type, name='csv_corpus', train_file=None, test_file=None, dev_file=None, max_tokens_per_doc=-1, max_chars_per_doc=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, in_memory=False, skip_header=False, encoding='utf-8', no_class_label=None, sample_missing_splits=True, **fmtparams)View on GitHub #

Bases: Corpus

Classification corpus instantiated from CSV data files.

__init__(data_folder, column_name_map, label_type, name='csv_corpus', train_file=None, test_file=None, dev_file=None, max_tokens_per_doc=-1, max_chars_per_doc=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, in_memory=False, skip_header=False, encoding='utf-8', no_class_label=None, sample_missing_splits=True, **fmtparams)View on GitHub #

Instantiates a Corpus for text classification from CSV column formatted data.

Parameters:

data_folder (Union[str, Path]) – base folder with the task data
column_name_map (dict[int, str]) – a column name map that indicates which column is text and which the label(s)
label_type (str) – name of the label
train_file – the name of the train file
test_file – the name of the test file
dev_file – the name of the dev file, if None, dev data is sampled from train
max_tokens_per_doc – If set, truncates each Sentence to a maximum number of Tokens
max_chars_per_doc – If set, truncates each Sentence to a maximum number of chars
tokenizer (Tokenizer) – Tokenizer for dataset, default is SegtokTokenizer
in_memory (bool) – If True, keeps dataset as Sentences in memory, otherwise only keeps strings
skip_header (bool) – If True, skips first line because it is header
encoding (str) – Default is ‘utf-8’ but some datasets are in ‘latin-1
fmtparams – additional parameters for the CSV file reader

Returns:

a Corpus with annotated train, dev and test data

class flair.datasets.document_classification.CSVClassificationDataset(path_to_file, column_name_map, label_type, max_tokens_per_doc=-1, max_chars_per_doc=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, in_memory=True, skip_header=False, encoding='utf-8', no_class_label=None, **fmtparams)View on GitHub #

Bases: FlairDataset

Dataset for text classification from CSV column formatted data.

__init__(path_to_file, column_name_map, label_type, max_tokens_per_doc=-1, max_chars_per_doc=-1, tokenizer=<flair.tokenization.SegtokTokenizer object>, in_memory=True, skip_header=False, encoding='utf-8', no_class_label=None, **fmtparams)View on GitHub #

Instantiates a Dataset for text classification from CSV column formatted data.

Parameters:

path_to_file (Union[str, Path]) – path to the file with the CSV data
column_name_map (dict[int, str]) – a column name map that indicates which column is text and which the label(s)
label_type (str) – name of the label
max_tokens_per_doc (int) – If set, truncates each Sentence to a maximum number of Tokens
max_chars_per_doc (int) – If set, truncates each Sentence to a maximum number of chars
tokenizer (Tokenizer) – Tokenizer for dataset, default is SegTokTokenizer
in_memory (bool) – If True, keeps dataset as Sentences in memory, otherwise only keeps strings
skip_header (bool) – If True, skips first line because it is header
encoding (str) – Most datasets are ‘utf-8’ but some are ‘latin-1’
fmtparams – additional parameters for the CSV file reader

Returns:

a Corpus with annotated train, dev and test data

is_in_memory()View on GitHub #

Return type:: bool

class flair.datasets.document_classification.AMAZON_REVIEWS(split_max=30000, label_name_map={'1.0': 'NEGATIVE', '2.0': 'NEGATIVE', '3.0': 'NEGATIVE', '4.0': 'POSITIVE', '5.0': 'POSITIVE'}, skip_labels=['3.0', '4.0'], fraction_of_5_star_reviews=10, tokenizer=<flair.tokenization.SegtokTokenizer object>, memory_mode='partial', **corpusargs)View on GitHub #

Table of Contents

Versions

flair.datasets.document_classification#