flair.datasets.sequence_labeling#

class flair.datasets.sequence_labeling.MultiFileJsonlCorpus(train_files=None, test_files=None, dev_files=None, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', use_tokenizer=True, **corpusargs)View on GitHub #

Bases: Corpus

This class represents a generic Jsonl corpus with multiple train, dev, and test files.

__init__(train_files=None, test_files=None, dev_files=None, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', use_tokenizer=True, **corpusargs)View on GitHub #

Instantiates a MuliFileJsonlCorpus as, e.g., created with doccanos JSONL export.

Note that at least one of train_files, test_files, and dev_files must contain one path. Otherwise, the initialization will fail.

Parameters:

corpusargs – Additional arguments for Corpus initialization
train_files – the name of the train files
test_files – the name of the test files
dev_files – the name of the dev files, if empty, dev data is sampled from train
encoding (str) – file encoding (default “utf-8”)
text_column_name (str) – Name of the text column inside the jsonl files.
label_column_name (str) – Name of the label column inside the jsonl files.
metadata_column_name (str) – Name of the metadata column inside the jsonl files.
label_type (str) – he type of label to predict (default “ner”)
use_tokenizer (Union[bool, Tokenizer]) – Specify a custom tokenizer to split the text into tokens.

Raises:

RuntimeError – If no paths are given

class flair.datasets.sequence_labeling.JsonlCorpus(data_folder, train_file=None, test_file=None, dev_file=None, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', autofind_splits=True, name=None, use_tokenizer=True, **corpusargs)View on GitHub #

Bases: MultiFileJsonlCorpus

__init__(data_folder, train_file=None, test_file=None, dev_file=None, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', autofind_splits=True, name=None, use_tokenizer=True, **corpusargs)View on GitHub #

Instantiates a JsonlCorpus with one file per Dataset (train, dev, and test).

Parameters:

data_folder (Union[str, Path]) – Path to the folder containing the JSONL corpus
train_file (Union[str, Path, None]) – the name of the train file
test_file (Union[str, Path, None]) – the name of the test file
dev_file (Union[str, Path, None]) – the name of the dev file, if None, dev data is sampled from train
encoding (str) – file encoding (default “utf-8”)
text_column_name (str) – Name of the text column inside the JSONL file.
label_column_name (str) – Name of the label column inside the JSONL file.
metadata_column_name (str) – Name of the metadata column inside the JSONL file.
label_type (str) – The type of label to predict (default “ner”)
autofind_splits (bool) – Whether train, test and dev file should be determined automatically
name (Optional[str]) – name of the Corpus see flair.data.Corpus
use_tokenizer (Union[bool, Tokenizer]) – Specify a custom tokenizer to split the text into tokens.

class flair.datasets.sequence_labeling.JsonlDataset(path_to_jsonl_file, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', use_tokenizer=True)View on GitHub #

Bases: FlairDataset

__init__(path_to_jsonl_file, encoding='utf-8', text_column_name='data', label_column_name='label', metadata_column_name='metadata', label_type='ner', use_tokenizer=True)View on GitHub #

Instantiates a JsonlDataset and converts all annotated char spans to token tags using the IOB scheme.

The expected file format is:

{
    "<text_column_name>": "<text>",
    "<label_column_name>": [[<start_char_index>, <end_char_index>, <label>],...],
    "<metadata_column_name>": [[<metadata_key>, <metadata_value>],...]
}

Parameters:

path_to_jsonl_file (Union[str, Path]) – File to read
encoding (str) – file encoding (default “utf-8”)
text_column_name (str) – Name of the text column
label_column_name (str) – Name of the label column
metadata_column_name (str) – Name of the metadata column
label_type (str) – The type of label to predict (default “ner”)
use_tokenizer (Union[bool, Tokenizer]) – Specify a custom tokenizer to split the text into tokens.

_add_label_to_sentence(text, sentence, start, end, label)View on GitHub #

Adds a NE label to a given sentence.

Parameters:

text (str) – raw sentence (with all whitespaces etc.). Is used to determine the token indices.
sentence (Sentence) – Tokenized flair Sentence.
start (int) – Start character index of the label.
end (int) – End character index of the label.
label (str) – Label to assign to the given range.

Returns:

Nothing. Changes sentence as INOUT-param

is_in_memory()View on GitHub #

Return type:: bool

class flair.datasets.sequence_labeling.MultiFileColumnCorpus(column_format, train_files=None, test_files=None, dev_files=None, column_delimiter='\\\\s+', comment_symbol=None, encoding='utf-8', document_separator_token=None, skip_first_line=False, in_memory=True, label_name_map=None, banned_sentences=None, default_whitespace_after=1, **corpusargs)View on GitHub #

Bases: Corpus

__init__(column_format, train_files=None, test_files=None, dev_files=None, column_delimiter='\\\\s+', comment_symbol=None, encoding='utf-8', document_separator_token=None, skip_first_line=False, in_memory=True, label_name_map=None, banned_sentences=None, default_whitespace_after=1, **corpusargs)View on GitHub #

Instantiates a Corpus from CoNLL column-formatted task data such as CoNLL03 or CoNLL2000.

Parameters:

data_folder – base folder with the task data
column_format (Dict[int, str]) – a map specifying the column format
train_files – the name of the train files
test_files – the name of the test files
dev_files – the name of the dev files, if empty, dev data is sampled from train
column_delimiter (str) – default is to split on any separatator, but you can overwrite for instance with “t” to split only on tabs
comment_symbol (Optional[str]) – if set, lines that begin with this symbol are treated as comments
encoding (str) – file encoding (default “utf-8”)
document_separator_token (Optional[str]) – If provided, sentences that function as document boundaries are so marked
skip_first_line (bool) – set to True if your dataset has a header line
in_memory (bool) – If set to True, the dataset is kept in memory as Sentence objects, otherwise does disk reads
label_name_map (Optional[Dict[str, str]]) – Optionally map tag names to different schema.
banned_sentences (Optional[List[str]]) – Optionally remove sentences from the corpus. Works only if in_memory is true

class flair.datasets.sequence_labeling.ColumnCorpus(data_folder, column_format, train_file=None, test_file=None, dev_file=None, autofind_splits=True, name=None, comment_symbol='# ', **corpusargs)View on GitHub #

Bases: MultiFileColumnCorpus

__init__(data_folder, column_format, train_file=None, test_file=None, dev_file=None, autofind_splits=True, name=None, comment_symbol='# ', **corpusargs)View on GitHub #

Instantiates a Corpus from CoNLL column-formatted task data such as CoNLL03 or CoNLL2000.

Parameters:

data_folder (Union[str, Path]) – base folder with the task data
column_format (Dict[int, str]) – a map specifying the column format
train_file – the name of the train file
test_file – the name of the test file
dev_file – the name of the dev file, if None, dev data is sampled from train
column_delimiter – default is to split on any separatator, but you can overwrite for instance with “t” to split only on tabs
comment_symbol – if set, lines that begin with this symbol are treated as comments
document_separator_token – If provided, sentences that function as document boundaries are so marked
skip_first_line – set to True if your dataset has a header line
in_memory – If set to True, the dataset is kept in memory as Sentence objects, otherwise does disk reads
label_name_map – Optionally map tag names to different schema.
banned_sentences – Optionally remove sentences from the corpus. Works only if in_memory is true

class flair.datasets.sequence_labeling.ColumnDataset(path_to_column_file, column_name_map, column_delimiter='\\\\s+', comment_symbol=None, banned_sentences=None, in_memory=True, document_separator_token=None, encoding='utf-8', skip_first_line=False, label_name_map=None, default_whitespace_after=1)View on GitHub #

Bases: FlairDataset

SPACE_AFTER_KEY = 'space-after'#

FEATS = ['feats', 'misc']#

HEAD = ['head', 'head_id']#

__init__(path_to_column_file, column_name_map, column_delimiter='\\\\s+', comment_symbol=None, banned_sentences=None, in_memory=True, document_separator_token=None, encoding='utf-8', skip_first_line=False, label_name_map=None, default_whitespace_after=1)View on GitHub #

Instantiates a column dataset.

Parameters:

path_to_column_file (Union[str, Path]) – path to the file with the column-formatted data
column_name_map (Dict[int, str]) – a map specifying the column format
column_delimiter (str) – default is to split on any separator, but you can overwrite for instance with “t” to split only on tabs
comment_symbol (Optional[str]) – if set, lines that begin with this symbol are treated as comments
in_memory (bool) – If set to True, the dataset is kept in memory as Sentence objects, otherwise does disk reads
document_separator_token (Optional[str]) – If provided, sentences that function as document boundaries are so marked
skip_first_line (bool) – set to True if your dataset has a header line
label_name_map (Optional[Dict[str, str]]) – Optionally map tag names to different schema.
banned_sentences (Optional[List[str]]) – Optionally remove sentences from the corpus. Works only if in_memory is true

is_in_memory()View on GitHub #

Return type:: bool

class flair.datasets.sequence_labeling.ONTONOTES(base_path=None, version='v4', language='english', domain=None, in_memory=True, **corpusargs)View on GitHub #

Bases: MultiFileColumnCorpus

archive_url = 'https://data.mendeley.com/public-files/datasets/zmycy7t9h9/files/b078e1c4-f7a4-4427-be7f-9389967831ef/file_downloaded'#

classmethod get_available_domains(base_path=None, version='v4', language='english', split='train')View on GitHub #

Return type:: List[str]

classmethod _process_coref_span_annotations_for_word(label, word_index, clusters, coref_stacks)View on GitHub #

For a given coref label, add it to a currently open span(s), complete a span(s) or ignore it, if it is outside of all spans.

This method mutates the clusters and coref_stacks dictionaries.

Parameters:

label (str) – The coref label for this word.
word_index (int) – The word index into the sentence.
clusters (DefaultDict[int, List[Tuple[int, int]]]) – A dictionary mapping cluster ids to lists of inclusive spans into the sentence.
coref_stacks (DefaultDict[int, List[int]]) – Stacks for each cluster id to hold the start indices of open spans. Spans with the same id can be nested, which is why we collect these opening spans on a stack, e.g: [Greg, the baker who referred to [himself]_ID1 as ‘the bread man’]_ID1

Return type:

None

classmethod dataset_document_iterator(file_path)View on GitHub #

An iterator over CONLL formatted files which yields documents, regardless of the number of document annotations in a particular file.

This is useful for conll data which has been preprocessed, such as the preprocessing which takes place for the 2012 CONLL Coreference Resolution task.

Return type:: Iterator[List]

classmethod sentence_iterator(file_path)View on GitHub #

An iterator over the sentences in an individual CONLL formatted file.

Return type:: Iterator

class flair.datasets.sequence_labeling.CONLL_03(base_path=None, column_format={0: 'text', 1: 'pos', 3: 'ner'}, in_memory=True, **corpusargs)View on GitHub #

Table of Contents

Versions

flair.datasets.sequence_labeling#