Other embeddings in Flair#
Flair supports many other embedding types. This section introduces these embeddings.
Note
We mostly train our models with either TransformerEmbeddings or FlairEmbeddings. The embeddings presented here might be useful
for specific use cases or for comparison purposes.
One-Hot Embeddings#
OneHotEmbeddings are embeddings that encode each word in a vocabulary as a one-hot vector, followed by an embedding
layer. These embeddings
thus do not encode any prior knowledge as do most other embeddings. They also differ in that they
require to see a vocabulary (vocab_dictionary) during instantiation. Such dictionary can be passed as an argument
during class initialization or constructed directly from a corpus with a OneHotEmbeddings.from_corpus method. The dictionary consists
of all unique tokens contained in the corpus plus an UNK token for all rare words.
You initialize these embeddings like this:
from flair.embeddings import OneHotEmbeddings
from flair.datasets import UD_ENGLISH
from flair.data import Sentence
# load a corpus
corpus = UD_ENGLISH()
# init embedding
embeddings = OneHotEmbeddings.from_corpus(corpus)
# create a sentence
sentence = Sentence('The grass is green .')
# embed words in sentence
embeddings.embed(sentence)
By default, the ‘text’ of a token (i.e. its lexical value) is one-hot encoded and the embedding layer has a dimensionality of 300. However, this layer is randomly initialized, meaning that these embeddings do not make sense unless they are trained in a task.
Vocabulary size#
By default, all words that occur in the corpus at least 3 times are part of the vocabulary. You can change
this using the min_freq parameter. For instance, if your corpus is very large you might want to set a
higher min_freq:
embeddings = OneHotEmbeddings.from_corpus(corpus, min_freq=10)
Embedding dimensionality#
By default, the embeddings have a dimensionality of 300. If you want to try higher or lower values, you can use the
embedding_length parameter:
embeddings = OneHotEmbeddings.from_corpus(corpus, embedding_length=100)
Byte Pair Embeddings#
BytePairEmbeddings are word embeddings that are precomputed on the subword-level. This means that they are able to
embed any word by splitting words into subwords and looking up their embeddings. BytePairEmbeddings were proposed
and computed by Heinzerling and Strube (2018) who found that they offer nearly the same accuracy as word embeddings, but at a fraction
of the model size. So they are a great choice if you want to train small models.
You initialize with a language code (275 languages supported), a number of ‘syllables’ (one of ) and a number of dimensions (one of 50, 100, 200 or 300). The following initializes and uses byte pair embeddings for English:
from flair.embeddings import BytePairEmbeddings
# init embedding
embedding = BytePairEmbeddings('en')
# create a sentence
sentence = Sentence('The grass is green .')
# embed words in sentence
embedding.embed(sentence)
More information can be found on the byte pair embeddings web page.
BytePairEmbeddings also have a multilingual model capable of embedding any word in any language.
You can instantiate it with:
# init embedding
embedding = BytePairEmbeddings('multi')
You can also load custom BytePairEmbeddings by specifying a path to model_file_path and embedding_file_path arguments. They correspond respectively to a SentencePiece model file and to an embedding file (Word2Vec plain text or GenSim binary). For example:
# init custom embedding
embedding = BytePairEmbeddings(model_file_path='your/path/m.model', embedding_file_path='your/path/w2v.txt')
Document Pool Embeddings#
DocumentPoolEmbeddings calculate a pooling operation over all word embeddings in a document.
The default operation is mean which gives us the mean of all words in the sentence.
The resulting embedding is taken as document embedding.
To create a mean document embedding simply create any number of TokenEmbeddings first and put them in a list.
Afterwards, initiate the DocumentPoolEmbeddings with this list of TokenEmbeddings.
So, if you want to create a document embedding using GloVe embeddings together with FlairEmbeddings,
use the following code:
from flair.embeddings import WordEmbeddings, DocumentPoolEmbeddings
# initialize the word embeddings
glove_embedding = WordEmbeddings('glove')
# initialize the document embeddings, mode = mean
document_embeddings = DocumentPoolEmbeddings([glove_embedding])
Now, create an example sentence and call the embedding’s embed() method.
# create an example sentence
sentence = Sentence('The grass is green . And the sky is blue .')
# embed the sentence with our document embedding
document_embeddings.embed(sentence)
# now check out the embedded sentence.
print(sentence.embedding)
This prints out the embedding of the document. Since the document embedding is derived from word embeddings, its dimensionality depends on the dimensionality of word embeddings you are using.
You have the following optional constructor arguments:
Argument |
Default |
Description |
|---|---|---|
|
|
One of |
|
|
One of |
Pooling operation#
Next to the mean pooling operation you can also use min or max pooling. Simply pass the pooling operation you want
to use to the initialization of the DocumentPoolEmbeddings:
document_embeddings = DocumentPoolEmbeddings([glove_embedding], pooling='min')
Fine-tune mode#
You can also choose which fine-tuning operation you want, i.e. which transformation to apply before word embeddings get pooled. The default operation is ‘linear’ transformation, but if you only use simple word embeddings that are not task-trained you should probably use a ‘nonlinear’ transformation instead:
# instantiate pre-trained word embeddings
embeddings = WordEmbeddings('glove')
# document pool embeddings
document_embeddings = DocumentPoolEmbeddings([embeddings], fine_tune_mode='nonlinear')
If on the other hand you use word embeddings that are task-trained (such as simple one hot encoded embeddings), you are often better off doing no transformation at all. Do this by passing ‘none’:
# instantiate one-hot encoded word embeddings
embeddings = OneHotEmbeddings(corpus)
# document pool embeddings
document_embeddings = DocumentPoolEmbeddings([embeddings], fine_tune_mode='none')
Document RNN Embeddings#
Besides simple pooling we also support a method based on an RNN to obtain a DocumentEmbeddings.
The RNN takes the word embeddings of every token in the document as input and provides its last output state as document
embedding. You can choose which type of RNN you wish to use.
In order to use the DocumentRNNEmbeddings you need to initialize them by passing a list of token embeddings to it:
from flair.embeddings import WordEmbeddings, DocumentRNNEmbeddings
glove_embedding = WordEmbeddings('glove')
document_embeddings = DocumentRNNEmbeddings([glove_embedding])
By default, a GRU-type RNN is instantiated. Now, create an example sentence and call the embedding’s embed() method.
# create an example sentence
sentence = Sentence('The grass is green . And the sky is blue .')
# embed the sentence with our document embedding
document_embeddings.embed(sentence)
# now check out the embedded sentence.
print(sentence.get_embedding())
This will output a single embedding for the complete sentence. The embedding dimensionality depends on the number of hidden states you are using and whether the RNN is bidirectional or not.
RNN type#
If you want to use a different type of RNN, you need to set the rnn_type parameter in the constructor. So,
to initialize a document RNN embedding with an LSTM, do:
from flair.embeddings import WordEmbeddings, DocumentRNNEmbeddings
glove_embedding = WordEmbeddings('glove')
document_lstm_embeddings = DocumentRNNEmbeddings([glove_embedding], rnn_type='LSTM')
Need to be trained on a task#
Note that while DocumentPoolEmbeddings are immediately meaningful, DocumentRNNEmbeddings need to be tuned on the
downstream task. This happens automatically in Flair if you train a new model with these embeddings.
Once the model is trained, you can access the tuned DocumentRNNEmbeddings object directly from the classifier object and use it to embed sentences.
document_embeddings = classifier.document_embeddings
sentence = Sentence('The grass is green . And the sky is blue .')
document_embeddings.embed(sentence)
print(sentence.get_embedding())
DocumentRNNEmbeddings have a number of hyperparameters that can be tuned, please take a look at their API docs to find out more.