flair.tokenization.JapaneseTokenizer#
- class flair.tokenization.JapaneseTokenizer(tokenizer, sudachi_mode='A')View on GitHub#
Bases:
Tokenizer
Tokenizer using konoha to support popular japanese tokenizers.
Tokenizer using konoha, a third party library which supports multiple Japanese tokenizer such as MeCab, Janome and SudachiPy.
- For further details see:
- __init__(tokenizer, sudachi_mode='A')View on GitHub#
Methods
__init__
(tokenizer[, sudachi_mode])tokenize
(text)Attributes
- tokenize(text)View on GitHub#
- Return type:
list
[str
]
- property name: str#