# omorfi-tokenize

## A hfst-pmatch script for compiling a tokenizer with OmorFi

`tokenize.txt` is the main tokenizer script for processing with `hfst-pmatch2fst`. It requires a compiled OmorFi morphology to be present.

`omorfi_additions.txt` and `proper_names.txt` are as yet small lists containing tokens that should arguably be in OmorFi or in a gazetteer of proper names, respectively.
