= unitok =

* splits input text into tokens (one token per line)
* recognizes URLs, e-mail addreses, DNS domains, IP addresses
* for specified languages recognizes abbreviations and clictics (such as 've or n't in English)
* preserves XML-like tags
* replaces entities with unicode equivalents
* adds glue (<g/>) tags between tokens not separated by space

{{{
#!html
<a class="lnk" href="http://nlp.fi.muni.cz/raslan/raslan14.pdf#page=79">Paper</a>
|
<a class="lnk" href="/wiki/Unitok/Cite">Cite</a>
|
<a class="lnk" href="https://www.mozilla.org/MPL/2.0/">Licence</a>
}}}

== Get unitok ==
See [wiki:Downloads] for the latest version.


== Licence ==
Unitok is licensed under [https://www.mozilla.org/MPL/2.0 Mozilla Public License Version 2.0].