= unitok = * splits input text into tokens (one token per line) * recognizes URLs, e-mail addreses, DNS domains, IP addresses * for specified languages recognizes abbreviations and clictics (such as 've or n't in English) * preserves XML-like tags * replaces entities with unicode equivalents * adds glue () tags between tokens not separated by space {{{ #!html Paper | Cite | Licence }}} == Get unitok == See [wiki:Downloads] for the latest version. == Licence == Unitok is licensed under [https://www.mozilla.org/MPL/2.0 Mozilla Public License Version 2.0].