wiki:Unitok

unitok

  • splits input text into tokens (one token per line)
  • recognizes URLs, e-mail addreses, DNS domains, IP addresses
  • for specified languages recognizes abbreviations and clictics (such as 've or n't in English)
  • preserves XML-like tags
  • replaces entities with unicode equivalents
  • adds glue (<g/>) tags between tokens not separated by space
Paper | Cite | Licence

Get unitok

See Downloads for the latest version.

Licence

Unitok is licensed under Mozilla Public License Version 2.0.

Last modified 18 months ago Last modified on Jun 18, 2020, 9:01:58 PM