unitok is a universal text tokeniser

 

• ​unitok is a universal text tokenizer
• ​jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora.
• ​onion (ONe Instance ONly) is a tool for removing duplicate parts from large collections of texts.