This is a joint portal of the [http://nlp.fi.muni.cz Masaryk University's NLP Centre] and [http://www.sketchengine.co.uk Lexical Computing] dedicated to a number of software tools for corpus processing.

If you have any questions or suggestions, please subscribe to the [https://groups.google.com/a/sketchengine.co.uk/forum/#!forum/noske NoSketch Engine] Google group, where you can get involved in the discussion with the developers and other users.

{{{
#!html
<table style="border-spacing: 1em"><tr>

<td class="app" style="background-color:#000080 ; background-image:url('/chrome/site/justext_nb.png')">
<p><a href="/wiki/Justext">
JusText is a HTML boilerplate removal tool. It can strip navigation links, headers, footers, etc. from HTML pages and leave just regular text containing full sentences.</a><p>
<p>
<a class="lnk" href="http://is.muni.cz/th/45523/fi_d/phdthesis.pdf">Paper</a>
|
<a class="lnk" href="/wiki/Justext/Cite">Cite</a>
|
<a class="lnk" href="http://opensource.org/licenses/BSD-3-Clause">Licence</a>
</p>
</td>

<td class="app" style="background-color:#800000 ; background-image:url('/chrome/site/chared_nb.png')">
<p><a href="/wiki/Chared">
Chared is a tool for detecting the character encoding of a text in a known language. It contains models for a wide range of languages.</a><p>
<p>
<a class="lnk" href="#">Paper</a>
|
<a class="lnk" href="/wiki/Chared/Cite">Cite</a>
|
<a class="lnk" href="http://opensource.org/licenses/BSD-3-Clause">Licence</a>
</p>
</td>

</tr><tr>

<td class="app" style="background-color:#800080 ; background-image:url('/chrome/site/spiderling_nb.png')">
<p><a href="/wiki/SpiderLing">Spiderling is a web spider for linguistics. It can crawl text-rich parts of the web and collect a lot of data suitable for text corpora.
</a><p>
<p>
<a class="lnk" href="http://nlp.fi.muni.cz/~xsuchom2/papers/PomikalekSuchomel_SpiderlingEfficiency.pdf">Paper</a>
|
<a class="lnk" href="/wiki/SpiderLing/Cite">Cite</a>
|
<a class="lnk" href="http://www.gnu.org/licenses/gpl.txt">Licence</a>
</p>
</td>

<td class="app" style="background-color:#008000 ; background-image:url('/chrome/site/onion_nb.png')">
<p><a href="/wiki/Onion">
Onion (ONe Instance ONly) is a de-duplicator for large collections of texts. It can measure the similarity of paragraphs or whole documents and drop duplicate ones based on the threshold you set.</a></p>
<p>
<a class="lnk" href="http://is.muni.cz/th/45523/fi_d/phdthesis.pdf">Paper</a>
|
<a class="lnk" href="/wiki/Onion/Cite">Cite</a>
|
<a class="lnk" href="http://opensource.org/licenses/BSD-3-Clause">Licence</a>
</p>
</td>

</tr><tr>

<td class="app" style="background-color:#808000 ; background-image:url('/chrome/site/unitok_nb.png')">
<p><a href="/wiki/Unitok">
Unitok is a universal text tokeniser with specific settings for many languages. It can turn plain text into a sequence of newline-separated tokens (“vertical” format), while preserving XML-like tags containing metadata.</a></p>
<p>
<a class="lnk" href="http://nlp.fi.muni.cz/raslan/raslan14.pdf#page=79">Paper</a>
|
<a class="lnk" href="/wiki/Unitok/Cite">Cite</a>
|
<a class="lnk" href="https://www.mozilla.org/MPL/2.0/">Licence</a>
</p>
</td>

<td class="app" style="background-color:#008080 ; background-image:url('/chrome/site/noske_nb.png')">
<p><a href="http://nlp.fi.muni.cz/trac/noske">NoSketch Engine is the open-sourced little brother of the corpus querying system Sketch Engine.
</a><p>
<p>
<a class="lnk" href="https://link.springer.com/article/10.1007%2Fs40607-014-0009-9">Paper</a>
|
<a class="lnk" href="/wiki/noske_cite">Cite</a>
|
<a class="lnk" href="http://www.gnu.org/licenses/gpl2.txt">Licence</a>
</p>
</td>

</tr>
<tr>

<td class="app black" style="background-color:#a7d7f9; background-image: url('/chrome/site/w2c_44.png');">
<p><a href="/wiki/wiki2corpus">wiki2corpus is a script which downloads Wikipedia articles (for a given language) and outputs them in the form of prevertical which can be further processed by other corpus tools.
</a><p>
</td>

<td class="app black" style="background-color:#ff1493; background-image: url('/chrome/site/noske_nb.png');">
<p><a href="/wiki/languagefilter">Language Filter is a language discriminating tool. It works with the vertical format. The language of paragraphs and documents is determined according to pre-defined lists of words with corpus frequency.
</a><p>
<p>
<a class="lnk" href="https://nlp.fi.muni.cz/raslan/raslan19.pdf#page=137">Paper</a>
|
<a class="lnk" href="/wiki/languagefilter/Cite">Cite</a>
|
<a class="lnk" href="https://www.mozilla.org/MPL/2.0/">Licence</a>
</p>
</td>

</tr></table>
}}}