Changes between Initial Version and Version 1 of Justext

Timestamp:

01/12/15 18:19:57 (9 years ago)

Author:

admin

Comment:

--

Justext

@@ +1 @@
+= justext = 
+
+jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora.
+
+== What's new== 
+Mišo Belica created a [https://github.com/miso-belica/jusText jusText fork on GitHub] with some tweaks.
+
+jusText is now also [https://pypi.python.org/pypi/jusText available on PyPi].
+
+[http://corpus.tools/browser/justext/CHANGES Changelog]
+
+== Installation ==
+1. Make sure you have Python and lxml library version 2.2.4 or later installed.
+2. Download the sources:
+{{{
+wget http://corpus.tools/attachment/wiki/Downloads/justext-1.2.tar.gz
+}}}
+3. Extract the downloaded file:
+{{{
+tar xzvf justext-1.2.tar.gz
+}}}
+4. Install the package (you may need sudo or a root shell for the latter command):
+{{{
+cd justext-1.2/
+python setup.py install
+}}}
+
+== Quick start ==
+{{
+wget -O page.html http://planet.python.org/
+justext -s English page.html > cleaned-page.txt
+}}
+For usage information see:
+{{{
+justext --help
+}}}
+
+== Python API ==
+{{{
+import urllib2
+import justext
+
+page = urllib2.urlopen('http://planet.python.org/').read()
+paragraphs = justext.justext(page, justext.get_stoplist('English'))
+for paragraph in paragraphs:
+    if paragraph['class'] == 'good':
+        print paragraph['text']
+}}}
+
+== Online demo ==
+[http://nlp.fi.muni.cz/projects/justext/]
+
+== Acknowledgements ==
+This software has been developed at the [http://nlp.fi.muni.cz/en/nlpc Natural Language Processing Centre] of [http://www.muni.cz/ Masaryk University in Brno] with financial support from [http://presemt.eu PRESEMT] and [http://www.sketchengine.co.uk Lexical Computing Ltd.] It also relates to Jan Pomikálek's [http://is.muni.cz/th/45523/fi_d/phdthesis.pdf PhD research].