Changes between Version 3 and Version 4 of wiki2corpus

Timestamp:

12/07/16 16:55:17 (9 years ago)

Author:

admin

Comment:

--

wiki2corpus

@@ -5 +5 @@
 * HTML files are converted into plain text using jusText
 * texts are tokenized using unitok
+* title, categories, translations, number of paragraphs and number of chars are put as attributes to ```<doc>``` structure
 
 == Requirements ==
@@ -15 +16 @@
 See [wiki:Downloads] for the latest version.
 
+== Usage ==
+
+{{{
+usage: wikidownloader.py [-h] [--cache CACHE] [--wait WAIT] [--newest]
+                         [--links LINKS]
+                         langcode
+
+Wikipedia downloader
+
+positional arguments:
+  langcode       Wikipedia language prefix
+
+optional arguments:
+  -h, --help     show this help message and exit
+  --cache CACHE  Directory with previously downloaded pages and data
+  --wait WAIT    Time interval between GET requests
+  --newest       Download the newest versions of articles (do not use cache)
+  --links LINKS  Gather external links from Wikipedia (Reference section)
+}}}
+
+== Example ==
+
+Let us say you want to download fr.wikipedia.org. You can use this command:
+{{{
+python wikidownloader.py be --wait 7 --links bewiki.links > bewiki.prevert
+}}}
+
+The ```.prevert``` file can be used to feed a pipeline for following processing of the data.
 
 == Licence ==