Changes between Version 4 and Version 5 of wiki2corpus

Timestamp:

12/07/16 17:02:20 (8 years ago)

Author:

admin

Comment:

--

wiki2corpus

@@ -5 +5 @@
 * HTML files are converted into plain text using jusText
 * texts are tokenized using unitok
-* title, categories, translations, number of paragraphs and number of chars are put as attributes to ```<doc>``` structure
+* title, categories, translations, number of paragraphs and number of chars are put as attributes to `<doc>` structure
 
 == Requirements ==
@@ -38 +38 @@
 == Example ==
 
-Let us say you want to download fr.wikipedia.org. You can use this command:
+Let us say you want to download Belarusian Wikipedia. The ISO 639-1 language code is "be", so you want to download articles from be.wikipedia.org. You can use this command:
+
 {{{
 python wikidownloader.py be --wait 7 --links bewiki.links > bewiki.prevert
 }}}
 
-The ```.prevert``` file can be used to feed a pipeline for following processing of the data.
+The `bewiki.prevert` file can be used to feed a pipeline for following processing of the data. It is in a simple XML-like format with element `<doc>`.
 
 == Licence ==