= wiki2corpus =

* downloads articles from Wikipedia for a given language id (URL prefix)
* works with Wikipedia API (HTML output) as it is not straightforward to turn mediawiki syntax into plain text
* HTML files are converted into plain text using jusText
* title, categories, translations, number of paragraphs and number of chars are put as attributes to `<doc>` structure

== Requirements ==

* [wiki:Justext jusText]

== Get wiki2corpus ==

See [wiki:Downloads] for the latest version.

== Usage ==

{{{
usage: wikidownloader.py [-h] [--cache CACHE] [--wait WAIT] [--newest]
                         [--links LINKS]
                         langcode

Wikipedia downloader

positional arguments:
  langcode       Wikipedia language prefix

optional arguments:
  -h, --help     show this help message and exit
  --cache CACHE  Directory with previously downloaded pages and data
  --wait WAIT    Time interval between GET requests
  --newest       Download the newest versions of articles (do not use cache)
  --links LINKS  Gather external links from Wikipedia (Reference section)
}}}

== Example ==

Let us say you want to download Belarusian Wikipedia. The ISO 639-1 language code is "be", so you want to download articles from be.wikipedia.org. You can use this command:

{{{
python wikidownloader.py be --wait 7 --links bewiki.links > bewiki.prevert
}}}

The `bewiki.prevert` file can be used to feed a pipeline for following processing of the data. It is in a simple XML-like format with element `<doc>`.

You may e.g. tokenize the prevertical with this script (`wikipipe.sh`):

{{{
#!/bin/bash

LANG=$1

if [ -d "${LANG}wiki.cache" ]; then
    CACHE="--cache ${LANG}wiki.cache"
else
    CACHE=""
fi

python wikidownloader.py $LANG $CACHE --links ${LANG}wiki.links |\
unitok --trim 200 /usr/lib/python2.7/site-packages/unitok/configs/other_2.py |\
onion -m -s -n 7 -t 0.7 -d doc 2> /dev/null |\
xz - > ${LANG}wiki.vert.xz
}}}

It can be invoked with the command `bash wikipipe.sh be`. Both required tools are available too: [wiki:Unitok unitok], [wiki:Onion onion].

== Licence ==

wiki2corpus is licensed under [http://choosealicense.com/licenses/mit/ MIT licence]