Changes between Version 5 and Version 6 of wiki2corpus

Timestamp:

12/07/16 17:09:44 (8 years ago)

Author:

admin

Comment:

--

wiki2corpus

@@ -4 +4 @@
 * works with Wikipedia API (HTML output) as it is not straightforward to turn mediawiki syntax into plain text
 * HTML files are converted into plain text using jusText
-* texts are tokenized using unitok
 * title, categories, translations, number of paragraphs and number of chars are put as attributes to `<doc>` structure
 
@@ -10 +9 @@
 
 * [wiki:Justext jusText]
-* [wiki:Unitok unitok]
 
 == Get wiki2corpus ==
@@ -46 +44 @@
 The `bewiki.prevert` file can be used to feed a pipeline for following processing of the data. It is in a simple XML-like format with element `<doc>`.
 
+You may e.g. tokenize the prevertical with this script (`wikipipe.sh`):
+
+{{{
+#!/bin/bash
+
+LANG=$1
+
+if [ -d "${LANG}wiki.cache" ]; then
+    CACHE="--cache ${LANG}wiki.cache"
+else
+    CACHE=""
+fi
+
+python wikidownloader.py $LANG $CACHE --links ${LANG}wiki.links |\
+unitok --trim 200 /usr/lib/python2.7/site-packages/unitok/configs/other_2.py |\
+onion -m -s -n 7 -t 0.7 -d doc 2> /dev/null |\
+xz - > ${LANG}wiki.vert.xz
+}}}
+
+It can be invoked with the command `bash wikipipe.sh be`. Both required tools are available too: [wiki:Unitok unitok], [wiki:Onion onion].
+
 == Licence ==