Changes between Version 12 and Version 13 of SpiderLing

Timestamp:

07/23/21 14:06:34 (3 years ago)

Author:

admin

Comment:

version 2.0

SpiderLing

@@ -16 +16 @@
 
 == Get SpiderLing ==
-Download [https://nlp.fi.muni.cz/projects/spiderling/ the latest version]. Please note the software is distributed as is, without a guaranteed support.
+Download [https://corpus.tools/raw-attachment/wiki/Downloads/spiderling-src-2.0.tar.xz the latest version]. Please note the software is distributed as is, without a guaranteed support.
 
 == Publications ==
@@ -75 +75 @@
   - pdftotext (from poppler-utils),
   - ps2ascii (from ghostscript-core),
-  - antiword (from antiword),
+  - antiword (from antiword + perl-Time-HiRes),
+  - odfpy,
 - nice (coreutils) (optional),
 - ionice (util-linux) (optional),
@@ -88 +89 @@
 Recommended hardware configuration (crawling ~30 bn words of English text):
 - 8-32 core CPU (the more CPUs the faster the processing of crawled data),
-- 32-256 GB system memory
+- 32-512 GB system memory
     (the more RAM the more domains kept in memory and thus more webs visited),
 - lots of storage space,
@@ -105 +106 @@
 
 == Installation ==
-- unpack,
+- unpack: tar -xJvf spiderling-src-*.tar.xz,
 - install required tools, see install_rpm.sh for rpm based systems
 - check importing the following dependences by pypy3/python3:
@@ -119 +120 @@
 - raise ulimit -n accoring to MAX_OPEN_CONNS;
 - then increase MAX_OPEN_CONNS and OPEN_AT_ONCE;
-- configure language dependent settings.
+- configure language and TLD dependent settings.
 
 == Language models for all recognised languages ==