| | 266 | === Changelog 2.0 → 2.1 → 2.2 === |
| | 267 | Language detection: |
| | 268 | * CLD2 is used together with the simple character trigram model at the document level now. CLD2 is more accurate and it is able to identify languages that the old model does not know. The old model is more permissive but introduces other language texts that have to be filtered later. CLD2 => precision, simple trigrams => recall. |
| | 269 | * Language detection at the paragraph level was removed since a paragraph was too short for reliable predictions. |
| | 270 | * Lowercase and trim all strings for the trigram langid. |
| | 271 | Performance improvements: |
| | 272 | * Gzip compression of files to save space where performance is not crucial (i.e. in document processors, not in the downloader). |
| | 273 | * Split the ARC output to 100 GB parts, split the prevertical output to 10 GB parts to allow post-processing of closed parts even before the crawl is done. |
| | 274 | * When the document processors are overloaded, a limited number of documents from each domain is processed from a single wpage file (the rest is postponed) so for all domains there is some data getting steadily from processors to the scheduler not to stop crawling domains because of their perceived inactivity caused by overloaded processors, not by depleted domains. |
| | 275 | * Ignore edit pages and diff pages which usually contribute no new texts. |
| | 276 | * Limit the size of wpage processing queue not to eat up the memory by big crawls: config.MAX_WPAGE_PROCESSING_COUNT. |
| | 277 | Bugfixes: |
| | 278 | * Write the rest of duplicate doc IDs after termination. |
| | 279 | * Fixed a missing primary multilanguage flag in re-processing code. |
| | 280 | Other: |
| | 281 | * More domains in the default blacklist. |
| | 282 | * Pad prevertical file names to two digits to ease their processing and sorting. |
| | 283 | * Allow keeping arc output when re-processing wpage data. |
| | 284 | * Use --saved-hashes to initialize the hashes of raw data and plaintext by savepoint values when re-processing wpage data. |
| | 285 | * url/download_queue added to the restart command line to load a saved downloader queue. |
| | 286 | |
