wiki:Chared

Chared

Chared is a tool for detecting the character encoding of a text in a known language. The language of the text has to be specified as an input parameter so that correspondent language model can be used. The package contains models for a wide range of languages. In general, it should be more accurate than character encoding detection algorithms with no language constraints.

Installation

  1. Make sure you have Python 2.6 or later and lxml library version 2.2.4 or later installed.
  2. Download the sources:
    wget http://corpus.tools/raw-attachment/wiki/Downloads/chared-1.2.2.tar.gz
    
  3. Extract the downloaded file:
    tar xzvf chared-1.2.tar.gz
    
  4. Install the package (you may need sudo or a root shell for the latter command):
    cd chared-1.2/
    python setup.py install
    

Quick start

Detect the character encoding for a file or URL:

chared -m czech http://nlp.fi.muni.cz/cs/nlplab

Create a custom character encoding detection model from a collection of HTML pages (e.g. for Swahili):

chared-learn -o swahili.edm swahili_pages/*.html

... or if you have a sample text in Swahili (plain text, UTF-8) and want to apply language filtering on the input HTML files (recommended):

chared-learn -o swahili.edm -S swahili_sample.txt swahili_pages/*.html

For usage information see:

chared --help
chared-learn --help

Python API

>>> import urllib2
>>> import chared.detector
>>> page = urllib2.urlopen('http://nlp.fi.muni.cz/cs/nlplab').read()
>>> cz_model_path = chared.detector.get_model_path('czech')
>>> cz_model = chared.detector.EncodingDetector.load(cz_model_path)
>>> cz_model.classify(page)
['utf_8']

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with financial support from PRESEMT and Lexical Computing Ltd.

See also

Unicode over 60 percent of the web at Google blog

Last modified 2 years ago Last modified on Aug 6, 2015, 3:51:07 PM