= Chared =

Chared is a tool for detecting the character encoding of a text in a known language. The language of the text has to be specified as an input parameter so that correspondent language model can be used. The package contains models for a wide range of languages. In general, it should be more accurate than character encoding detection algorithms with no language constraints.

== Installation ==
1. Make sure you have Python 2.6 or later and lxml library version 2.2.4 or later installed.
2. Download the sources:
{{{
wget http://chared.googlecode.com/files/chared-1.2.tar.gz
}}}
3. Extract the downloaded file:
{{{
tar xzvf chared-1.2.tar.gz
}}}
4. Install the package (you may need sudo or a root shell for the latter command):
{{{
cd chared-1.2/
python setup.py install
}}}

== Quick start ==
Detect the character encoding for a file or URL:
{{{
chared -m czech http://nlp.fi.muni.cz/cs/nlplab
}}}
Create a custom character encoding detection model from a collection of HTML pages (e.g. for Swahili):
{{{
chared-learn -o swahili.edm swahili_pages/*.html
}}}
... or if you have a sample text in Swahili (plain text, UTF-8) and want to apply language filtering on the input HTML files (recommended):
{{{
chared-learn -o swahili.edm -S swahili_sample.txt swahili_pages/*.html
}}}
For usage information see:
{{{
chared --help
chared-learn --help
}}}

== Python API ==

{{{
>>> import urllib2
>>> import chared.detector
>>> page = urllib2.urlopen('http://nlp.fi.muni.cz/cs/nlplab').read()
>>> cz_model_path = chared.detector.get_model_path('czech')
>>> cz_model = chared.detector.EncodingDetector.load(cz_model_path)
>>> cz_model.classify(page)
['utf_8']
}}}

== Acknowledgements ==
This software has been developed at the [http://nlp.fi.muni.cz/en/nlpc Natural Language Processing Centre] of [http://www.muni.cz/ Masaryk University in Brno] with financial support from [http://presemt.eu PRESEMT] and [http://www.sketchengine.co.uk Lexical Computing Ltd.]

== See also ==
[http://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html Unicode over 60 percent of the web] at Google blog