Lande

A tiny neural network for natural language detection.

This model is trained with data from the Tatoeba project, so it works well on short inputs too.

Model

Architecture:

• A very simple fully connected neural network with 1 hidden layer is trained, with character ngrams as input, and supported languages as output.

Details:

1. Character ngrams, at length 1, 2, 3, and 4, are extracted from the input string, with the assumption that with this data it should be possible to tell apart natural languages fairly reliably, if the input string is long enough.
2. Only a fixed number of the most frequent ngrams of supported languages are considered, which are extracted and stored during training, as they carry the most amount of signal overall, and the rest are simply discarded.
3. Compared to CLD3 ngrams are not hashed, as I think that kinda unpredictably confuses the model, as now different ngrams can have the same hash, so unimportant ngrams could be confused for important ones. Also that requires allocating a somewhat large key space for the hashes, while instead we want to be parsimonious with that, if we only need to remember 100 1-grams that's what we should be allocating memory for, but a small key space would get quickly crowded with colliding hashes, making it somewhat useless.
4. Compared to CLD3 no embedding layers are used here, as I think they are not needed with this approach because the model is being given clearer signals, since each input always corresponds to a specific ngram.
5. The value of each input neuron is the normalized frequency of the ngram corresponding to that neuron in the input string. Basically we tell the network if each supported ngram is present and how frequent it is.
6. To make a prediction the frequency of each supported input ngram is calculated, and a forward pass is performed.

Languages

The following 50 languages are supported.

If you want to experiment you can tweak some internal constants and add more languages, remove languages (which necessarily improves accuracy for the remaining ones), tweak hyper-parameters for the neural network etc.

ISO 639-3	Name
afr	Afrikaans
ara	Arabic
aze	Azerbaijani
bel	Belarusian
ben	Bengali
bul	Bulgarian
cat	Catalan
ces	Czech
ckb	Kurdish
cmn	Mandarin
dan	Danish
deu	German
ell	Greek
eng	English
est	Estonian
eus	Basque
fin	Finnish
fra	French
hau	Hausa
heb	Hebrew
hin	Hindi
hrv	Croatian
hun	Hungarian
hye	Armenian
ind	Indonesian
isl	Icelandic
ita	Italian
jpn	Japanese
kat	Georgian
kaz	Kazakh
kor	Korean
lit	Lithuanian
mar	Marathi
mkd	Macedonian
nld	Dutch
nob	Norwegian
pes	Persian
pol	Polish
por	Portuguese
ron	Romanian
run	Rundi
rus	Russian
slk	Slovak
spa	Spanish
srp	Serbian
swe	Swedish
tgl	Tagalog
tur	Turkish
ukr	Ukrainian
vie	Vietnamese

Install

npm install --save lande

Usage

The library exports a single function which gives you a sorted list of detected languages and their probabilities.

import lande from 'lande';

lande ( 'What language is this sentence written in?' ); // => [['eng', 0.9999921321868896], ['deu', 0.000002357382982154377], ['heb', 0.000001461773877053929], ...]
lande ( 'In che lingua è scritta questa frase?' ); // => [['ita', 0.9999935626983643], ['ell', 0.0000025603442281862954], ['vie', 0.000002114558583343751], ...]

License

MIT © Fabio Spampinato