Detect-File-Encoding-and-Language

Functionality

Determine the encoding and language of text files!

Detects 40 languages as well as the appropriate encoding
Available as CLI, in Node.js and in the browser
Supports .txt, .srt, .sub, .html, .csv, .tsv
Works best with large inputs
Completely free, no API key required

For reliable encoding and language detection, use files containing 500 words or more. Smaller inputs can work as well but the results might be less accurate and in some cases incorrect.

Feel free to test the functionality of this NPM package here. Upload your own files and see if the encoding and language are detected correctly!

Index

Detect-File-Encoding-and-Language

Usage

There are several ways in which you can use this NPM package. You can use it as a command-line interface, server-side with Node.js or client-side in the browser.

In the browser

In the body section of your html file, create an input element of type file and give it an id.

// index.html
<body>
  <input type="file" id="my-input-field" />
  <script src="app.js"></script>
</body>

Next, load the module either by using the script tag or by using a bundler!

Using the script tag

When loading it via the <script> tag, you can either use the CDN version or download the code itself and include it in your project. For a quickstart use the CDN version. If you want to be able to use it offline, download and include it!

Via CDN

// index.html

<body>
  <input type="file" id="my-input-field" />
  <script src="https://unpkg.com/detect-file-encoding-and-language/umd/language-encoding.min.js"></script>
  <script src="app.js"></script>
</body>

Now that you've loaded the module, you can start using it.

Via download

Create a new folder called lib inside your root directory
Inside lib create a new file and call it language-encoding.min.js
Make sure the encoding of your newly created file is either UTF-8 or UTF-8 with BOM before proceeding!
Go to https://unpkg.com/detect-file-encoding-and-language/umd/language-encoding.min.js and copy the code
Paste it into language-encoding.min.js and save it
Use the code below to load language-encoding.min.js via the <script> tag.

// index.html

<body>
  <input type="file" id="my-input-field" />
  <script src="lib/language-encoding.min.js"></script>
  <script src="app.js"></script>
</body>

Usage

The <script> tag exposes the languageEncoding function to everything in the DOM located beneath it. When you call it and pass in the file that you want to analyze, it'll return a Promise that you can use to retrieve the encoding, language and confidence score as shown in the example below.

// app.js

document
  .getElementById("my-input-field")
  .addEventListener("change", inputHandler);

function inputHandler(e) {
  const file = e.target.files[0];

  languageEncoding(file).then((fileInfo) => console.log(fileInfo));
  // Possible result: { language: english, encoding: UTF-8, confidence: { encoding: 1, language: 1 } }
}

Using a bundler

Installation

$ npm install detect-file-encoding-and-language

Usage

// app.js

const languageEncoding = require("detect-file-encoding-and-language");

document
  .getElementById("my-input-field")
  .addEventListener("change", inputHandler);

function inputHandler(e) {
  const file = e.target.files[0];

  languageEncoding(file).then((fileInfo) => console.log(fileInfo));
  // Possible result: { language: french, encoding: CP1252, confidence: { encoding: 1, language: 0.97 } }
}

Note: This works great with frameworks such as React because they are doing the bundling for you. However, if you're using pure vanilla Javascript you will have to bundle it yourself!

In Node.js

Installation

$ npm install detect-file-encoding-and-language

Usage

// index.js

const languageEncoding = require("detect-file-encoding-and-language");

const pathToFile = "/home/username/documents/my-text-file.txt";

languageEncoding(pathToFile).then((fileInfo) => console.log(fileInfo));
// Possible result: { language: japanese, encoding: Shift-JIS, confidence: { encoding: 0.94, language: 0.94 } }

In the terminal (CLI)

Installation

$ npm install -g detect-file-encoding-and-language

Usage

Once installed you'll be able to use the command dfeal to retrieve the encoding and language of your text files.

$ dfeal "/home/user name/Documents/subtitle file.srt"
# Possible result: { language: french, encoding: CP1252, confidence: { encoding: 0.99, language: 0.99 } }

or without quotation marks, using backslashes to escape spaces:

$ dfeal /home/user\ name/Documents/subtitle\ file.srt
# Possible result: { language: french, encoding: CP1252, confidence: { encoding: 0.97, language: 0.97 } }

Supported Languages

Polish
Czech
Hungarian
Romanian
Slovak
Slovenian
Albanian
Russian
Ukrainian
Bulgarian
English
French
Portuguese
Spanish
German
Italian
Danish
Norwegian
Swedish
Dutch
Finnish
Serbo-Croatian
Estonian
Icelandic
Malay-Indonesian
Greek
Turkish
Hebrew
Arabic
Farsi-Persian
Lithuanian
Chinese-Simplified
Chinese-Traditional
Japanese
Korean
Thai
Bengali
Hindi
Urdu
Vietnamese

Used Encodings

UTF-8
CP1250
CP1251
CP1252
CP1253
CP1254
CP1255
CP1256
CP1257
GB18030
BIG5
Shift-JIS
EUC-KR
TIS-620

Confidence Score

The confidence score ranges from 0 to 1. It's an object that contains two different confidence scores. The language confidence score and the encoding confidence score. Both confidence scores will be the same if the detected encoding is Unicode. Otherwise the confidence score for the language and the encoding is calculated seperately. It is based on the amount of matches that were found for a particular language and the frequency of those matches. If you want to learn more about how it all works, check out the Wiki entry!

Known Issues

Unable to detect Shift-JIS encoded Japanese text files when using Node.js. Solutions are welcome!

License

This project is licensed under the MIT License