wordfreq

    0.1.0-dev • Public • Published

    wordfreq

    Text corpus calculation in Javascript. Supports Chinese, English. See demo.

    This library is a spin-off project from HTML5 Word Cloud.

    Simple usage

    Load wordfreq.js script to the web page, and run:

    // Create an options object for initialization
    var options = {
      workerUrl: 'path/to/wordfreq.worker.js' };
    // Initialize and run process() function
    var wordfreq = WordFreq(options).process(text, function (list) {
      // console.log the list returned in this callback.
      console.log(list);
    });
    

    WordFreq() methods are chainable, for example,

    // Process 3 strings and get corpus of all the texts.
    WordFreq(options)
      .process(text).process(text2).process(text3)
      .getList(function (list) {
        console.log(list);
      });
    

    To use this library synchronously, load wordfreq.worker.js and use the WordFreqSync interface. Check API.md for available options and methods.

    Command-line interface

    Command-line interface is available, powered by Node.js. To install globally, run

    npm install -g wordfreq
    

    Example usage:

    wordfreq ~/mypost.txt | less
    cat ~/mypost.txt | wordfreq - | less
    

    Algorithm

    Corpus is calculated with a simple N-gram algorithm and sub-string filter. Here is an article in Traditional Chinese on how HTML5 Word Cloud is being done.

    Porter Stemming Algorithm is included for processing English.

    Testing

    To run tests, first you would need to pull the required QUnit library by running

    git submodule init
    git submodule update
    

    Then, start a localhost HTTP server, for example,

    python -m SimpleHTTPServer 8009
    

    Point your browser to http://localhost:8009/test/ to start testing.

    You may also run the tests with PhantomJS by running

    phantomjs test/qunit/addons/phantomjs/runner.js http://localhost:8009/test/
    

    You will find all the information you need to write testcases on the QUnit website. All non-trivial code submission are expected to accompany with testcases.

    Known Gecko issue: The testcases will make Firefox choke; Web Worker will stop working after a few reloads. This was since fixed in bug 785248 on Oct. 3, 2012, so use Firefox 18 (currently Aurora) instead for testing.

    travis-ci.org Status

    PhantomJS on Travis-CI exhibits random hang, results random falling test result; but here is the badge:

    Build Status

    Keywords

    none

    Install

    npm i wordfreq

    DownloadsWeekly Downloads

    5

    Version

    0.1.0-dev

    License

    none

    Last publish

    Collaborators

    • timdream