rapid-automated-keyword-extraction

RAKE: Rapid automatic keyword extraction

The goal of this library was to create a well tested Javascript translation of the python implementation.

Differences in regular expressions and stopword lists have big impacts on this algorithm and sticking close to the python means that the code was easy to compare to ensure that it was in the ballpark.

This algorithm is described in Text Mining: Applications and Theory and also in this excellent blog post by Alyona Medelyan.

It operates using only the text you give it and produces surprisingly good results. There are likely better results possible but these mostly seem to involve a combination of Python, Machine Learning and a corpus of data.

The appeal of RAKE is of the "bang for the buck" variety.

Currently this library produces subtly different results than either the paper or the original Python implementation. While the results (especially the top scoring ones) line up nicely, these little deviations represent something to understand and resolve.

Installation

npm i rapid-automated-keyword-extraction

Usage

> var rake = require('rapid-automated-keyword-extraction').default
undefined
> rake('Compatibility of systems of linear constraints over the set of natural numbers', 'test/data/salton_1971_smartstoplist.txt').then(console.log)
{ 'natural numbers': 4,
  'linear constraints': 4,
  set: 1,
  systems: 1,
  compatibility: 1 }

Stopword lists

The stopword list used by the python version is here. It has a comment as the first line which might break the world...

Links to other stopword lists can be found here

Any file with one word per line should be fine.

What's next

After hammering out differences in the results, plans are to focus on

• Fully embracing JS idioms (Promises/ES201X)
• Explore ways to improve the results as described here
• Options to control result format (number, result|result+rank, etc)
• Include default stopword list.
• Improve handling of special characters and italics
• Deal with sentences that have been split over multiple lines (sentence now ends with -)