lexemic
Requirements
- Node.js (v0.10.17+)
- npm (v1.3.9+)
Install
npm install lexemic -g
Description
Tools for working with human language data.
Features
- Sentiment analysis
- Stemming
- Tokenization
- Statistical analysis
Usage
$ lexemic [command] [implementation] [target...]
NOTE: The target
may be an inline string or the path to a text file encoded as UTF-8.
Sentiment analysis
$ lexemic sentiment "I am mad at you." # => {
:score -1,
:comparative -0.25,
:positive {
:score 0,
:comparative 0,
:words ()
},
:negative {
:score 1,
:comparative 0.25,
:words (mad)
}
}
Sentiment analysis attempts to determine the affective state of the
speaker or the writer. The default implementation returns an
EDN map of this analysis. The :score
represents the number of emotive words in the text while the
:comparative
rates the occurrence of these words with regards to the
length of the text. The nested values (i.e. those under :positive
and
:negative
) provide a list of matched :words
and take only into account
their respective affectivity. The top level values incorporate both
affective states – returning negative values for texts with overall negative
affects and positive values for texts with overall positive affects.
Stemming
$ lexemic stem "My education has been educational" # => #{"My"
"educ"
"ha"
"been"}
Stemming attempts to reduce inflected words to their stem. This is
useful in reducing your working set of words and expanding possible
search matches. The default implementation uses the porter
algorithm, though you may explicitly specify an implementation; -p
and -porter
for the porter algorithm (standard and gentle) or -l
and
-lancaster
for the lancaster algorithm (much more aggressive). This
command returns an EDN set of the
reduced working set.
Tokenization
$ lexemic tokenize "This is a sample sentence." # => ["This"
"is"
"a"
"sample"
"sentence"
"."]
Tokenization attempts to break a text into its desired constituent parts – typically, that is into words. This command returns an EDN vector of the result of this process.
Statistical analysis
Levenshtein distance
$ lexemic distance "This is a sentence." "This is a similar sentence." # => 8
Levenshtein distance measures the distance between two strings of text, often two documents. Informally, this distance corresponds to the minimum number of single-character edits required to change one string into the other (e.g. in the above example, '8' represents the insertion of 1 space and the 7 letters of the word 'similar'). This command returns an integer representing the result of this process.
Issues
If you need help, find a bug, want to request a feature or want to contribute, please create an issue.
Copyright
Copyright (c) 2013 Bertrand Karerangabo
See LICENSE.txt
for details.