NLP.js
"NLP.js" is a general natural language utilities for nodejs. Currently supporting:
- Guess the language of a phrase
- Fast levenshtein distance of two strings
- Search the best substring of a string with less levenshtein distance to a given pattern.
- Get stemmers and tokenizers for several languages.
- Sentiment Analysis for phrases (with negation support).
- Named Entity Recognition and management, multilanguage, and accepting similar strings, so the introduced text does not need to be exact.
- Natural Language Processing Classifier, to classify utterance into intents.
- Natural Language Generation Manager, so from intents and conditions it can generate an answer.
- NLP Manager: a tool able to manage several languages, the Named Entities for each language, the utterance and intents for the training of the classifier, and for a given utterance return the entity extraction, the intent classification and the sentiment analysis. Also, it is able to maintain a Natural Language Generation Manager for the answers.
TABLE OF CONTENTS
- Installation
- Example of use
- Language Support
- Language Guesser
- Similar Search
- NLP Classifier
- NER Manager
- Sentiment Analysis
- NLP Manager
- Loading from Excel
- Contributing
- Code of Conduct
- Who is behind it
- License
Installation
If you're looking to use NLP.js in your node application, you can install via NPM like so:
npm install node-nlp
Example of use
You can see a great example of use at the folder \examples\console-bot. This example is able to train the bot and save the model to a file, so when the bot is started again, the model is loaded instead of trained again.
You can start to build your NLP from scratch with few lines:
const NlpManager = ; const manager = languages: 'en' ; // Adds the utterances and intents for the NLP manager; manager; manager; manager; manager; manager; manager; manager; // Train also the NLG manager; manager; manager; manager; // Train and save the model. manager; manager; console;
This will show this result in console:
Language Support
There are several languages supported. The language support can be for the Stemmers or for Sentiment Analysis. Inside Stemmers there are three type of stemmers: Natural, Snowball and Custom. Natural stemmers are these supported by the Natural library, while Snowball stemmers are the ported version from the Snowball ones from Java. Custom stemmers are those with custom development out of the scope of Natural or Snowball. Inside Sentiment Analysis, there are three possible algoritms: AFINN, Senticon and Pattern.
Language | Natural | Snowball | Custom | AFINN | Senticon | Pattern |
---|---|---|---|---|---|---|
Chinese (zh) | X | |||||
Danish (da) | X | |||||
Dutch (nl) | X | X | X | |||
English (en) | X | X | X | X | X | |
Farsi (fa) | X | |||||
Finnish (fi) | X | |||||
French (fr) | X | X | X | |||
German (de) | X | X | ||||
Hungarian (hu) | X | |||||
Indonesian (id) | X | |||||
Italian (it) | X | X | X | |||
Japanese (ja) | X | |||||
Norwegian (no) | X | X | ||||
Portuguese (pt) | X | X | ||||
Romanian (ro) | X | |||||
Russian (ru) | X | X | ||||
Spanish (es) | X | X | X | X | ||
Swedish (sv) | X | X | ||||
Turkish (tr) | X |
Language Guesser
The language object gives your code the skill to guess the language of a text. The method guess do that returning to you an array of all the languages ordered descending by the score.
const Language = ; const language = ; const guess = language; console;
This piece of code should write in console:
You can limit the amount of results with the third parameter of the method:
const Language = ; const language = ; let guess = language; console; console;
In console you'll see:
3
You can also provide a whitelist of accepted language to find the one that fits better
const Language = ; const language = ; let guess = language; console;
In console you'll see:
You can also use the method guessBest that returns only the best result.
const Language = ; const language = ; let guess = language; console; let guess = language; console;
That will show this in console:
Similar Search
Similar Search is used to calculate the levenshtein distance between two strings and also is able to search the best substring inside a string, i.e., the substring of a string which levenshtein distance is the smaller to another string.
You can calculate the levenshtein distance:
const SimilarSearch = ; const similar = ; similar; // returns 3
Also you can use collation so case and special characters are compared using collation:
const SimilarSearch = ; const similar = useCollation: true ; similar; // returns 1
Unfortunately, collation is very slow, but you can use normalization. Normalization preprocess strings converting to lowercase and converting accented characters to their unaccented equivalent, and this is pretty much faster than collation:
const SimilarSearch = ; const similar = normalize: true ; similar; // returns 1
You can search the best substring of string with the lower levenshtein distance. The accuracy is calculated as (length - distance) / length:
const SimilarSearch = ; const similar = ; const text1 = 'Morbi interdum ultricies neque varius condimentum. Donec volutpat turpis interdum metus ultricies vulputate.'; const text2 = 'interdumaultriciesbneque'; const result = similar; // result is { start: 6, end: 30, levenshtein: 2, accuracy: 0.9166666666666666 }
NLP Classifier
You can train a classifier (indicating language) with utterances and their intents. Then you can give a different utterance, and get the classifications for each intent, sorted descending by the score value.
const NlpClassifier = ; const classifier = language: 'fr' ; classifier; classifier; classifier; classifier; classifier; classifier; classifier; const classifications = classifier; // value is [ { label: 'keys', value: 0.994927593677957 }, { label: 'greet', value: 0.005072406322043053 } ]
Or you can get only the best classification
const NlpClassifier = ; const classifier = language: 'fr' ; classifier; classifier; classifier; classifier; classifier; classifier; classifier; const classification = classifier; // value is { label: 'keys', value: 0.994927593677957 }
Currently 19 languages are supported:
- Chinese (zh)
- Danish (da)
- Dutch (nl)
- Enlish (en)
- Farsi (fa)
- Finnish (fi)
- French (fr)
- German (de)
- Hungarian (hu)
- Indonesian (id)
- Italian (it)
- Japanese (ja)
- Norwegian (no)
- Portuguese (pt)
- Romanian (ro)
- Russian (ru)
- Spanish (es)
- Swedish (sv)
- Turkish (tr)
NER Manager
The Named Entity Recognition manager is able to store an structure of entities and options of the entity for each language. Then, given an utterance and the language, is able to search the options of the entity inside the utterance, and return a list of the bests substrings. This is done using a threshold for the accuracy, by default the accuracy is 0.5 but you can provide it in the options when creating the instance.
const NerManager = ; const manager = threshold: 08 ; manager; manager; manager; manager; manager; manager; const entities = manager; // value is [ { start: 6, end: 15, levenshtein: 1, accuracy: 0.8888888888888888, option: 'spiderman', // sourceText: 'Spiderman', entity: 'hero', utteranceText: 'spederman' }, // { start: 23, end: 32, levenshtein: 1, accuracy: 0.8888888888888888, option: 'pasta', // sourceText: 'spaghetti', entity: 'food', utteranceText: 'speghetti' } ]
It also support Regular Expression entities
const NerManager = ; const manager = threshold: 08 ; manager; manager; manager; manager; manager; manager; manager; const entities = manager; console; // [ { start: 6, // end: 15, // levenshtein: 0, // accuracy: 1, // option: 'spiderman', // sourceText: 'Spiderman', // entity: 'hero', // utteranceText: 'spiderman' }, // { start: 23, // end: 32, // levenshtein: 1, // accuracy: 0.8888888888888888, // option: 'pasta', // sourceText: 'spaghetti', // entity: 'food', // utteranceText: 'speghetti' }, // { start: 61, // end: 80, // accuracy: 1, // sourceText: 'spiderman@gmial.com', // utteranceText: 'spiderman@gmial.com', // entity: 'email' } ]
Sentiment Analysis
The Sentiment Analysis module is able to calculate the sentiment based on the AFINN. Languages accepted:
- en: English
- es: Spanish
- nl: Dutch
- fr: French
- it: Italian
- de: German
Language | AFINN | Senticon | Pattern | Negations |
---|---|---|---|---|
Dutch | X | X | ||
English | X | X | X | X |
French | X | |||
Italian | X | |||
Spanish | X | X | X | |
German | X | X |
By default Senticon is used if possible, otherwise AFINN, and last one Pattern:
Language | AFINN | Senticon | Pattern |
---|---|---|---|
Dutch | X | ||
English | X | ||
French | X | ||
Italian | X | ||
Spanish | X |
You can use a SentimentAnalyzer if you want to manage only one language:
const SentimentAnalyzer = ; const sentiment = language: 'en' ; let result = sentiment; console; // { score: 0.313, // numWords: 3, // numHits: 1, // comparative: 0.10433333333333333, // type: 'senticon', // language: 'en' } result = sentiment; console; // { score: -0.458, // numWords: 3, // numHits: 1, // comparative: -0.15266666666666667, // type: 'senticon', // language: 'en' }
Or you can use the SentimentManager if you want to manage several languages:
const SentimentManager = ; const sentiment = ; let result = sentiment; console; // { score: 0.313, // numWords: 3, // numHits: 1, // comparative: 0.10433333333333333, // type: 'senticon', // language: 'en' } result = sentiment; console; // { score: 0.278, // comparative: 0.0695, // vote: 'positive', // numWords: 4, // numHits: 1, // type: 'senticon', // language: 'es' }
NLP Manager
The NLP Manager is able to manage several languages. For each one, he manages the Named Entities, and is able to train the NLP classifier. Once we have it trained, we can ask the NLP manager to process one utterance. We can even don't tell the language and the NLP Manger will guess it from the languages that it knows. When the utterance is processed, the NLP manager will:
- Identify the language
- Classify the utterance using ML, and returns the classifications and the best intent and score
- Gets the entities from the utterance. If the NLP was trained using entities in the format %entity%, then the search for entities will be limited to those that are present in this intent; otherwise, all the possible entities will be checked.
- Gets the sentiment analysis.
const NlpManager = ; const manager = languages: 'en' ; manager; manager; manager; manager; manager; manager; manager; manager; manager; manager; const result = manager; console; // { locale: 'en', // localeIso2: 'en', // language: 'English', // utterance: 'I saw spiderman eating spaghetti today in the city!', // classification: // [ { label: 'sawhero', value: 0.9920519933583061 }, // { label: 'wanteat', value: 0.00794800664169383 } ], // intent: 'sawhero', // score: 0.9920519933583061, // entities: // [ { start: 6, // end: 15, // levenshtein: 0, // accuracy: 1, // option: 'spiderman', // sourceText: 'Spiderman', // entity: 'hero', // utteranceText: 'spiderman' }, // { start: 23, // end: 32, // levenshtein: 0, // accuracy: 1, // option: 'pasta', // sourceText: 'spaghetti', // entity: 'food', // utteranceText: 'spaghetti' } ], // sentiment: // { score: 0.708, // comparative: 0.07866666666666666, // vote: 'positive', // numWords: 9, // numHits: 2, // type: 'senticon', // language: 'en' } }
Also, you can save and load the NLP Manager to be reused without having to train it, because the thetas of the ML are also stored.
manager.train();
manager.save(filename);
manager = new NlpManager();
manager.load(filename);
const result = manager.process('I saw spiderman eating spaghetti today in the city!');
If no filename is provided by default it is './model.nlp'.
Loading from Excel
The NLP manager can load all the information from an excel file. You can find an example excel file at https://github.com/axa-group/nlp.js/blob/master/test/nlp/rules.xls Inside the excel there must exists 4 tables: Languages, Named Entities, Intents and Responses. It's very important to keep the existing format of the tables: first row is the name of the table, second row are the column names, next rows are the da.
Contributing
You can read the guide of how to contribute at Contributing.
Code of Conduct
You can read the Code of Conduct at Code of Conduct.
Who is behind it?
This project is developed by AXA Shared Services Spain S.A.
If you need to contact us, you can do it at the email jesus.seijas@axa-groupsolutions.com
License
Copyright (c) AXA Shared Services Spain S.A.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.