methodius

Methodius (an NGram utility)

A utility for analyzing frequency of text chunks on the web.

Supply a bit o' text to the Methodius class, and let it determine your bigrams, trigrams, ngrams, letter-frequencies, word frequencies, bigram relationships, and create ngram trees.

Example

const { Methodius } = require('methodius');
// or import { Methodius } from 'methodius';

const udhr1 = `
All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
`;
const nGrams = new Methodius(udhr1);

const topLetters = nGrams.getTopLetters(10);
const topWords = nGrams.getTopWords(10);

API

Methodius

Global Class

new Methodius(text)

Parameters

name	type	Description
text	string	raw text to be analyzed

Static Members

Punctuations

characters to ignore when analyzing text period, comma, semicolon, colon, bang, question mark, interrobang, Spanish bang+, parens, bracket, brace, single quote, some spaces

\\.,;:!?‽¡¿⸘()\\[\\]{}<>’'…\"\n\t\r

wordSeparators

characters to ignore AND CONSUME when trying to find words em-dash, period, comma, semicolon, colon, bang, question mark, interrobang, Spanish bang+, parens, bracket, brace, single quote, space

—\\.,;:!?‽¡¿⸘()\\[\\]{}<>…"\\s

Static Methods

hasPunctuation(string)

determines if string contains punctuation

Parameters

name	type	Description
string	string

Returns boolean

hasSymbols(string)

determines if string contains symbols

Parameters

name	type	Description
string	string

Returns boolean

hasSpace(string)

determines if a string has a space

Parameters

name	type	Description
string	string

Returns boolean

sanitizeText(string)

lowercases text and removes diacritics and other characters that would throw off n-gram analysis

Parameters

name	type	Description
string	string

Returns string

getWords(text)

extracts an array of words from a string

Parameters

name	type	Description
text	string

Returns Array<string>

getNGrams(text, gramSize)

gets ngrams from text

Parameters

name	type	Description
text	string
gramSize	Number	Default = 2

Returns Array<string>

getMeanWordSize(wordArray)

Gets average size of a word

Parameters

name	type	Description
wordArray	string[]

Returns number

getMedianWordSize(wordArray)

Gets the median (middle) size of a word

Parameters

name	type	Description
wordArray	string[]

Returns number

getWordNGrams(text)

Gets 2-word pairs from text.

Note: This doesn't use sentence punctuation as a boundary. Should it?

Parameters

name	type	Description
text	string
gramSize	number	default=2

Returns Array<string>

getFrequencyMap(frequencyMap)

converts an array of strings into a map of those strings and number of occurences

Parameters

name	type	Description
ngramArray	Array<string>

Returns Map<string, number>

getPercentMap(frequencyMap)

converts a frequency map into a map of percentages

Parameters

name	type	Description
frequencyMap	Map<string, number>

Returns Map<string, number>

getTopGrams(frequencyMap)

filters a frequency map into only a small subset of the most frequent ones

Parameters

name	type	Description
frequencyMap	Map<string, number>
limit	number	default=20

Returns Map<string, number>

getIntersection(iterable1, iterable2)

returns an array of items that occur in both iterables

Parameters

name	type	Description
iterable1	`Map	Array`
iterable2	`Map	Array`

Returns Array<any> An array of items that occur in both iterables. It will compare the keys, if sent a map

getUnion(iterable1, iterable2)

Returns an array that is the union of two iterables

Parameters

name	type	Description
iterable1	`Map	Array`
iterable2	`Map	Array`

Returns Array<any> A union of the items that occur in both iterables.

getDisjunctiveUnion(iterable1, iterable2)

returns an array of arrays of the unique items in either iterable

Parameters

name	type	Description
iterable1	`Map	Array`
iterable2	`Map	Array`

Returns Array<Array<any> An array of arrays of the unique items. The first item is the first parameter, 2nd item second param

getComparison(iterable1, iterable2)

returns a map containing various comparisons between two iterables

Parameters

name	type	Description
iterable1	`Map	Array`
iterable2	`Map	Array`

Returns Map<string, <array>> A map containing various comparisons between two iterables. Those comparisons will be some kind of array (See intersection or disjunctiveUnion)

getWordPlacementForNGram(ngram, wordsArray)

determines the placement of a single ngram in an array of words

Parameters

name	type	Description
ngram	string
wordsArray	Array<string>

Returns Map<string, number> a map with the keys 'start', 'middle', and 'end' whose values correspond to how often the provided ngram occurs in this position

getWordPlacementForNGrams(ngrams, wordsArray)

determines the placement of ngrams in an array of words

Parameters

name	type	Description
ngram	Array<string>
wordsArray	Array<string>

Returns Map<string, Map<string, number>> a map with the key of the ngram, and the value that is a map containing start, middle, end

getNgramCollections(ngrams, wordsArray)

gets ngrams from an array of words

Parameters

name	type	Description
wordArray	Array<string>	an array of words
ngramSize	number	default = 2. The size of the ngrams to return

Returns Array<Array<string>> An array containing arrays of ngrams, each array corresponds to a word.

getNgramSiblings(searchText, ngramCollections, siblingSize)

using a collection returned from getNgramCollections, searches for a string and returns what comes before and after it

Parameters

name	type	Description
searchText	string	the string to search for
ngramCollections	`Array	Array<Array>`
siblingSize	number	default = 1. How many siblings to find in front or behind

Returns Map<'before'|'after',Map<string, number>> a Map with the keys 'before' and 'after' which contain maps of what comes before and after

Example

        const words = ['revolution', 'nation'];
        const ngramCollections = Methodius.getNgramCollections(words, 2);
        const onSiblings = Methodius.getNgramSiblings('io', ngramCollections);
        /* 
        new Map([
          ['before', new Map(
            ['ti', 2]
          )],
          ['after', new Map(
            ['on', 2]
          )]
        ])
        */

getRelatedNgrams(words, ngrams, ngramSize)

Gets the ngrams that will occur before or after other ngrams. Useful for finding patterns of ngrams.

Parameters

name	type	Description
words	Array<string>	an array of words to evaluate
ngrams	Map<string, number>	a frequency map of ngrams
ngramSize	number	default = 2. the size of the ngram

Returns

Map<string, number> A frequency map of how often ngrams occured before or after other ngrams

Example

This requires several steps. You'll need an array of words and a frequency map of ngrams.

    const ngrams = getNGrams('the revolution of the nation was on television. It was about pollution and the terrible situation ', 2);
    const frequencyMap = getFrequencyMap(ngrams);
    const topNgrams = getTopGrams(frequencyMap, 5);
    const words = ['the', 'revolution', 'of', 'the', 'nation', 'was', 'on', 'television', 'it', 'was', 'about', 'pollution', 'and', 'the', 'terrible', 'situation' ];
    const relatedNgrams = getRelatedNgrams(words, topNgrams, 2, 5);

getNgramTreeCollection(words)

Gets a nested map of maps that breaks down unique words into their smallest ngrams

Parameters

name	type	Description
words	Array<string>	an array of words to evaluate

Returns

Map<string, Array<string>| Map<string, <Array|string>> A nested map of maps that breaks down unique words into their smallest ngrams.

Instance Members

sanitizedText

lowercased text with diacritics removed

string

letters

an array of letters in the text

Array<string>

words

an array of words in the text

Array<string>

bigrams

an array of letter bigrams in the text

Array<string>

trigrams

an array of letter trigrams in the text

Array<string>

uniqueLetters

an array of unique letters in the text

Array<string>

uniqueBigrams

an array of unique bigrams in the text

Array<string>

uniqueTrigrams

an array of unique trigrams in the text

Map<string, Map<string, number>>

letterPositions

a map of placements of letters within words

Map<string, Map<string, number>>

bigramPositions

a map of placements of bigrams within words

Map<string, Map<string, number>>

uniqueTrigrams

a map of placements of trigrams within words

Array<string>

uniqueWords

an array of unique words in the text

Array<string>

letterFrequencies

a map of letter frequencies in the sanitized text

Map<string, number>

bigramFrequencies

a map of bigram frequencies in the sanitized text

Map<string, number>

trigramFrequencies

a map of trigram frequencies in the sanitized text

Map<string, number>

wordFrequencies

a map of word frequencies in the sanitized text

Map<string, number>

letterPercentages

a map of letter percentages in the sanitized text

Map<string, number>

bigramPercentages

a map of bigram percentages in the sanitized text

Map<string, number>

trigramPercentages

a map of trigram percentages in the sanitized text

Map<string, number>

wordPercentages

a map of word percentages in the sanitized text

Map<string, number>

meanWordSize

The average size of a word

number

medianWordSize

The middle size of a word

number

ngramTreeCollection

A nested map of maps that breaks down unique words into their smallest ngrams.

Instance Methods

getLetterNGrams(size)

gets an array of customizeable ngrams in the text

Parameters

name	type	Description
size	number	default = 2 size of the n-gram to return

Returns Array<string>

getTopLetters(limit)

a map of the most used letters in the text

Parameters

name	type	Description
limit	number	default = 20 number of top letters to return

Returns Map<string, number>

getTopBigrams(limit)

a map of the most used bigrams in the text

Parameters

name	type	Description
limit	number	default = 20 number of top bigrams to return

Returns Map<string, number>

getTopTrigrams(limit)

a map of the most used trigrams in the text

Parameters

name	type	Description
limit	number	default = 20 number of top trigrams to return

Returns Map<string, number>

getTopWords(limit)

a map of the most used words in the text

Parameters

name	type	Description
limit	number	default = 20 number of top words to return

Returns Map<string, number>

compareTo(methodius)

Compare this methodius instance to another

Parameters

name	type	Description
methodius	Methodius	another Methodius instance

Returns Map<string, Map> A map of property names and their comparisons (intersection, disjunctiveUnions, etc) for a set of properties

getRelatedTopNgrams(ngramSize, limit)

Gets the ngrams that will occur before or after other ngrams based on what the most frequent ngrams are. Useful for finding patterns of ngrams.

Parameters

name	type	Description
ngramSize	number	default = 2. the size of the ngram
limit	number	default = 20. the number of top ngrams to use

Returns

Map<string, number> A frequency map of how often the most common ngrams occured before or after other common ngrams