Neoanthropic Preternatural Murmurings

    @byte-this/text-process
    TypeScript icon, indicating that this package has built-in type declarations

    1.0.8 • Public • Published

    text-process

    A library which provides functionality for reducing and processing words and phrases:

    • Stemmer: reduce words to a common base word, such as "running" => "run"
    • Tokenizer: reduce sentences to a list of stemmed words
    • String Distance: determine the edit distance between strings

    For more info on how to use + the concepts behind the implementation, please visit: https://bytethisstore.com/articles/pg/search-text-process

    Stemming

    Stemming is a process which converts a word into a common base form which may or may not be a word itself. For example:

    • running => run
    • error => error
    • worrying => worri (not a real word)
    • exemplifying => exemplifi (not a real word)
    • contained => contain

    Stemming can be used as a means of normalizing words to a conceptual "base form" which an algorithm can then leverage to perform some operation, such as:

    • Comparing: checking if two strings or documents contain the same words, even if there are different suffixes, such as "ing"
    • Searching: reducing words in a search query to base forms so different forms of words can return the same results
    • Quantifying: collecting logs or metrics about how often some particular word is included in a text, including variations on that word.

    There is a similar process called Lemmatization which takes this a step further by considering the context in which a particular word is in, such as noun vs verb, and is able to convert the word to its correct base word which is called the Lemma. While more accurate, it is also more expensive to perform this operation.

    Stop Words

    Stop words are words which are highly common, such as "the", "and", "I". In certain cases these words can be filtered out of strings when performing processing, analysis, or other operations in which these words are superfluous.

    Tokens and Tokenizing

    The tokenizer in this library uses the Stemmer to convert a string to a list of tokens which are stemmed words with only alphanumeric characters. It filters out certain stop-words. A few examples:

    • "The quick brown fox jumps over the lazy dog" => ["quick", "brown" , "fox", "jump", "lazi", "dog"]
    • "An apple a day keeps the doctor away" => ["appl", "dai", "keep", "doctor", "awai"]
    • "While I was running I ran into the wall" => ["run", "ran", "wall"]

    Keywords

    none

    Install

    npm i @byte-this/text-process

    DownloadsWeekly Downloads

    19

    Version

    1.0.8

    License

    ISC

    Unpacked Size

    28.9 kB

    Total Files

    24

    Last publish

    Collaborators

    • byte-this