Ninja Pokémon Master

    gibberish-detective

    1.0.3 • Public • Published

    Gibberish Detector

    This utility is used to identify if a string of text contains gibberish, using a relatively simple implementation of Machine Learning and Markov Chaining. The utility is trained with a large sample of legitimate sentences with which it builds a model to determine the probability that any two characters would be found adjacently.

    Configuration

    Options can be set during or after initialization:

    const gibberish = require("gibberish-detective")({useCache: false});
    gibberish.set("useCache", true)
    

    Options

    Name Type Default Description
    useCache bool true Use cache during matrix querying
    model obj shipped model A learning model outputted from .train()that is used to calculate letter pairing frequencies
    thresholdFn fn(model) avg(baseline.good.min, baseline.bad.max) A function that outputs a number to determine the minimum value that an .assignScore() output can be before it is considered to be gibberish

    Detection

    The .detect()(.isGibberish() is an alias) function takes a string and returns a boolean on whether or not it detects the string as gibberish.

    gibberish.detect('Luke, I am your second cousin!'); // returns false
    gibberish.detect('fasdfhaiufaewroawifasdaeta'); // returns true 
    

    You may override the training model it is testing against by passing it in as the second (and optional) argument.

    Training

    This package is shipped with a default model that was trained against a Sherlock Holmes novel. If necessary, a new model can be generated. As an example, if someone wishes to use this with another language, it might be prudent to generate a model based off of this language.

    For best results the training text used should be decently long (at least few megabytes) with diverse and legitimate words. It is not recommended that it be trained using literature from the science fiction and fantasy genres because those selections often contain unusual word formations.

    To create a new learning model, you must provide it with the aforementioned large sample of text to build the learning matrix, a sample of short lines of sentences that make sense, and a sample of lines made up of gibberish. It uses the last two data sets to form a threshold between gibberish and non-gibberish text.

    let gibberish = require("gibberish-detective")();
    let fs = require('fs');
    let sample = fs.readFileSync('./data/good.txt', 'utf-8');
    let sample_good_small = fs.readFileSync('./data/good_sm.txt', 'utf-8');
    let sample_bad = fs.readFileSync('./data/bad.txt', 'utf-8');
    
    let newModel = gibberish.train(sample, sample_good_small, sample_bad);
    
    gibberish.set("model", newModel);
    

    It is recommended that if re-training is necessary that this process only be done once and the resultant model be saved to a file and then thereafter that file is loaded into into the module at init-time:

    At train time:

    let fs = require('fs');
    let newModel = gibberish.train(sample, sample_good_small, sample_bad);
    fs.writeFileSync("mymodel.json", JSON.stringify(newModel))
    

    At run-time

    let myLearningModel = require('mymodel.json');
    let gibberish = require("gibberish-detector")({model: myLearningModel});
    

    Attributions

    rrenaud originally created a projected similar to this in python, called Gibberish-Detector.

    Install

    npm i gibberish-detective

    DownloadsWeekly Downloads

    192

    Version

    1.0.3

    License

    MIT

    Unpacked Size

    44.7 kB

    Total Files

    5

    Last publish

    Collaborators

    • alpha4615