snapdragon-scanner

    1.0.0 • Public • Published

    snapdragon-scanner NPM version NPM monthly downloads NPM total downloads Linux Build Status

    Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

    Please consider following this project's author, Jon Schlinkert, and consider starring the project to show your ❤️ and support.

    Install

    Install with npm:

    $ npm install --save snapdragon-scanner

    What is this?

    This is a simple Lexical Scanner that takes an object of regex patterns, and uses those patterns to process an input string into an array of tokens.

    What is the difference between this and snapdragon-lexer?

    snapdragon-lexer uses registered handler functions to capture and handle tokens, snapdragon-scanner simply iterates over an object of regular expression patterns to create tokens. You can think of snapdragon-scanner as the "lite" version of snapdragon-lexer.

    Usage

    const Scanner = require('snapdragon-scanner');

    API

    Scanner

    Create a new Scanner with the given str and optional rules.

    Params

    • input {String}: Input string to scan.
    • options {Object}: (optional) Pass an object of regex patterns on options.rules, or use .addRules() or .addRule() after instantiating.

    Example

    const Scanner = require('snapdragon-scanner');
    const scanner = new Scanner('var foo = "bar";', {
      rules: {
        space: /^ +/,
        tab: /^\t+/,
        newline: /^\n+/,
        text: /^\w+/,
        equal: /^=/,
        quote: /^["']/,
        semi: /^;/,
        dot: /^\./
      }
    });

    .addRule

    Add a rule to the scanner.

    Params

    • rule {String}
    • match {RegExp}: Match array from RegExp.exec().

    Example

    console.log(scanner.token('text', ['foo']);
    //=> { rule: 'text', value: 'foo', match: [foo] };

    .addRule

    Add a rule to the scanner.

    Params

    • rule {String}
    • regex {RegExp}: Regular expression to use when scanning.

    Example

    scanner.addRule(rule, regex);
    // example
    scanner.addRule('text', /^\w+/);

    .addRules

    Add an object of rules to the scanner.

    Params

    • rules {Object}

    Example

    scanner.addRules({
      text: /^\w+/,
      slash: /^\//,
      dot: /^\./
    });

    .match

    Attempts to match scanner.string with the given regex. Also validates the regex to ensure that it starts with ^ since matching should always be against the beginning of the string, and throws if the regex matches an empty string, to avoid catastrophic backtracking.

    Params

    • regex {RegExp}: (required)
    • returns {Array|null}: Returns the match array or null from RegExp.exec.

    Example

    const scanner = new Scanner('foo/bar', { text: /^\w+/ });
    const match = scanner.match(scanner.rules.get('text'));
    console.log(match);
    //=> [ 'foo', index: 0, input: 'foo/bar', groups: undefined ]

    .consume

    Remove the given length of substring from scanner.string.

    Params

    • len {Number}
    • value {String}: Optionally pass the value being consumed for minor performance improvement.
    • returns {String}: Returns the consumed value

    Example

    scanner.consume(1);
    scanner.consume(1, '*');

    .enqueue

    Push a token onto the scanner.queue array.

    Params

    • token {object}
    • returns {Object}: Returns the token.

    Example

    console.log(scanner.queue.length); // 0
    scanner.enqueue({ rule: 'foo' });
    console.log(scanner.queue.length); // 1

    .dequeue

    Shift a token from scanner.queue.

    • returns {Object}: Returns the first token in the scanner.queue.

    Example

    console.log(scanner.queue.length); // 0
    scanner.enqueue({ rule: 'foo' });
    console.log(scanner.queue.length); // 1
    scanner.dequeue();
    console.log(scanner.queue.length); // 0

    .advance

    Iterates over the registered regex patterns until a match is found, then returns a token from the match and regex rule.

    • returns {Object}: Returns a token with rule, value and match properties.

    Example

    const token = scanner.advance();
    console.log(token) // { rule: 'text', value: 'foo' }

    .lookahead

    Lookahead n tokens and return the last token. Pushes any intermediate tokens onto scanner.tokens. To lookahead a single token, use .peek().

    Params

    • n {number}
    • returns {Object}

    Example

    const token = scanner.lookahead(2);

    .peek

    Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).

    • returns {Object|undefined}: Returns a token, or undefined if no match was found.

    Example

    const token = scanner.peek();

    .peek

    Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).

    • returns {Object|undefined}: Returns a token, or undefined if no match was found.

    Example

    const token = scanner.peek();

    .scan

    Returns the next token and advances the cursor position.

    • returns {Object|undefined}: Returns a token, or undefined if no match was found.

    Example

    const token = scanner.scan();

    .scanWhile

    Scan until the given fn does not return true.

    Params

    • fn {Function}: Must return true to continue scanning.
    • returns {Array}: Returns an array if scanned tokens.

    Example

    scanner.scanWhile(tok => tok.rule !== 'space');

    .bos

    Returns true if the scanner has not consumed any of the input string.

    • returns {Boolean}

    .eos

    Returns true if scanner.string and scanner.queue are empty.

    • returns {Boolean}

    Token objects

    Scanner tokens are plain JavaScript objects with the following properties:

    {
      type: String;
      value: String
      match: Array
    }

    Token properties

    • type {String} - The name of the regex that matched the substring.
    • value {String} - The substring that was captured by the regex.
    • match {Array} - The match array from RegExp.exec()

    Release history

    See the changelog.

    About

    Contributing

    Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.

    Running Tests

    Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:

    $ npm install && npm test
    Building docs

    (This project's readme.md is generated by verb, please don't edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)

    To generate the readme, run the following command:

    $ npm install -g verbose/verb#dev verb-generate-readme && verb

    Related projects

    You might also be interested in these projects:

    Author

    Jon Schlinkert

    License

    Copyright © 2018, Jon Schlinkert. Released under the MIT License.


    This file was generated by verb-generate-readme, v0.8.0, on November 19, 2018.

    Install

    npm i snapdragon-scanner

    DownloadsWeekly Downloads

    2

    Version

    1.0.0

    License

    MIT

    Unpacked Size

    20.8 kB

    Total Files

    4

    Last publish

    Collaborators

    • jonschlinkert