Nanobot: Polygonal Mascot


    2.0.6 • Public • Published

    Universal Lexer

    Travis Code Climate Coverage Status NPM Downloads

    Lexer which can parse any text input to tokens, according to provided regular expressions.

    In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first stage of a lexer. A lexer is generally combined with a parser, which together analyze the syntax of programming languages, web pages, and so forth.


    • Allow named regular expressions, so you don't have to work with it a lot
    • Allow post-processing tokens, to get more information you require

    How to install

    Package is available as universal-lexer in NPM, so you can use it in your project using npm install universal-lexer or yarn add universal-lexer

    What are requirements?

    Code itself is written in ES6 and should work in Node.js 6+ environment. If you would like to use it in browser or older development, there is also transpiled and bundled (UMD) version included. You can use universal-lexer/browser in your requires or UniversalLexer in global environment (in browser):

    // Load library
    const UniversalLexer = require('universal-lexer/browser')
    // Create lexer
    const lexer = UniversalLexer.compile(definitions)
    // ...

    How it works

    You've got two sets of functions:

    // Load library
    const UniversalLexer = require('universal-lexer')
    // Build code for this lexer
    const code1 =[ { type: 'Colon', value: ':' } ])
    const code2 = UniversalLexer.buildFromFile('json.yaml')
    // Compile dynamically a function which can be used
    const func1 = UniversalLexer.compile([ { type: 'Colon', value: ':' } ])
    const func2 = UniversalLexer.compileFromFile('json.yaml')

    There are two ways of passing rules to this lexer: from file or array of definitions.

    Pass as array of definitions

    Simply, pass definitions to lexer:

    // Load library
    const UniversalLexer = require('universal-lexer')
    // Create token definition
    const Colon = {
      type: 'Colon',
      value: ':'
    // Build array of definitions
    const definitions = [ Colon ]
    // Create lexer
    const lexer = UniversalLexer.compile(definitions)

    A definition is more complex object:

    // Required fields: 'type' and either `regex` or `value`
      // Token name
      type: 'String',
      // String value which should be searched on beginning on string
      value: 'abc',
      value: '(',
      // Regular expression to validate
      // if current token should be parsed as this token
      // Useful i.e. when you require separator after sentence,
      // but you don't want to include it.
      valid: '"',
      // Regular expression flags for 'valid' field
      validFlags: 'i',
      // Regular expression to find current token
      // You can use named groups as well (?<name>expression):
      // Then it will attach this information to token.
      regex: '"(?<value>([^"]|\\.)+)"',
      // Regular expression flags for 'regex' field
      regexFlags: 'i'

    Pass YAML file

    // Load library
    const UniversalLexer = require('universal-lexer')
    const lexer = UniversalLexer.compileFromFile('scss.yaml')

    YAML file for now should contain only Tokens property with definitions. Later it may have more advanced stuff like macros (for simpler syntax).


      # Whitespaces 
      type: NewLine
        value: "\n"
      type: Space
        regex: '\t]+'
      # Math 
      type: Operator
        regex: '[+-*/]'
      # Color 
      # It has 'valid' field, to be sure that it's not i.e. blacker 
      # Now, it will check if there is no text after 
      type: Color
        regex: '(?<value>black|white)'
        valid: '(black|white)[^\w]'

    Processing data

    Processing input data, after you created a lexer is pretty straight-forward with for method:

    // Load library
    const UniversalLexer = require('universal-lexer')
    // Create lexer
    const tokenize = UniversalLexer.compileFromFile('scss.yaml')
    // Build processor
    const tokens = tokenize('some { background: code }').tokens

    Post-processing tokens

    If you would like to make more advanced parsing on parsed tokens, you can do it with addProcessor method:

    // Load library
    const UniversalLexer = require('universal-lexer')
    // Create lexer
    const tokenize = UniversalLexer.compileFromFile('scss.yaml')
    // That's 'Literal' definition:
    const Literal = {
      type: 'Literal',
      regex: '(?<value>([^\t \n;"'',{}()\[\]#=:~&\\]|(\\.))+)'
    // Create processor which will replace all '\X' to 'X' in value
    function process (token) {
      if (token.type === 'Literal') { =\\(.)/g, '$1')
      return token
    // Also, you can return a new token
    function process2 (token) {
      if (token.type !== 'Literal') {
        return token
      return {
        type: 'Literal',
        data: {
          value:\\(.)/g, '$1')
        start: token.start,
        end: token.end
    // Get all tokens...
    const tokens = tokenize('some { background: code }', process).tokens

    Beautified code

    If you would like to get beautified code of lexer, you can use second argument of compile functions:

    UniversalLexer.compile(definitions, true)
    UniversalLexer.compileFromFile('scss.yaml', true)

    Possible results

    On success you will retrieve simple object with array of tokens:

      tokens: [
        { type: 'Whitespace', data: { value: '     ' }, start: 0, end: 5 },
        { type: 'Word', data: { value: 'some' }, start: 5, end: 9 }

    When something is wrong you will get error information:

      error: 'Unrecognized token',
      index: 1,
      line: 1,
      column: 2


    For now, you can see example of JSON semantics in examples/json.yaml file.


    After installing globally (or inside of NPM scripts) universal-lexer command is available:

    Usage: universal-lexer [options] output.js
      --version       Show version number                                  [boolean]
      -s, --source    Semantics file                                      [required]
      -b, --beautify  Should beautify code?                [boolean] [default: true]
      -h, --help      Show help                                            [boolean]
      universal-lexer -s json.yaml lexer.js  build lexer from semantics file


    Version 2

    • 2.0.6 - bugfix for single characters
    • 2.0.5 - fix mistake in README file (post-processing code)
    • 2.0.4 - remove unneeded benchmark dependency
    • 2.0.3 - add unit and E2E tests, fix small bugs
    • 2.0.2 - added CLI command
    • 2.0.1 - fix typo in README file
    • 2.0.0 - optimize it (even 10x faster) by expression analysis and some other things

    Version 1

    • 1.0.8 - change that current position in syntax error starts from 1 always
    • 1.0.7 - optimize definitions with "value", make syntax errors developer-friendly
    • 1.0.6 - optimized Lexer performance (20% faster in average)
    • 1.0.5 - fix browser version to be put into NPM package properly
    • 1.0.4 - bugfix for debugging
    • 1.0.3 - add proper sanitization for debug HTML
    • 1.0.2 - small fixes for README file
    • 1.0.1 - added Rollup.js support to build version for browser


    npm i universal-lexer

    DownloadsWeekly Downloads






    Unpacked Size

    108 kB

    Total Files


    Last publish


    • rangoo