Neoanthropic Preternatural Murmurings

    rale

    0.0.9 • Public • Published

    Rale or Rantemma Advanced Lexing Evaluation

    Is the official lexer of the Rantemma Language System (aka RLS).

    No need to create a lexer, with Rale it's easy to parse any string into Token as you want or with default lexer parameters.


    Exemple is generally in TypeScript

    Create your own lexer

    Unique Token

    The unique token is the most simple kind of tokens, allow you to tokenize one character with a name.

    const uniques: Unique[] = {
        {name: "semicolon", character: ";"}
    }

    Long Token

    The long token is a token formed by RegExp. The regex match is executed character by character

    const longs: Longs[] = {
        {name: "string", regex: /[a-z0-9]/i}
    }

    Wrapper Token

    The wrapper token is a specific token that wrap other tokens beetween two defined character inside the same token.

    const wrappers: Wrapper[] = {
        {name: "braces", begin: "{", end: "}"}
    }

    Create the instance

    With tokens made before, we create the Lexer.

    const lexer = new Rale(longs, uniques, wrappers, true); // the last value is for tokenizing unknown tokens.

    Use your lexer

    const query = "hey { test }";
    lexer.parse(query);

    That gave you :

    [
      {
        "begin": {
          "index": 0,
          "coloumn": 0,
          "line": 0
        },
        "end": {
          "index": 3,
          "coloumn": 3,
          "line": 0
        },
        "type": "long",
        "name": "string",
        "content": "hey"
      },
      {
        "begin": {
          "index": 3,
          "coloumn": 3,
          "line": 0
        },
        "end": {
          "index": 4,
          "coloumn": 4,
          "line": 0
        },
        "type": "long",
        "name": "space",
        "content": " "
      },
      {
        "begin": {
          "index": 4,
          "coloumn": 4,
          "line": 0
        },
        "end": {
          "index": 12,
          "coloumn": 12,
          "line": 0
        },
        "type": "wrapper",
        "name": "curly_bracket",
        "wrapperContent": [
          {
            "begin": {
              "index": 5,
              "coloumn": 5,
              "line": 0
            },
            "end": {
              "index": 6,
              "coloumn": 6,
              "line": 0
            },
            "type": "long",
            "name": "space",
            "content": " "
          },
          {
            "begin": {
              "index": 6,
              "coloumn": 6,
              "line": 0
            },
            "end": {
              "index": 10,
              "coloumn": 10,
              "line": 0
            },
            "type": "long",
            "name": "string",
            "content": "test"
          },
          {
            "begin": {
              "index": 10,
              "coloumn": 10,
              "line": 0
            },
            "end": {
              "index": 11,
              "coloumn": 11,
              "line": 0
            },
            "type": "long",
            "name": "space",
            "content": " "
          }
        ]
      }
    ]

    Enjoy!

    Install

    npm i rale

    DownloadsWeekly Downloads

    4

    Version

    0.0.9

    License

    MIT

    Unpacked Size

    38.7 kB

    Total Files

    8

    Last publish

    Collaborators

    • yirasan