Rale or Rantemma Advanced Lexing Evaluation
Is the official lexer of the Rantemma Language System (aka RLS).
No need to create a lexer, with Rale it's easy to parse any string into Token as you want or with default lexer parameters.
Exemple is generally in TypeScript
Create your own lexer
Unique Token
The unique token is the most simple kind of tokens, allow you to tokenize one character with a name.
const uniques: Unique[] = {
{name: "semicolon", character: ";"}
}
Long Token
The long token is a token formed by RegExp. The regex match is executed character by character
const longs: Longs[] = {
{name: "string", regex: /[a-z0-9]/i}
}
Wrapper Token
The wrapper token is a specific token that wrap other tokens beetween two defined character inside the same token.
const wrappers: Wrapper[] = {
{name: "braces", begin: "{", end: "}"}
}
Create the instance
With tokens made before, we create the Lexer.
const lexer = new Rale(longs, uniques, wrappers, true); // the last value is for tokenizing unknown tokens.
Use your lexer
const query = "hey { test }";
lexer.parse(query);
That gave you :
[
{
"begin": {
"index": 0,
"coloumn": 0,
"line": 0
},
"end": {
"index": 3,
"coloumn": 3,
"line": 0
},
"type": "long",
"name": "string",
"content": "hey"
},
{
"begin": {
"index": 3,
"coloumn": 3,
"line": 0
},
"end": {
"index": 4,
"coloumn": 4,
"line": 0
},
"type": "long",
"name": "space",
"content": " "
},
{
"begin": {
"index": 4,
"coloumn": 4,
"line": 0
},
"end": {
"index": 12,
"coloumn": 12,
"line": 0
},
"type": "wrapper",
"name": "curly_bracket",
"wrapperContent": [
{
"begin": {
"index": 5,
"coloumn": 5,
"line": 0
},
"end": {
"index": 6,
"coloumn": 6,
"line": 0
},
"type": "long",
"name": "space",
"content": " "
},
{
"begin": {
"index": 6,
"coloumn": 6,
"line": 0
},
"end": {
"index": 10,
"coloumn": 10,
"line": 0
},
"type": "long",
"name": "string",
"content": "test"
},
{
"begin": {
"index": 10,
"coloumn": 10,
"line": 0
},
"end": {
"index": 11,
"coloumn": 11,
"line": 0
},
"type": "long",
"name": "space",
"content": " "
}
]
}
]
Enjoy!