Get unlimited public & private packages + package-based permissions with npm Pro.Get started »


1.0.7 • Public • Published

Mysql Tokenizer

Splits strings into tokens using the syntax of Mysql 8.

const tokenize = require('mysql-tokenizer')();
const tokens = tokenize("SELECT 0, 0x1, '2', X'33';")
/* Output:
["SELECT"," ","0",",","0x1",","," ","'2'",","," ","X'33'",";"]

Basic Usage

const tokenize = require('mysql-tokenizer')();
const tokens = tokenize("SELECT tbl.x->'$', 1--2=<=>-1 FROM tbl;");
console.log(`Mysql: ${JSON.stringify(tokens)}`);
/* Output:
mysql: ["SELECT"," ","tbl",".","x","->","'$'",","," ","1","-","-","2","=","<=>","-","1"," ","FROM"," ","tbl",";"]

Comment Syntax

Comments are supported using --, # and /*...*/ syntaxes.

By default, comments using -- must start with a space, or come at the end of a line, as in Mysql.

const tokenize = require('mysql-tokenizer')();
const tokens = tokenize("SELECT 1--2=3 -- One less minus two equals three")
/* Output:
["SELECT"," ","1","-","-","2","=","3"," ","-- One less minus two equals three"]

To implement the SQL standard, which does not require following whitespace, include { regExp: require('mysql-tokenizer/lib/regexp-sql92') } in the options.


The following options are supported:

options.operators: Array

Specify the list of valid operator tokens. If not specified, all operators from Mysql 8 are supported.

If this is the only option required, options can be specified as an Array.

Alternatively, specify options.regExp.Token.Operator.

options.regExp.Initial: Object

Override the parser's built-in regular expressions for character parsing.
Initial RegExps must start with ^ and match at most one character.

declare namespace options.regExp.Initial {
    let BinaryOrHex: RegExp;
    let Identifier: RegExp;
    let PossibleComment: RegExp;
    let Punctuation: RegExp;
    let Quote: RegExp;
    let Whitespace: RegExp;

options.regExp.Token: Object

Token RegExps must have the 'g' and 'y' flags set.

Token.Operator is optional. Without it, the parser treats punctuation as invalid characters.

declare namespace options.regExp.Token {
    let StartComment: RegExp;
    let HexNumber: RegExp;
    let Identifier: RegExp;
    let IdentifierAfterAt: RegExp;
    let Number: RegExp;
    let Operator: RegExp|null;
    let Whitespace: RegExp;

options.commentHandlers: Object

Override the parser's built-in parsing of comments.

After a successful match on Token.StartComment, the matching string is used to look up a parsing function.

The default comment object is

comment = {
    '/*': Parser.prototype.starComment,
    '--': Parser.prototype.singleLineComment,
    '#': Parser.prototype.singleLineComment

Example: Comparing Mysql and SQL-92 Operator and Comment Parsing

const tokenizerFactory = require('mysql-tokenizer');
const tokenizeMysql = tokenizerFactory();
const tokensMysql = tokenizeMysql("SELECT tbl.x->'$', -1<=>1--2 FROM tbl;");
console.log(`Mysql: ${JSON.stringify(tokensMysql)}`);
/* Output:
Mysql: ["SELECT"," ","tbl",".","x","->","'$'",","," ","-","1","<=>","1","-","-","2"," ","FROM"," ","tbl",";"]
const sql92Operators = require('sql92-operators')
const sql92RegExp = require('mysql-tokenizer/lib/regexp-sql92');
const tokenizeSQL92 = tokenizerFactory({
    operators: sql92Operators,
    regExp: sql92RegExp
const tokensSQL92 = tokenizeSQL92("SELECT tbl.x->'$', -1<=>1--2 FROM tbl;");
console.log(`SQL92: ${JSON.stringify(tokensSQL92)}`);
/* Output:
SQL92: ["SELECT"," ","tbl",".","x","-",">","'$'",","," ","-","1","<=",">","1","--2 FROM tbl;"]


npm i mysql-tokenizer

DownloadsWeekly Downloads






Unpacked Size

101 kB

Total Files


Last publish


  • avatar