@jscpd/tokenizer
TypeScript icon, indicating that this package has built-in type declarations

3.5.4 • Public • Published

@jscpd/tokenizer

tokenizer is package from @jscpd used for convert programming code to list of tokens

Installation

npm install @jscpd/tokenizer --save

Usage

import {IOptions,  ITokensMap} from '@jscpd/core';
import {Tokenizer} from '@jscpd/tokenizer';

const tokenizer = new Tokenizer();
const options: IOptions = {};

const maps: ITokensMap[] = tokenizer.generateMaps('source_id', 'let a = "11"', 'javascript', options);

ga tracker

License

MIT © Andrey Kucherenko

Versions

Current Tags

Version History

Package Sidebar

Install

npm i @jscpd/tokenizer

Weekly Downloads

54,576

Version

3.5.4

License

MIT

Unpacked Size

65.1 kB

Total Files

40

Last publish

Collaborators

  • apk