Search results
146 packages found
Splits a JSON string into an annotated list of tokens
A simple, Twitter-aware tokenizer.
- tokenise
- tokenize
- tokenising
- tokenizing
- tokeniser
- tokenizer
- token
- NLP
- language
- text
- strings
- stanford
- dlatk
Module for fast css selector tokenization from string
Lemmize and tokenize string which contains Chinese and English words
Tokenizer for tokenizing sentences, for BERT or other NLP preprocessing.
A simple iterative lexer written in TypeScript
Tokenize Excel formulas
Transform hypertext strings (e.g., HTML, Markdown) into plain text for natural language processing (NLP) normalization
Uses snapdragon to tokenize a single JavaScript block comment into an object, with description, tags, and code example sections that can be passed to any other comment parsers for further parsing.
Module for fast element to selector matching (with custom :hover, :focus etc. handing)
Tokenizes utterances that contain a mix of English and Chinese words.
- tokenizer
- chinese
- english
- language
- tokenize
- nltk
- nlp
- natural-language-processing
- chinese-tokenizer
- contains-chinese
A small library for building toy lexers.
A complex string based CSS managment library
A replacement library for JavaScript's standard JSON functions and more
- javascript
- js
- nodejs
- node
- json
- Traverse
- HasPath
- FindName
- FindValue
- GetValue
- SetValue
- Stringify
- Tablify
- ToIniText
- View more
Used to interact with the OpenToken API.
String Tokenizer for Node.js using ICU's BreakIterators
Uses esprima to extract line and block comments from a string of JavaScript. Also optionally parses code context (the next line of code after a comment).
A NodeJS library to tokenize strings