Nutritious Polygonal Meatball

    @bitacode/tokenator
    TypeScript icon, indicating that this package has built-in type declarations

    1.0.5 • Public • Published

    tokenator: a Node.js tokenization library

    <CircleCI> npm License

    tokenator is a simple to use library that rework a string splitting it to meaningful tokens.

    Table of Contents

    Description

    This library transforms a series of characters to tokens identified by input strings.

    e.g.
    "The itsy bitsy spider crawled up the water spout. | Down came the rain, and washed the spider out. | Out came the sun, and dried up all the rain, | and the itsy bitsy spider went up the spout again"
    
    Splitting strings: spider, rain
    
    Result tokens:
      > "The itsy bitsy "
      > "spider"
      > " crawled up the water spout. | Down came the "
      > "rain"
      > ", and washed the "
      > "spider"
      > " out. | Out came the sun, and dried up all the "
      > "rain"
      > ", | and the itsy bitsy "
      > "spider"
      > " went up the spout again"
    

    Installation

    npm install tokenator

    Code Examples

    The signatures of the available methods are:

    function tokenator(source: string, ...separators: string[]): string[]
    function tokenatorStream(...separators: string[]): (separatorsEncoding: string = "utf-8") => stream.Transform

    Code examples could be consulted within the test folder:

    These functions work over multiple separators, like in the example above. The tokenatorStream() function creates a Transform stream. NodeJS Stream usage specification is available on the NodeJS official documentation.

    Changelog

    Available here

    License

    MIT

    Available here

    Install

    npm i @bitacode/tokenator

    DownloadsWeekly Downloads

    18

    Version

    1.0.5

    License

    MIT

    Unpacked Size

    20.2 kB

    Total Files

    16

    Last publish

    Collaborators

    • ccarcaci