Have ideas to improve npm?Join in the discussion! »

    readlineiter
    TypeScript icon, indicating that this package has built-in type declarations

    1.0.1 • Public • Published

    Fetch Line JavaScript packages: read remote or local file line by line as async iterator

    Read text file (remote over HTTP(S) or local) line by line as async iterator, with Node, browsers and Deno.

    This GitHub monorepo hosts 6 npm packages and 1 Deno module. They all serve a similar, simple purpose: read text file line by line and return an asynchronous iterable iterator of strings. They all try to be efficient and fast, and are written in TypeScript. However, their environment / platforms and exact purpose, features and behavior differ.

    Actions Status Node.js Deno lerna License

    TLDR: read the "Purpose & environment" table, pick the package you need, have a look at the Usage section, know at least a little bit about JavaScript's async / await, and go ahead to use them.

    Comparison

    Purpose & environment

    Package / Module Name Rec? Fetch remote file over HTTP(S) Read local file Version
    Node.js Deno Google ChromeFirefox
    Safari Microsoft Edge
    OperaSamsung Internet
    Node.js Deno
    fetchline 👍 version number
    nodefetchline 👍 version number
    isomorphic-fetchline version number
    naivefetchline version number
    getfileline version number
    readlineiter 👍 version number
    readlineiter for Deno 👍

    If you are not sure, just use the recommended one that has the correct environment and purpose you need.

    Tested:

    • Node.js: ≥ 12
    • Deno: ≥ v1.2.0
    • Modern browsers (Google Chrome, Firefox, Safari, Microsoft Edge, Opera, Samsung Internet): all latest

    Usage

    fetchline, nodefetchline, isomorphic-fetchline, and naivefetchline

    Examples

    npm install fetchline
    import fetchline from 'fetchline'
    const lineIterator =
      fetchline('https://raw.githubusercontent.com/tomchen/fetchline/main/testfile/crlf_finalnewline')
    // This is the same as:
    // fetchline(
    //   'https://raw.githubusercontent.com/tomchen/fetchline/main/testfile/crlf_finalnewline',
    //   {
    //     includeLastEmptyLine: true,
    //     encoding: 'utf-8',
    //     delimiter: /\r?\n/g,
    //   }
    // )
    ;(async () => {
      for await (const line of lineIterator) {
        // do something with `line`
      }
    })()

    Change 'fetchline' to nodefetchline, isomorphic-fetchline, or naivefetchline if you use these packages instead.

    For Deno: import fetchline from 'https://github.com/tomchen/fetchline/blob/main/packages/fetchline/src/index.ts'

    Change the import line to syntax like const nodefetchline = require('nodefetchline') if you use nodefetchline or isomorphic-fetchline package in Node's CommonJS.

    For browsers:

    <script src="https://unpkg.com/fetchline/dist/umd"></script>
    <script>
    fetchline(...) // same as above
    </script>

    Details

    These four packages have exactly the same interface (parameters and return value):

    Parameter Name Required? Type Default Value Description
    filepath Required string N/A URL or path of the text file
    options Optional object {} options, including the following three
    options.includeLastEmptyLine Optional boolean true Should it count the last empty line?
    options.encoding Optional string 'utf-8' File encoding
    options.delimiter Optional string or RegExp /\r?\n/g Line (or other item)'s delimiter / separator.
    NOTE: do not set it as something like /\r\n|\n|\r/g, it causes trouble when one of the chunks of a CRLF (\r\n)-EOL file ends with CR (\r)

    Return value: { AsyncIterableIterator<string> } An asynchronous iterable iterator containing each line in string from the text file

    readlineiter, readlineiter for Deno, and getfileline

    They have similar interface as the aforementioned fetchline, nodefetchline, isomorphic-fetchline, and naivefetchline, but do not have the second parameter, options, and everything options contains.

    npm install readlineiter
    import readlineiter from 'readlineiter' // For Deno: import readlineiter from 'https://raw.githubusercontent.com/tomchen/fetchline/main/packages/readlineiter-deno/mod.ts'
    const lineIterator = readlineiter('./crlf_finalnewline')
    ;(async () => {
      for await (const line of lineIterator) {
        // do something with `line`
      }
    })()

    Further comparison

    Characteristics

    ASAP 0 dependency TypeScript
    fetchline
    nodefetchline
    isomorphic-fetchline .d.ts
    naivefetchline
    getfileline
    readlineiter
    readlineiter for Deno

    ASAP:

    • These remote file requesting libs should resolve with the line text string as soon as possible, i.e. as soon as the chunks that have arrived can form the next complete line
      • Except for naivefetchline that is, well, naïve, I really can't blame it
    • The local file reading libs read the file with pointer, rather than get a whole string in memory then split the string

    0 dependency: no external non-dev dependency for npm packages. Note that:

    • Node libraries inevitably use native Node libraries http and https, or fs
    • getfileline and readlineiter also use readline native lib directly thus are just wrappers, but other packages here use own low-level method
    • "readlineiter for Deno" uses Deno Standard Module bufio.ts.

    TypeScript: the source code is in TypeScript, except for isomorphic-fetchline's source which is in JavaScript but has type definition (.d.ts).

    As for the production / dist files, these packages are all compiled into different module versions where possible: Node.js' Common.js (which uses require()), ES Module (native JS module with import), and minified UMD that is good for browser.

    Parameters amd return value

    filepath parameter includeLastEmptyLine option encoding option delimiter option Return AsyncIterableIterator<string>
    fetchline
    nodefetchline
    isomorphic-fetchline
    naivefetchline
    getfileline , always doesn't , always utf-8 , always EOL detected by readline
    readlineiter , always doesn't , always utf-8 , always EOL detected by readline
    readlineiter for Deno , always does , always utf-8 , always EOL detected by bufio.ts

    getfileline and readlineiter's delimiter is EOL character detected by readline native module with its crlfDelay option set to Infinity.

    Tips & thoughts

    async / await

    Of course, you should at least know a little bit about async / await, await asyncIterator.next() or for await of before using the packages here. If you don't, click the links to read the article from MDN Web Docs. Basically, you can do this:

    ;(async () => {
      for await (const line of lineIterator) {
        // do something with `line`
      }
    })()

    Or this:

    ;(async () => {
      let line
      let isDone
      while (1) {
        ;({ value: line, done: isDone } = await lineIterator.next())
        // do something with `line`
        if (isDone) {
          break
        }
      }
    })()

    Line-delimited JSON

    These packages, especially 'fetchline' (the first one) for browsers, could be helpful for line-delimited JSON (aka. ndjson (Newline Delimited JSON), JSON Lines) parsing. You could write something like:

    import fetchline from 'fetchline'
    const lineIterator = fetchline(lineDelimitedJsonUrl)
    ;(async () => {
      for await (const line of lineIterator) {
        const lineJson = JSON.parse(line)
        // do something with `lineJson`
      }
    })()

    Development

    This is a Lerna powered monorepo with mixed code (TypeScript / JavaScript), mixed module version (CommonsJS, ES Module, UMD) and cross-environment (Node.js, Deno, browsers) support, automated tests with GitHub Actions CI. Look at the root package.json and individual packages' package.json for available scripts, to get started, yarn then yarn bootstrap. You could also use the repo as an example of Lerna monorepo.

    Install

    npm i readlineiter

    DownloadsWeekly Downloads

    79

    Version

    1.0.1

    License

    MIT

    Unpacked Size

    22.8 kB

    Total Files

    6

    Last publish

    Collaborators

    • avatar