Irritable Iterable

async-iterable is a zero-dependency JavaScript library enhancing JavaScript iterables and generators¹, and their asynchronous counterparts². Unlike other collection packages, async-iterable excels in memory efficiency across any iterable size, including standard arrays and it supports asynchronous versions of iterables/iterators/generators.

Key features include on-demand data processing without preloading entire collections. For instance, when iterating through a data stream from an external source, async-iterable filters or accesses initial segments (head and headAsync) without needing to load all items, optimizing performance for large datasets or when individual items are slow to retrieve.

• Usage / Quick Start
• Show your support
• Contributing 🤝
• Release Process (Deploying to NPM) 🚀
• License 📝

Usage

Install

npm add irritable-iterable

Quick Start

import { filter } from "irritable-iterable"

const result = filter([1, 2, 3, 4], (num) => num % 2 === 0)
  .map((num) => `${num} is even`)
  .collect()

assert.deepEqual(result, ["2 is even", "4 is even"])

There are also *Async versions of each function (e.g. filterAsync, mapAsync, groupAsync, etc.) supporting Asynchronous AsyncIterator and AsyncGenerator functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncGenerator). For example:

import { filterAsync } from "irritable-iterable"

const promisedResult = filterAsync(
  generateOneTwoThree(),
  (num) => num % 2 === 0
)
  .map((num) => `${num} is even`)
  .collect()

const result = await promisedResult
assert.deepEqual(result, ["2 is even"])

// for demonstration purposes, but these could each be expensive asynchronous `fetch` calls
async function* generateOneTwoThree() {
  yield 1
  yield 2
  yield 3
}

chain

All of the methods returned from the root irritable-iterable package return an object that implements the Chain interface defined in ./src/chain.ts. This allows you to use the chaining syntax shown in the examples below.

import { chain } from "irritable-iterable"

chain([1, 2, 3]).filter((num) => num == 2) // => 2
chain([1, 2, 3]).find(["a", "b", "c"], (item) => item === "b") // => "b"
chain([1, 2, 3]).map((num) => num.toString()) // => [ "1", "2", "3" ]
chain([1, 2, 3]).size() // => 3
chain([1, 2, 3]).collect() // => [1, 2, 3]
chain([1, 2, 3]).collect() // => [1, 2, 3]

A more typical example might be:

import { filter } from "irritable-iterable"

const result = filter([1, 2, 3, 4], (num) => num % 2 === 0)
  .map((num) => `${num} is even`)
  .find((str) => str.startsWith("4"))

assert.equal(result, "4 is even")

filter

import { filter } from "irritable-iterable"

const result = filter([1, 2, 3], (num) => num % 2 === 0).collect()

assert.deepEqual(result, [2])

map

import { map } from "irritable-iterable"

const result = map([1, 2, 3], (num) => "number " + num).collect()

assert.deepStrictEqual(result, ["number 1", "number 2", "number 3"])

range

Similar Python's range function The stop value is exclusive; it is not included in the result.

import { range } from "irritable-iterable"

let result = range(3).collect()
assert.deepStrictEqual(result, [0, 1, 2])

result = range(0, 20, 5).collect()
assert.deepStrictEqual(result, [0, 5, 10, 15])

size

import { size } from "irritable-iterable"

const result = size(["a", "b", "c", "d"])

assert.equal(result, 4)

find

import { find } from "irritable-iterable"

const result = find(["a", "b", "c", "d"], (item) => item === "c")

assert.equal(result, "c")

first

import { first } from "irritable-iterable"

const result = first(["a", "b", "c", "d"])

assert.equal(result, "a")

head

import { head } from "irritable-iterable"

const result = head(["a", "b", "c", "d"], 2)

assert.equal(result, ["a", "b"])

headAsync

import { headAsync } from "irritable-iterable"

const promisedResult = headAsync(generateOneTwoThree(), 2).collect()

const result = await promisedResult
assert.deepEqual(result, [1, 2])

collect

Collect converts the iterable to an array and returns it.

import { chain } from "irritable-iterable"

const result = chain(generateABC()).collect()

assert.deepEqual(result, ["a", "b", "c"])

// for demonstration purposes:
function* generateABC() {
  yield "a"
  yield "b"
  yield "c"
}

group

Groups the items in the iterable into a map with keys specified by key-generation function and each value in the map is an array of the items with that key.

import { group } from "irritable-iterable"

const map = group(
  [
    { first: "john", last: "doe" },
    { first: "john", last: "foe" },
    { first: "jane", last: "doe" },
    { first: "jane", last: "foe" },
  ],
  (person) => person.last
)

// the result of `group` is a JavaScript Map (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map)
// ...which we convert to an array here:
const result = Array.from(map.entries())

assert.deepEqual(result, [
  [
    "doe",
    [
      { first: "john", last: "doe" },
      { first: "jane", last: "doe" },
    ],
  ],
  [
    "foe",
    [
      { first: "john", last: "foe" },
      { first: "jane", last: "foe" },
    ],
  ],
])

product

Produces a cartesian product of the provided iterables.

import { product } from "irritable-iterable"

const result = product([1, 2], [3, 4])
const resultArray = Array.from(result)
assert.deepEqual(resultArray, [
  [1, 3],
  [1, 4],
  [2, 3],
  [2, 4],
])

Show your support

Please give a ⭐️ if this project helped you!

Contributing

This is a community project. We invite your participation through issues and pull requests! You can peruse the contributing guidelines and see Contributing Notes below.

Building

The package is written in TypeScript. To build the package run the following from the root of the repo:

npm run build # It will be built in /dist

Release Process (Deploying to NPM)

We use semantic-release to consistently release semver-compatible versions. This project deploys to multiple npm distribution tags. Each of the below branches correspond to the following npm distribution tags:

branch	npm distribution tag
master	latest
beta	beta

To trigger a release use a Conventional Commit following Angular Commit Message Conventions on one of the above branches.

Todo / Roadmap

operations:

• [x] group (see d3.group, uses map)

• [ ] groups (see d3.groups, uses arrays not map)

• [ ] head

• [ ] tail

• [ ] reduce

• [ ] includes: returns true (?) when any one element satisfies predicate

• [ ] some: alias for includes

• [ ] every: returns true (?) when EVERY one element satisfies predicate

• [ ] none: returns true (?) when NO element satisfies predicate

• [ ] each: call an Action for every item.

• [ ] zip

• [ ] without: yields the items that do not satisfy the predicate (i.e. opposite of filter)

• [ ] with: alias for filter

• [ ] where: alias for filter

• Operations that should not be added:

  • Most operations that necessarily require a full iteration or a full count of items (e.g. unique, sort, reverse, sample) should just call collect and use other methods to perform the operation.
  • Possible Exceptions:
    • ...when they're extremely like find so you don't have to keep importing alternatives.
    • ...when the option is reduced to a single value or a smaller set of values in a single iteration of the elements (e.g. count, reduce).
    • ...when array has methods that mutate the array (reverse, sort)?

Roadmap Ideas

SQL-Style API

• maybe a sql-style API like:

from(iterator)
  .where(v => ...)
  .select(v => { foo: v.foo, bar: v.bar })  // i.e. alias for "map" function
  .groupBy(v => v.foo) // [ ["foo-value1", [v1, v2, v3]], ... ]

ChainAsync Rejection Handling Options

  /*
   * Like collect but allows replacing any rejected promise with a substitute value rather than rejecting.
   * @param rejectHandler
  collectDefault(rejectHandler: (reason: any) => Promise<Array<TItem>>): Promise<Array<TItem>>
  /*
   * Like collect but allows skipping any rejected promises rather than rejecting.
  collectSkipRejections(rejectHandler: (reason: any) => Promise<Array<TItem>>): Promise<Array<TItem>>
   */

Contributing Notes

Some notes for contributors...

for...of is fast enough

In our tests for...of is basically equivelent to manual iteration performancing when using TypeScript on nodejs. When using the ES5 target in TypeScript, the TypeScript compiler unwraps for..of to the manual iteration syntax anyway. In ES6 it emits for..of

For node v15.0.1 ES6 manual iteration was slightly faster at an ~0.124ms versus ~0.135ms for for...of (average of 1K iterations).

The for...of code used for testing:

for (const item of iterable) {
  if (predicate(item)) {
    yield item
  }
}

The manual iteration code used for testing:

const iterator: Iterator<TItem, any, undefined> = iterable[Symbol.iterator]()
let value: TItem
for (let next = iterator.next(); !next.done; next = iterator.next()) {
  value = next.value
  if (predicate(value)) yield value
}

1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Iterators_and_Generators ↩

2. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncIterator ↩