cardex

Indexes for CARs.

Implementations of CARv2 indexes in JavaScript. Status:

• IndexSorted ✅ (complete & compatible)
• MultihashIndexSorted ✅ (complete & compatible)

Install

npm install cardex

Usage

Write index

import fs from 'fs'
import { Readable } from 'stream'
import { CarIndexer } from '@ipld/car/indexer'
import { IndexSortedWriter } from 'cardex'

const carStream = fs.createReadStream('my.car')
const indexer = await CarIndexer.fromIterable(carStream)

const { readable, writable } = new TransformStream()
const writer = IndexSortedWriter.createWriter({ writer: writable.getWriter() })

readable.pipeTo(Readable.toWeb(fs.createWriteStream('my.car.idx')))

for await (const { cid, offset } of indexer) {
  await writer.add(cid, offset)
}
await writer.close()

Read index

import fs from 'fs'
import { Readable } from 'stream'
import { IndexSortedReader } from 'cardex'

const carStream = fs.createReadStream('my.car.idx')
const reader = IndexSortedReader.createReader({ reader: Readable.toWeb(carStream).getReader() })

while (true) {
  const { done, value } = await reader.read()
  if (done) break
  console.log(`${Buffer.from(value.digest).toString('hex')} @ ${value.offset}`)
}

Multi-index index

The multi-index index is a custom index allowing multiple CAR indexes to be grouped together in a single index.

Write multi-index

import { MultihashIndexSortedWriter } from 'cardex'
import { MultiIndexWriter } from 'cardex/multi-index'

const { readable, writable } = new TransformStream()
const writer = MultiIndexWriter.createWriter({ writer: writable.getWriter() })

readable.pipeTo(new WritableStream()) // destination

writer.add(carCID0, async ({ writer }) => {
  const index0 = MultihashIndexSortedWriter.createWriter({ writer })
  index0.add(cid, offset)
  await index0.close()
})

writer.add(carCID1, async ({ writer }) => {
  const index1 = MultihashIndexSortedWriter.createWriter({ writer })
  index1.add(cid, offset)
  await index1.close()
})

await writer.close()

Read multi-index

import { MultihashIndexSortedReader, IndexSortedReader } from 'cardex'
import { MultiIndexReader } from 'cardex/multi-index'

const readable = new ReadableStream() // reader of a multi-index bytes
const reader = MultiIndexReader.createReader({ reader: readable.getReader() })

// add readers to the multi-index reader (to allow reading different index types)
reader.add(MultihashIndexSortedReader)
reader.add(IndexSortedReader)

while (true) {
  const { done, value } = await reader.read()
  if (done) break

  const { origin, multihash, digest, offset } = value // (origin is a CAR CID)
  console.log(`${origin} -> ${CID.createV1(raw.code, multihash)} @ ${offset}`)
}

Universal reader

The universal reader is for when you don't know what type of CARv2 index you're reading. The universal reader automatically instantiates the correct reader for a given index based on the codec:

import fs from 'fs'
import { Readable } from 'stream'
import { UniversalReader } from 'cardex/universal'

const carStream = fs.createReadStream('my.car.idx')
const reader = UniversalReader.createReader({ reader: Readable.toWeb(carStream).getReader() })

while (true) {
  const { done, value } = await reader.read()
  if (done) break
  console.log(`${Buffer.from(value.digest).toString('hex')} @ ${value.offset}`)
  // Note: `value` might have `multihash` if reading from MultihashIndexSorted
  // and it might have `origin` if reading from `MultiIndex`.
}

Contributing

Feel free to join in. All welcome. Open an issue!

License

Dual-licensed under MIT + Apache 2.0