@commonwealth/chain-events

"@commonwealth/chain-events" is a library for subscribing and processing synthetic blockchain events.

Installation

Available on npm and designed to work both in browser and with nodejs.

yarn add @commonwealth/chain-events

For developing on this project itself, first you must build the project to replicate the npm package structure (using the typescript compiler), and then you can install your local version via yarn link:

~/chain-events$ yarn build
~/chain-events$ yarn link
~/chain-events$ cd ~/project-name
~/project-name$ yarn link @commonwealth/chain-events

Be sure to call yarn unlink once development has been completed and the new changes have been published.

Please submit any enhancements or bug fixes as a Pull Request on the project's github page.

Development

npm install -g npm-install-peers

For using a local version of Chain Events in other projects, we recommend you use yalc, which functions as a local package repository for your npm libraries in development.

To install yalc, run:

$ yarn global add yalc

Then, publish Chain Events to the yalc respository (which will first build the project):

~/chain-events$ yalc publish

Navigate to the project you want to test Chain Events inside, and use yalc to add it. This will update its package.json to point the "@commonwealth/chain-events" dependency to a local file.

~/commonwealth$ yalc add @commonwealth/chain-events
~/commonwealth$ yarn

Any time you update Chain Events after publishing and adding, simply run the following to build and propagate a new update:

~/chain-events$ yalc publish --push

Publishing

First ensure you bump the package version in the package.json file. Then build, and publish to the npm repository. A --dry-run is useful beforehand to ensure the version and file lists are correct.

~/chain-events$ yarn build
~/chain-events$ npm publish [--tag <tag>] --dry-run
~/chain-events$ npm publish [--tag <tag>]

Publishing Types

First navigate to types then bump the package version and publish.

~/chain-events/types$ npm publish [--tag <tag>] --dry-run
~/chain-events/types$ npm publish [--tag <tag>]

Standalone Usage

This package includes an "event listener" script located at listener.ts, which permits real-time listening for on-chain events, and can be used for testing a chain connection, pushing events to a queue, or/and running chain-events as a node.

The following is an example usage, connecting to a local node running on edgeware mainnet:

~/chain-events$ yarn build
~/chain-events$ yarn listen -n edgeware -u ws://localhost:9944

The full set of options is listed as, with only -n required:

Options:
      --help             Show help                                     [boolean]
      --version          Show version number                           [boolean]
  -z, --config           Path to a config file to setup multiple        [string]
                         listeners (see below)
  -n, --network          chain to listen on
          [required] [choices: "edgeware", "edgeware-local", "edgeware-testnet",
     "kusama", "kusama-local", "polkadot", "polkadot-local", "kulupu", "moloch",
                                                                 "moloch-local"]
  -u, --url              node url                                       [string]
  -a, --archival         run listener in archival mode                 [boolean]
  -b, --startBlock       when running in archival mode, which block     [number]
                         should we start from
  -s, --skipCatchup      Whether to attempt to retrieve historical     [boolean]
                         events not collected due to down-time
  -c, --contractAddress  eth contract address                           [string]
  -q, --rabbitmq         Publish messages to queue hosted on RabbitMQ  [boolean]
  -e, --eventNode        Run chain-events as a node that allows        [boolean]
                         interacting with listeners over http
                         (only updating substrate specs for now)

If the -z option is passed then only -q and -e can be used (all other options conflict with the config defined by -z)

Environment Variables

• NODE_ENV: dictates where a listener will get its initial spec. when NODE_ENV = "production" the listener gets its spec from commonwealth.im. Otherwise, the listener will get its spec from the commonwealth server hosted locally.

Listener config file

Must be a json file with the following format:

[
  {
    "network": "Required (string) - The name of the network",
    "url": "Optional (string) - Node url to connect to",
    "archival": "Optional (boolean) - run listener in archival mode",
    "startBlock": "Optional (number) - when running in archival mode, which block should we start from",
    "skipCatchup": "Optional (boolean) - Whether to attempt to retrieve historical events not collected due to down-time",
    "excludedEvents": "Optional (array of strings) - An array of EventKinds to ignore. Currently only relevant for the RabbitMQ producer."
  }
]

See manyListenerConfigEx.json for an example configuration

Library Usage

The easiest usage of the package involves using the Listener class which initializes the various components. Do this for Substrate chains as follows:

import { Listener as SubstrateListener } from "";

// TODO: listener argument docs
// create a listener instance
const listener = new SubstrateListener();

// initialize the listener
await listener.init();

// subscribe/listen to events on the specified chain
await listener.subscribe();

The Listener classes have a variety functions that facilitate using the listener.

Updating the substrate spec

await listener.updateSpec({yourNewSpec})

Updating the url the listener should use

await listener.updateUrl('yourNewUrl')

Changing the event handlers

The event handlers are accessible through the eventHandlers property. The eventHandlers property is defined as follows:

eventHandlers: {
  [handlerName: string]: {
    "handler": IEventHandler,
    "excludedEvents": SubstrateEvents[]
  }
}

Thus, to change an event handler, or the events that it ignores simply access it directly:

// change the handler of "myEventHandler"
listener.eventHandlers["myEventHandler"].handler = newHandler;

Changing the excluded events

As described above you can change the events that a handler ignores either directly in the execution of the handler or by setting "excludedEvents" like so:

// change the events "myEventHandler" excludes
listener.eventHandlers["myEventHandler"].excludedEvents = ["someEventKind", "anotherEventKind"]

You can also exclude events from all handlers at one by changing the globalExcludedEvents property like so:

listener.globalExcludedEvents = ["someEventKind", "anotherEventKind"]

Provided Handlers

RabbitMQ Producer

HTTP Post Handler

Single Event Handler

Custom Handlers

A custom handler is necessary in many cases depending on what you are trying to build. Thankfully creating your own is very easy!

Just extend the IEventHandler and implement the handle method:

import {CWEvent, IEventHandler} from "chain-event-types"

class ExampleEventHandler implements IEventHandler {
  public async handle(event: CWEvent): Promise<void> {
    // your code goes here
  }
}

In order to use chain-event-types in your project you will need to install chain-event-types from 'git+https://github.com/timolegros/chain-events.git#build.types' and have the following dev dependencies:

• '@polkadot/types'
• '@polkadot/api'

The easiest usage of the package involves calling subscribeEvents directly, which initializes the various components automatically. Do this for Substrate as follows.

import { spec } from '@edgeware/node-types';
import { SubstrateEvents, CWEvent, IEventHandler } from '@commonwealth/chain-events';

// This is an example event handler that processes events as they are emitted.
// Add logic in the `handle()` method to take various actions based on the events.
class ExampleEventHandler extends IEventHandler {
  public async handle(event: CWEvent): Promise<void> {
    console.log(`Received event: ${JSON.stringify(event, null, 2)}`);
  }
}

async function subscribe(url) {
  // Populate with chain spec type overrides
  const api = await SubstrateEvents.createApi(url, spec);

  const handlers = [ new ExampleEventHandler() ];
  const subscriber = await SubstrateEvents.subscribeEvents({
    api,
    chain: 'edgeware',
    handlers,

    // print more output
    verbose: true,

    // if set to false, will attempt to poll past events at setup time
    skipCatchup: true,

    // if not skipping catchup, this function should "discover" the most
    // recently seen block, in order to limit how far back we attempt to "catch-up"
    discoverReconnectRange: undefined,
  });
  return subscriber;
}

Alternatively, the individual Subscriber, Poller, StorageFetcher, and Processor objects can be accessed directly on the SubstrateEvents object, and can be set up directly. For an example of this, see the initialization procedure in subscribeFunc.ts.

Class Details

The top level @commonwealth/chain-events import exposes various abstract types from the interfaces.ts file, as well as "per-chain" modules, e.g. for Substrate, SubstrateTypes and SubstrateEvents, with the former containing interfaces and the latter containing classes and functions.

The two main concepts used in the project are "ChainEvents" and "ChainEntities".

• A "ChainEvent" represents a single event or extrinsic call performed on the chain, although it may be augmented with additional chain data at production time. ChainEvents are the main outputs generated by this project.
• A "ChainEntity" represents a stateful object on chain, subject to one or more "ChainEvents" which manipulate its state. The most common usage of ChainEntity is to represent on-chain proposals, which may have a pre-voting phase, a voting phase, and a period post-voting before the proposal is marked completed, each phase transition represented by events that relate to the same object. This project defines types and simple utilities for ChainEntities but does not provide any specific tools for managing them.

Each chain implements several abstract classes, described in interfaces.ts. The list for Substrate is as follows:

• Subscriber exposes a subscribe() method, which listens to the chain via the API and constructs a synthetic Block type when events occur, containing necessary data for later processing.
• Poller exposes a poll() method, which attempts to fetch a range of past blocks and returns an Array of synthetic Blocks. This is used for "catching up" on past events.
• StorageFetcher exposes a fetch() method, which queries chain storage and constructs "fake" Blocks, that represent what the original events may have looked like. This is used to quickly catch up on stateful Chain Entities from chains that prune past blocks (as most do).
• Processor exposes a process() method, which takes a synthetic Block and attempts to convert it into a CWEvent (aka a ChainEvent), by running it through various "filters", found in the filters directory. The primary filter types used are as follows:
  • ParseType uses data from the chain to detect the ChainEvent kind of a Block. It is used to quickly filter out blocks that do not represent any kind of ChainEvent.
  • Enrich uses the API to query additional data about a ChainEvent that did not appear in the original Block, and constructs the final CWEvent object. This is used because many "events" on chains provide only minimal info, which we may want to augment for application purposes.
  • Two other filters exist, which are not used by the Processor, but may be useful in an application:
    • Title takes a kind of ChainEvent and produces an object with a title and description, useful for enumerating a human-readable list of possible ChainEvents.
    • Label takes a specific ChainEvent and produces an object with a heading, a label, and a linkUrl, useful for creating human-readable UIs around particular events. The linkUrl property in particular is currently specific to Commonwealth, but may in the future be generalized.

Note that every item on this list may not be implemented for every chain (e.g. Moloch does not have a Poller), but the combination of these components provides the pieces to create a more usable application-usable event stream than what is exposed on the chain.

Usage as Commonwealth Chain-Events DB Node

Running chain-events as a CW DB node lets us run a cluster of chain-events node each with multiple listeners without needing for each of them to be aware of each other or implementing load-balancing. This is achieved by having the chain events DB nodes poll the database for the information that is specific to them.

####Environment Variables

• NUM_WORKERS: The total number of chain-events DB nodes in the cluster. This is used to ensure even separation of listeners among the different chain-events DB nodes.

• WORKER_NUMBER: The unique number id that this chain-events DB node should have. Must be between 0 and NUM_WORKERS-1

• HANDLE_IDENTITY: ("handle" || "publish" || null)

  • handle: The node will directly update the database with identity data
  • publish: The node will publish identity events to an identity queue
  • null: The node will not query the identity cache

• NODE_ENV: ("production" || "development") - optional

• DATABASE_URL: The url of the database to connect to. If NODE_ENV = production this url is the default.