Never Publish Malarkey

    run-db

    1.0.32 • Public • Published

    Run-DB

    tests codecov

    Crawls the blockchain and indexes RUN state.

    Using Run-DB, you can self-host the State APIs that Run uses to work well.

    Use Run-DB to:

    • Operate a State Server to improve RUN performance by pre-loading jigs
    • Query balances, volume, history, and other information across many users and contracts
    • Blacklist individual transactions and their descendants in your app
    • Create your own local database of transactions your app uses

    Requirements

    Node 10+

    Getting started

    1. Install npm run install
    2. Download a db snapshot: wget https://run.network/run-db-snapshots/main/latest -O run.db (optional)
    3. Run npm run start
    4. Install a trustlist: curl -s https://api.run.network/v1/main/trust | curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8000/trust (optional)

    Note: For testnet, you may use test in place of main in the above commands.

    Use with your App Server

    Setup your server's Run instance as follows:

    const client = true
    const state = new Run.plugins.RunDB('http://localhost:8000')
    const trust = ['state']
    const run = new Run({ client, state, trust })

    Client mode makes Run-DB the source of truth for your server for all jig information. RUN will not load jigs that are not in your database, and your inventory will only be populated by jig UTXOs known to your database.

    Setting trust to 'state' makes Run use your database for its trustlist too. This means you only have to setup trust in one place using:

    curl -X POST localhost:8000/trust/<txid>
    

    You may also want to run additional instance of Run-DB in SERVE_ONLY mode. That allows you to have an writer that crawls transactions and puts data into the database, and multiple readers that serve your application servers.

    Use with a Browser or Mobile Client

    The same approach taken for servers can be used to improve performance of client Run instances. You should expose your Run-DB endpoints on a public or private domain rather than connect to localhost. If your client connections are not authenticated, be sure to only expose the GET endpoints and never the POST or DELETE endpoints, and use HTTPS to prevent MITM attacks.

    Configuration

    Create a .env file or set the following environment variables before running to configure the DB.

    Name Description Default
    API mattercloud, planaria, bitcoin-node, run, or none mattercloud
    MATTERCLOUD_KEY Mattercloud API key undefined
    PLANARIA_TOKEN Planaria API key undefined
    ZMQ_URL Only for bitcoin-node. ZMQ tcp url null
    RPC_URL Only for bitcoin-node. bitcoin RPC http url null
    NETWORK Bitcoin network (main or test) main
    DB Database file run.db
    PORT Port used for the REST server randomly generated
    WORKERS Number of threads used to index 4
    FETCH_LIMIT Number of parallel downloads 20
    START_HEIGHT Block height to start indexing block shortly before sep 2020
    TIMEOUT Network timeout in milliseconds 10000
    MEMPOOL_EXPIRATION Seconds until transactions are removed from the mempool 86400
    DEFAULT_TRUSTLIST Comma-separated values of trusted txids predefined trustlist
    SERVE_ONLY Whether to only serve data and not index transactions false

    Connecting with a bitcoin node

    During development is useful to connect to a local node. In order to do this you need to provide RUN-db with access to a bitcoin node trough RPC and ZMQ.

    export API="bitcoin-node"
    export ZMQ_URL="tcp://your-node-uri:port"
    export RPC_URL="http://user:password@your-node-uri:port"
    

    The only zmq message is needed is rawtx. ZMQ is only used to get the new transactions in the mempool.

    Direct connection with the node is tested in regtest and testnet, but it's not recommeded for production environments in mainnet at the moment.

    Endpoints

    • GET /jig/:location - Gets the state for a jig at a particular location

    • GET /berry/:location - Gets the state for a berry at a particular location

    • GET /tx/:txid - Gets the raw transaction hex for an added transaction

    • GET /time/:txid - Gets the block or mempool time of a transaction in seconds since unix epoch

    • GET /spends/:location - Gets the spending txid for an output at a particular location

    • GET /unspent - Gets the locations of all unspent jigs that are trusted. You may optionally pass in the following query params: class to filter by contract origin, address to filter by owner address, pubkey to filter by owner pubkey, scripthash to filter by hash of the owner script, lock to filter by lock class origin.

    • GET /trust/:txid? - Gets whether a particular txid is trusted, or the entire trust list

    • GET /ban/:txid? - Gets whether a particular txid is banned, or the entire ban list

    • GET /status - Prints status information

    • POST /trust/:txid? - Trusts a transaction to execute its code, as well as any untrusted ancestors. To trust multiple transactions at once, you may add an array of txids in the body as application/json.

    • POST /ban/:txid - Bans a transaction from being executed, and unindexes it and its descendents

    • POST /tx/:txid? - Indexes a transaction and any ancestors. You may optionally add the raw hex data for the transaction in the body as text/plain.

    • DELETE /trust/:txid - Removes trust for a transaction, and unindexes it and its descendents

    • DELETE /ban/:txid - Removes a transaction ban, and reindexes it and its descendents

    • DELETE /tx/:txid - Removes a transaction, its descendents, and any connected state

    Performing Custom Queries

    Run-DB uses SQLite as its underlying database in WAL mode. SQLite and WAL allows multiple connections to the database so long as there is only one writer, which should be Run-DB. Alternatively, forking Run-DB to create new endpoints for your application may be simpler.

    Example Queries

    For some of these queries, you will need the JSON1 SQLite extension.

    Calculate SHUA supply

    SELECT SUM(amount) as supply
    FROM (
        SELECT
            json_extract(jig.state, '$.props.amount') AS amount
        FROM jig JOIN spends ON jig.location = spends.location
        WHERE spends.spend_txid IS NULL
        AND jig.class = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78_o1')
    

    Calculate SHUA token balances by owner

    SELECT owner, SUM(amount) as amount
    FROM (SELECT
            json_extract(jig.state, '$.props.owner') AS owner,
            json_extract(jig.state, '$.props.amount') AS amount
        FROM jig JOIN spends ON jig.location = spends.location
        WHERE spends.spend_txid IS NULL
        AND jig.class = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78_o1')
    GROUP BY owner
    ORDER BY amount DESC
    

    Get transaction hex

    SELECT HEX(bytes) AS hex
    FROM tx
    WERE txid = 'ce8629aa37a1777d6aa64d0d33cd739fd4e231dc85cfe2f9368473ab09078b78'
    

    Re-execute all transactions

    UPDATE tx SET executed = 0; DELETE FROM jig; DELETE FROM berry;
    

    Database Schema

    There are currently 8 tables updated by Run-DB.

    jig

    Stores jig and code states at output locations or destroyed locations.

    Column Type Description
    location TEXT Jig or code location
    state TEXT JSON string describing the object state
    class TEXT Contract origin if this state is a jig
    scripthash TEXT Hex string of the reversed sha256 of the owner script
    lock TEXT Lock class origin if this state has a custom lock

    tx

    Stores all transactions known by Run-DB and their indexing state.

    Column Type Description
    txid TEXT Hex string for the transaction hash
    height INTEGER Block height for this transaction, or -1 for mempool, or NULL for unknown
    time INTEGER Transaction or bock time in seconds since the unix epoch
    bytes BLOB Raw transaction data, or NULL if not downloaded
    has_code INTEGER 1 if this transaction deployed or upgraded code and requires trust, 0 otherwise
    executable INTEGER 1 if this transaction is a valid RUN transaction, 0 otherwise
    executed INTEGER 1 if this transaction was executed, even if it failed, 0 otherwise
    indexed INTEGER 1 if this transaction's jig states were calculated successfully, 0 otherwise

    spends

    Stores spend information about transaction outputs.

    Column Type Description
    location TEXT <txid>_o<output-index> string describing an output
    spend_txid TXID Hex txid that spent this output, or NULL if unspent

    deps

    Stores the transaction needed to load a RUN transaction.

    Column Type Description
    up TEXT A transaction ID in hex
    down TEXT Hex txid for a transaction that depends on up

    berry

    Stores berry states for third-party protocol data.

    Column Type Description
    location TEXT Berry location without the &hash query param
    state TEXT JSON string describing the object state

    trust

    Stores the transactions which have been trusted and whose code will be executed.

    Column Type Description
    txid TEXT Hex string txid
    value INTEGER 1 if trusted, 0 if untrusted

    ban

    Stores the transactions which have been blacklisted.

    Column Type Description
    txid TEXT Hex string txid
    value INTEGER 1 if blacklisted, 0 otherwise

    crawl

    Stores the crawled block tip height and hash for data in the database.

    Column Type Description
    key TEXT 'height' or 'hash'
    value TEXT String value for the key

    Install

    npm i run-db

    DownloadsWeekly Downloads

    4

    Version

    1.0.32

    License

    MIT

    Unpacked Size

    19.9 MB

    Total Files

    271

    Last publish

    Collaborators

    • libitx
    • brentongunning
    • calvo.generico