kwildb

    1.2.28 • Public • Published

    API for connecting with a Kwil Database

    We will write and link documentation for how to set up a Kwil Database soon.

    Introduction

    This repo is a basic toolkit for interacting with Kwil Databases. Please note that, while we try our hardest to create the highest quality developer experience, that there will be some bugs, as up until very recently, Kwil Databases were only used by Kwil Social. Please notify us of any bugs, errors, or improvements you would like to see by emailing brennan@kwil.com.

    At face value, a Kwil Database is very similar to a normal relational database. Under the hood, each node uses a Postgresql database for storing its own data. Each node contains an application built around this database to allow for inter-database synchronization, distributed filesystem capabilities, and snapshot backups (stored on the blockchain, of course!).

    In the near future, we can expect functionality for fully encrypted distributed databases, a native ORM / SQL Builder, and the ability to switch between instant and block-based syncrhonization methods.

    Initialize:

    npm install kwildb
    
    //Node.js
    const KwilDB = require('kwildb')
    //JS
    import KwilDB from 'kwildb';

    Creating a Kwil Data Moat

    Any person can create a moat on Kwil DB. For the time being, we are allowing anybody to do so for free, however in the future, it will cost a token.

    const myMoat = await await KwilDB.createMoat('https://test-db.kwil.xyz', 'test-moat', 'mySuperSecretPassword', 'myWalletAddr')
    const privateKey = myMoat.privateKey
    const secret = myMoat.secret

    In testnet, there is no way to retrieve a lost private key or secret. The only way is for validators to vote to change the public key and for that private key to change the secret, however during testnet, this is not possible.

    Basic Queries

    In order to access a Kwil Database, the client should use the KwilDB connector. The connector interfaces with the Kwil Database, which then handles interactions with the underlying relational database (Postgres by default). When accessed through the connector, the Kwil Database automatically handles synchronization. All the client needs to do is specify if they want each query to propagate to the rest of the network.

    In general, you will want all write operations to sync, while read operations do not sync. By default, queries will not propagate.

    const kwilDB = KwilDB.createConnector({
        host: 'test-db.kwil.xyz',
        protocol: 'https',
        moat: 'test',
        privateKey: privateKey,
    }, secret)
    
    //Query that will sync with the rest of the network
    const result = await kwilDB.query('INSERT INTO ...', true)
    
    //Query that won't sync with the rest of the network.
    const result = await kwilDB.query('SELECT ...', false)
    
    //Or, if unspecified, it will not sync.
    
    const result = await kwilDB.query('SELECT ...')

    Images and Files

    On top of a database, Kwil Databases come loaded with a full-featured file system. Files will propagate across nodes just like queries, and can be accessed via the "public/{data_moat}" endpoint. Currently, Kwil Databases only support files with no extensions (can be used for any data) and JPEGs. The first input is the path to the file/image, the second is the file data, and the third is whether or not it should be propagated (false by default).

    It is important to note that the beginning of the file path should just be the directory name, and should NOT contain './'.

    Images that are submitted should be BASE64 encoded. Depending on how you read in the image, there may be padding at the start (if you are using the NodeJS 'fs' module, this won't be a problem). It is important that the padding is removed and only the raw image data is submitted.

    //Storing a file
    
    const settingsData = {settings: ...}
    await kwilDB.storeFile('users/satoshi/settings', settingsData, true) //Will propagate
    
    //Storing a JPEG
    
    let satoshiPFP = fs.readFileSync('./satoshiPFP')
    satoshiPFP = satoshiPFP.toString('base64')
    
    await kwilDB.storeJPEG('images/satoshi/profile_picture', satoshiPFP, true)

    Images can be found at the /public/{moat}/ endpoint.

    If the above image was sent to the data moat "satoshi-social", it would be found at https://test-db.kwil.xyz/public/satoshi-social/images/satoshi/profile_picture.jpg

    Prepared Statements

    In order to prevent against SQL injection attacks, KwilDB ships with the ability for users to submit prepared statements.

    await kwilDB.preparedStatement(`INSERT INTO user_posts (post_id, post_text, post_owner) VALUES ($1, $2, $3)`, ['wnv47vn213re', 'Hello Permaweb!', 'satoshi'], true)

    NOTE: kwilDB has been reconfigured to avoid the following issues as of version 1.2.27, this section remains for those using older versions

    Troubleshooting and known issues.

    kwildb depends on Web3, which can cause some issues when used in web development. The following are solutions for using this API with various JS frameworks.

    Web3 and Create-react-app

    If you are using create-react-app version >=5 you may run into issues building. This is because NodeJS polyfills are not included in the latest version of create-react-app.

    Solution

    • Install react-app-rewired and the missing modules

    If you are using yarn:

    yarn add --dev react-app-rewired process crypto-browserify stream-browserify assert stream-http https-browserify os-browserify url buffer

    If you are using npm:

    npm install --save-dev react-app-rewired crypto-browserify stream-browserify assert stream-http https-browserify os-browserify url buffer process
    • Create config-overrides.js in the root of your project folder with the content:
    const webpack = require('webpack');
    
    module.exports = function override(config) {
        const fallback = config.resolve.fallback || {};
        Object.assign(fallback, {
            "crypto": require.resolve("crypto-browserify"),
            "stream": require.resolve("stream-browserify"),
            "assert": require.resolve("assert"),
            "http": require.resolve("stream-http"),
            "https": require.resolve("https-browserify"),
            "os": require.resolve("os-browserify"),
            "url": require.resolve("url")
        })
        config.resolve.fallback = fallback;
        config.plugins = (config.plugins || []).concat([
            new webpack.ProvidePlugin({
                process: 'process/browser',
                Buffer: ['buffer', 'Buffer']
            })
        ])
        return config;
    }
    • Within package.json change the scripts field for start, build and test. Instead of react-scripts replace it with react-app-rewired

    before:

    "scripts": {
        "start": "react-scripts start",
        "build": "react-scripts build",
        "test": "react-scripts test",
        "eject": "react-scripts eject"
    },

    after:

    "scripts": {
        "start": "react-app-rewired start",
        "build": "react-app-rewired build",
        "test": "react-app-rewired test",
        "eject": "react-scripts eject"
    },

    The missing Nodejs polyfills should be included now and your app should be functional with web3.

    • If you want to hide the warnings created by the console:

    In config-overrides.js within the override function, add:

    config.ignoreWarnings = [/Failed to parse source map/];

    Web3 and Angular

    New solution

    If you are using Angular version >11 and run into an issue building, the old solution below will not work. This is because polyfills are not included in the newest version of Angular.

    • Install the required dependencies within your angular project:
    npm install --save-dev crypto-browserify stream-browserify assert stream-http https-browserify os-browserify
    • Within tsconfig.json add the following paths in compilerOptions so Webpack can get the correct dependencies
    {
        "compilerOptions": {
            "paths" : {
            "crypto": ["./node_modules/crypto-browserify"],
            "stream": ["./node_modules/stream-browserify"],
            "assert": ["./node_modules/assert"],
            "http": ["./node_modules/stream-http"],
            "https": ["./node_modules/https-browserify"],
            "os": ["./node_modules/os-browserify"],
        }
    }
    • Add the following lines to polyfills.ts file:
    import { Buffer } from 'buffer';
    
    (window as any).global = window;
    global.Buffer = Buffer;
    global.process = {
        env: { DEBUG: undefined },
        version: '',
        nextTick: require('next-tick')
    } as any;

    Old solution

    If you are using Ionic/Angular at a version >5 you may run into a build error in which modules crypto and stream are undefined

    a work around for this is to go into your node-modules and at /angular-cli-files/models/webpack-configs/browser.js change the node: false to node: {crypto: true, stream: true} as mentioned here

    Another variation of this problem was an issue opned on angular-cli

    Keywords

    none

    Install

    npm i kwildb

    DownloadsWeekly Downloads

    6

    Version

    1.2.28

    License

    ISC

    Unpacked Size

    93.7 kB

    Total Files

    34

    Last publish

    Collaborators

    • brennanjl
    • thomast123