warpcore

warpcore is a micro-service service-discovery framework. This stemmed out of lots of good ideas by super smart people, and is more of a proof-of-concept for making easy to setup micro-service instrumentation.

Installation

npm install --save warpcore

Concepts

Service Discovery is when a client needs to call out to an external service, but isn't given the exact configuration necessary to call the service, but instead reaches out or is told through dynamic means how to connect to that service. I'm sure there are libraries that could sniff across all ports and figure out the services it needs to connect to, but for this library, I opted for a "common peer" approach.

All nodes share a common peer, which we'll call a base. The base just acts as a central node that facilities the other nodes connecting to each other. The base will also be the notifier to tell nodes about dropped or new nodes in the mesh.

A service based node actually has two ports it listens on: one for peer updates, and another for service calls.

A client node registers which services it would like to communicate with, and when service calls are made, it will round-robin make calls out to them.

Once a client knows about a service, the base is no longer a requirement. The base can safely spin down and spin back up, and the mesh will stay intact. Clients can still make calls to services. The only thing that happens if there are no bases available is that the clients and services won't know about new services.

This module uses the swim module to facilitate the service discovery, and then warpfield to facilitate the actual service calls. The service disovery part is easy to configure. You just need to tell each node where the base(s) is(are). The service call stuff utilizes warpfield which uses Protocol Buffers to serialize and deserialize data across the wire. Right now it communicates over HTTP, but in the future can support things like tcp and http2.

Because of the way protocol buffers work, both the client and the service need to have the protocol buffer definition of the service in order to communicate.

Usage

// base.js
'use strict'
 
const warpcore = require('warpcore')
 
const PORT = 3000
const base = warpcore.base({ port: PORT })
 
base.start().then(() => console.log(`base started on port ${PORT}`)

// helloworld.proto
syntax = "proto3";
package helloworld;
 
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
}
 
message HelloRequest {
  string name = 1;
}
 
message HelloReply {
  string message = 1;
}

// service.js
'use strict'
 
const warpcore = require('warpcore')
 
const helloworld = warpcore.load(`${__dirname}/helloworld.proto`).helloworld
const PORT = 4000
const SERVICE_PORT = 4500
const BASES = '127.0.0.1:3000'
 
const member = warpcore.member({
  port: PORT,
  bases: '127.0.0.1:3000'
})
 
member.register('greeter', helloworld.Greeter)
  // Corresponds with the `rpc SayHello` in the Greeter service
  .use('sayHello', (call) => {
    return { message: `Hello ${call.request.name}` }
  })
 
member.start(SERVICE_PORT)
  .then(() => console.log(`greeter service started`)

// member.js
'use strict'
 
const warpcore = require('warpcore')
const helloworld = warpcore.load(`${__dirname}/helloworld.proto`).helloworld
const PORT = 5000
const BASES = '127.0.0.1:3000'
 
const member = warpcore.member({
  port: PORT,
  bases: BASES
})
 
const greeter = member.client('greeter', helloworld.Greeter)
 
setInterval(() => {
  greeter.sayHello({ name: 'Jack Bliss' })
    .then((res) => console.log('sayHello response': res))
}, 5000)
 
member.start()

Now, in three different terminal tabs, run:

node base

node service

node client

API

Base

const warpcore = require('warpcore')
const base = warpcore.base(options)

• options.host (String) - The host this node can be reached at. Defaults to '127.0.0.1'.
• options.port (String|Number) - The port to listen for gossip chatter. This option is required
• options.bases (String[]|String) - A list of bases. Can be an array of hosts or a comma delmited string of hosts. Defaults to []
• options.swim (Object) - Additional options to pass to the swim module.
• retryInterval (Number) - The interval (in milliseconds) at which to attempt reconnects to the base if it becomes disconnected. Defaults to 1000.

base.on(event, handler)

• event (String) - The event name to listen on. Can be 'add' (when a node is added), 'remove' (when a node is removed), or 'reconnect' (when the base was able to reconnect to another base).
• handler (Function) - The function that gets called when the event happens. On the 'add' and 'remove' events, it passes an object that looks like this:

  {
    host: '127.0.0.1:8000',
    incarnation: 1466684469736, // When it joined the mesh
    meta: {
      id: '127.0.0.1:8000~1466684469736~c2271ad1-0279-451c-a326-05d7ae6db4ca',
      version: '1.0.0', // The version of warpcore being run
      base: false, // Whether or not the node is a base
      hostname: '127.0.0.1', // Just the raw hostname without the port
      // The next two keys are only available if the node is a member, and not
      // a base
      services: [
        'greeter'
      ],
      port: 3000 // Could be 'null' if there are no services
    }
  }

base.start()

Starts listening for node connections. Returns a promise which resolves when the initial connection is made.

base.end()

Stops listening for node connections and disconnects itself from the mesh.

base.members

An array of member objects that look like the ones that come through the event handler.

Protocol Buffers

// helloworld.proto
syntax = "proto3";
package helloworld;
message HelloRequest {
  string name = 1;
}
message HelloReply {
  string message = 1;
}
service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply)
}

const warpcore = require('warpcore')
const proto = warpcore.load(__dirname, 'helloworld.proto') // arguments work like path.resolve or path.join
const helloworld = proto.helloworld // .helloworld comes from `package helloworld;`
const GreeterService = proto.Greeter // .Greeter comes from `service Greeter {}`

Service

const warpcore = require('warpcore')
const service = warpcore.service(protobufService, options)

• protobufService - This is the Service class pulled from the protocol buffer file. In the Protocol Buffers section, it's the GreeterService at the bottom of the example.
• options.context - The context to bind all of the service handlers.

service.handle(name, handler)

Registers a handler to a given rpc method. Returns the service so that you can chain the method calls.

• name - The name of the method to use. This should be the lowerCamelCase version of the rpc method name. SayHello should turn into 'sayHello'.
• handler - The function to call when the service is called. The object will be in the shape described in the protocol buffer definition and is available as the first argument

  service.use('sayHello', (request) => {
    console.log(request) // This would have a 'name' property
  })

  The object you return must match the protocol buffer definition. You may also return a promise that returns the object that matches the protocol buffer definition.

Member

const warpcore = require('warpcore')
const member = warpcore.member(options)

• options.host (String) - The host this node can be reached at. Defaults to '127.0.0.1'.
• options.port (String|Number) - The port to listen for gossip chatter. This option is required
• options.bases (String[]|String) - A list of bases. Can be an array of hosts or a comma delmited string of hosts. Defaults to []
• options.swim (Object) - Additional options to pass to the swim module.
• retryInterval (Number) - The interval (in milliseconds) at which to attempt reconnects to the base if it becomes disconnected. Defaults to 1000.

member.use(name, service)

Register a service with the warpcore member.

• name (String) - The name the service should be registered with.
• service (warpcore.Service|Protobuf Service) - The actual service object to register. This can be a warpcore service instance (created with warpcore.service() or it can be a protocol buffer service.

Returns the warpcore service instance which can be used to call .use() to handle methods.

member.client(name, proto)

Creates a client for a service.

• name (String) - The name of the service to register with
• proto (Protobuf Service) - The protocol buffer service to use to serialize and deserialize data.

Returns an object who's methods line up with the service methods.

member.call(serviceName, methodName, body)

Calls a service method

• serviceName (String) - The service to call
• methodName (String) - The method to call
• body (Object) - The object to pass to the service method

Returns a promise that resolves with the response object

member.start(options)

Starts the service/warpcore membership

• options.port (Number|String) - The port to bind any services to. This is required if any services were registered, and must be different from the warpcore member port.

Returns a promise that resolves when it has started the services.

member.stop()

Stops all services. Returns a promise that resolves when it has stopped everything.

Examples

See the examples in the examples/ folder.

CLI

There is a warpcore cli, but it is currently in the experimental stages. There aren't any tests written around it, and is subject to break the api at any time. That being said, I would love any feedback in your uses of it.

  Usage: warpcore [options]

  A CLI to start warpcore services

  Options:

    -h, --help                    output usage information
    -V, --version                 output the version number
    -c, --config <path>           Path to config file [warpcore.config.js]
    -b, --base                    Enable node as base
    -B, --bases <value>           Comma-delimited list of base hosts
    -H, --host <value>            The hostname this node can be reached at
    -p, --port <n>                The warpcore port
    -P, --service-port <n>        The service port
    -s, --services <name>:<path>  Service name and path. Can add more than one service.
    -r, --retry-interval <n>      The interval to retry base reconnection

The config file can be a .js or .json file. Right now it just uses a simple require() statement to include the file, so no fancy comment removing or anything like that.

This is an example config file (these are not the defaults:

module.exports = {
  base: false,               // Whether this instance is a base or not
  bases: '127.0.0.1:8000',   // Comma separated (or array) of bases
  host: '127.0.0.1',         // Hostname of this node
  port: '8001',              // Warpcore gossip port
  servicePort: '8002',       // Warpcore service port
  retryInterval: 1000,       // Interval at which to retry connection to base
 
  swim: {}                   // Options to pass to swim module
  services: {
    echo: './echo.js'        // Key/Value pairs where the key is the name of the
                             // service and the value is a path to the service
                             // module (relative to the config file) or the
                             // actual service object, as seen below
    echo2: require('./echo')
  }
}

Not all values need to be passed in. For example, if this is a base node, there is no need to pass in services or servicePort.

With the exception of the swim option, everything can be passed in via the command line arguments. The --service <name>:<path> option will assume the path is relative to the current working directory, whereas if it's passed into the config file, it's relative to the config file.

Questions

Feel free to submit questions as an issue. We're not big enough to have questions in a separate service.