pulsar-rest-api

REST API for executing Pulsar tasks.

(unstable, currently in development)

pulsar-rest-api

HTTP REST API for executing pulsar jobs.

Package is in nodejs and is available through npm registry:

npm install pulsar-rest-api [-g]
  • Prepare valid config.
  • Verify that mongodb is up and running.
  • Run the instance with command pulsar-rest-api -c 'your_config.yaml'

The instance of pulsar-rest-api can be run with different options. All of the options can be specified in the config. To run the instance with your own config you need to specify the filepath to it with the flag -c. For example pulsar-rest-api -c '~/my_pulsar_config.yaml'.

The default config is config.yaml and it can be found in bin directory of the pulsar-rest-api installation.

Please read carefully through the format of the config below. The options that are marked as required must be present in your config otherwise the instance won't start. There are no options that have default value. All values should be clearly defined.

  • port: required. Port where server listens for requests.
  • logPath: required. A file path where to write logs. Includes filename of the log. Can be absolute or relative.
  • mongodb: mongoDB connection parameters
    • host: required. hostname
    • port: required. port
    • db: required. database name. Now there is only one collection 'jobs' will be used.
  • pulsar:
    • repo: optional. Pulsar configuration repository. If omitted then pulsar rules applied.
    • branch: optional. Branch for pulsar configuration repository. If omitted then pulsar rules applied.
  • authentication: optional. Authentication. Only if presented it should have its required options to be filled, otherwise no need to fill authentication.githubOauthId and etc.
    • githubOauthId: required. Github OAuth Application ID.
    • githubOauthSecret: required. Github OAuth Application Secret.
    • githubOrg: required. Github organization. User needs to be member of that organization to get access to the interface of pulsar-rest-api.
    • baseUrl: required. URL where the pulsar-rest-api instance would have its web interface.
    • authorization: optional. Authorization. If you use an organization in this section, you should consider the restriction that only the users with public access in the organization would have the corresponding role of the organization. To remove this restriction the Pulsar-Rest-Api would require a user scope for the user's Github account which is too much. For more information read https://developer.github.com/v3/orgs/. Also the current role model does not have any inheritance or any other relation in it. That means if user has the write role, he is still forbidden to to the actions that require the read role. So, for solving this you need to mention user in the both roles descriptions.
      • read: required. Github users or organizations that have the read role.
      • write: required. Github users or organizations that have the write role.
  • ssl: optional. Only if presented it should have its required options to be filled, otherwise no need to fill ssl.key and etc.
    • key: required if pfx isn't presented. Ssl private key file. Combine with cert option.
    • cert: required if pfx isn't presented. Ssl public certificate file. Combine with key option. Append CA-chain within this file.
    • pfx: required if key or cert options aren't presented. Ssl pfx file (key + cert). Overrides key and cert options.
    • passphrase: optional. File containing the ssl passphrase.
  • Go to https://github.com/settings/applications/new.
  • There are going to be the fields. Fill them up. For example here is the values that I've used on my local setup:
    • Application name = pulsar-rest-api
    • Homepage URL = https://api.pulsar.local:8001. That value should be the same as baseUrl from the config.
    • Application description. It's optional. You can leave it empty.
    • Authorization callback URL. It's optional. You can leave it empty and it will be the same as Homepage URL.
  • Submit them and you will receive Client ID and Client Secret. They are githubOauthId and githubOauthSecret correspondingly.

The authorization model is separate from the logic. Because of that if you want to have authorization for some url endpoint, you need to set an authorization rule separately from the url. Those rules can be set only as the parameter-function restrictions of authentication.installHandlers. For example, Pulsar-Rest-Api has a 'Create Job' url endpoint /:app/:env. If you want to restrict this action to the write role then you need to have the next call to authentication.installHandlers:

  authentication.installHandlers(app, function(authorization) {
    router.post('/:app/:env', authorization.restrictTo('write'));
  });

The parameter-function restrictions has the instance of Authorization module as its the only argument.

The mongodb instance that you defined in your config should be up and running before you start the pulsar.

pulsar-rest-api -c 'your_config.yaml'. After that web interface should be browsable through url defined in authentication.baseUrl.

To run these tests you need the running instance of mongodb. The required configuration of mongodb can be found in test/config.yaml, section mongodb. When mongodb is running type in console npm test.

To see how the server is working you need to run its instance and open http://localhost:8001 to see its web interface. Do not forget that you may have another port in your config and hence you will need to adjust the port of page url.

To create a job type in console: curl -H "Content-Type: application/json" -X POST -d '{"task":"dummy:my_sleep"}' http://localhost:8001/example/production You can see the result in the web interface. Do not forget that you will need these application, environment and task to be present in your pulsar configuration.

:app - application name (e.g. foobar)

:env - environment name (e.g. production)

:task - capistrano task

:id - pulsar job ID

GET /jobs?currentPage=:currentPage&pageSize=:pageSize

:currentPage - number of page of job list to fetch. Possible values: Number. Default: 0. :pageSize - number of jobs per page. Possible values: Number. Default: 10.

HTTP response code 200

{
  "url": "http://api.pulsar.local:8001/pulsar/index.html",
  "jobs": [{Object}, {Object}, ...]
}

See the description of jobs in getJobData operation below.

POST /:app/:env
data:
{
   "task": "<string>",
   "wait": "<boolean>",
   "taskVariables": {Object.<string, string>}
}
 

:wait - does request wait for the job's end. Possible values: true/false. Default: false.

:taskVariables - capistrano task options. Optional. These options will be used as -s options of capistrano. Possible values: json object where keys and values are strings or objects that are convertible to string.

HTTP response code 200

{
  "id": "123",
  "url": "http://localhost:8001/web#job/532c3240f8214f0000177376"
}

In case of a blocking execution the job's data will be returned:

{
  "id": 123,
  "url": "http://localhost:8001/web#job/532c3240f8214f0000177376",
  "data": {
    "id": 123,
    "status": "failed",
    "app": "fuboo",
    "env": "production",
    "task": "shell",
    "exitCode": null,
    "stdout": "stdout of pulsar job's process",
    "output": "stdout + stderr of pulsar job's process",
    "pid": 48691
  }
}

Immediately returns all job data including output to date.

GET /job/:id

HTTP response code 200

{
  "id": 123,
  "status": "failed",
  "app": "fuboo",
  "env": "production",
  "task": "shell",
  "exitCode": null,
  "stdout": "stdout of pulsar job's process",
  "output": "stdout + stderr of pulsar job's process",
  "pid": 48691
}

GET /job/:id/kill

HTTP response code 200

Besides rest and web interface the instance of pulsar-rest-api also offers web socket interface. Interface is available on {config.baseUrl}/websocket. Currently the interface doesn't expect any incoming messages and only transmits events about jobs' lifecycle.

Job was created {message: {event: 'job.create', job: {Object}}}

Job was changed {message: {event: 'job.change', job: {Object}}}

To enable authentication of API you need to provide config options authentication. All users of API should remember that besides that you need to authenticate yourself with Github, you also need to be a member of Github organization that was defined in authentication.githubOrg of config.

If you interact with API through the web interface then you will need to follow standard Github Web Application Flow procedure. If everything is ok then you will be able to interact with web interface of API.

If you want access API directly, for example through the curl tool, then you need to provide your Github basic token with every request. If you don't have one then you can get it here https://github.com/settings/tokens/new. After that you can use API like this: curl -u {github-token}:x-oauth-basic -H "Content-Type: application/json" -X POST -d '{"task":"dummy:my_sleep"}' https://api.pulsar.local:8001/example/production

When socket client gets connected it needs to send authentication information as its first message. There are two options available:

  sock.onopen = function() {
    sock.send(JSON.stringify({
      token: 'put your Github token here'
    }));
  };

This option is available only if you are using websocket in pair with web interface. When web interface gets successful authentication it receives cookie userid which you can send instead of Github token.

  sock.onopen = function() {
    sock.send(JSON.stringify({
      cookie: $.cookie('userid')
    }));
  };

If token was wrong, connection would be closed, otherwise it would start to receive messages.