node package manager

cloudd

node based job execution engine

build status # cloudd
node.js cloud engine Apart from all the buzz words, key aspect of this software is to specify set of tasks that are interdependent in a config file (similar to DAG input on condor) and run them.

Basics

  • Task - is the smallest unit of execution (preferrable should be an idempotent task)
  • Job - collection of tasks with dependencies (acyclic)

Features

  • job execution engine
  • Task could be a javascript unit or executable that would be exec'ed
  • Command line to check the status of jobs, tasks and current running processes
  • Command line Submit of jobs
  • instantiate jobs cron style
  • Good unit test coverage

Upcoming

  • Failure of tasks to be handled
  • Job load balancing among cloudd hosts
  • Persistence of state
  • clustering among cloudd hosts

Example

  • Lets see how to define a Hello My World Job
    1. Job Hello My World comprise of three tasks
    • Hello Task
    • My Task
    • World Task
    1. Dependencies
    • My would be run only after Hello
    • World would be run only after My
  • Sample yaml config file would look like (see in examples/hellomyworld.yaml)
name: Hello World
description:
  Sample 'hello my world' job
jobs:
  my:
    executable: echo "my"
  hello:
    executable: echo "hello"
  world:
    executable: echo "world"
dependencies:
  my-world:
     parent: [my]
     child: [world]
  hello-my:
     parent: [hello]
     child: [my]

To try this example

Have two terminals, one for the server and another one for submitting the job

# terminal 1
cloudd server

# terminal 2
cloudd submit HW examples/hellomyworld.yaml

Usage

To run the server

cloudd server

Command line utilities

# to show current running tasks
cloudd ps

# to show total tasks that are ready to run (ordered list)
cloudd tasks

# to show total jobs in the system (non-completed)
cloudd jobs

# to submit a new job
cloudd submit job-id sample-job-file.yaml

# to submit a cron job (say every 5 seconds)
cloudd cron "*/5 * * * * *" job-id sample-job-file.yaml

Config file

  • id's for the job and task cannot have '#'
  • input is a field that would be passed to task