Scramjet Transform Hub API Client

⭐ Star us on GitHub — it motivates us a lot! 🚀

The package provides an API Client for use with Scramjet Transform Hub.

Usage:

import { HostClient, SequenceClient } from "@scramjet/api-client";

const host = new HostClient("http://192.168.0.34:8000/api/v1");

host.getSequence("3e989fe7-a3b7-401e-a269-2fa0a7ac70f4")

Docs

See the code documentation here: scramjetorg/transform-hub/docs/api-client/modules.md

Scramjet Transform Hub

This package is part of Scramjet Transform Hub.

Scramjet Transform Hub is a deployment and execution platform. Once installed on a server, it will allow you to start your programs and keep them running on a remote machine. You will be able to start programs in the background or connect to them and see their output directly on your terminal. You will be able to pipe your local data to the program, as if it was running from your terminal. You can start your server in AWS, Google Cloud or Azure, start it on your local machine, install it on a Raspberry Pi or wherever else you'd like.

Use cases

There's no limit what you can use it for. You want a stock checker? A chat bot? Maybe you'd like to automate your home? Retrieve sensor data? Maybe you have a lot of data and want to transfer and wrangle it? You have a database of cities and you'd like to enrich your data? You do machine learning and you want to train your set while the data is fetched in real time? Hey, you want to use it for something else and ask us if that's a good use? Ask us via email or hop on our Scramjet Discord!

Quick guide

Host operations

[ GET ] /api/v1/version - show the Host version

Parameters

No parameters

Responses

Successful operation code: 200

{ "version" : "0.12.2" }

[ GET ] /api/v1/sequences - show all Sequences saved on the Host

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [], // a list of all running Instances of this Sequence
    "id": "eea8bc33-440f-4a17-8931-eb22a17d5d56", // Sequence ID
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/hello-alice-out",
      "version": "0.12.2",
      "engines": {
        "node": ">=10"
      },
      "config": {},
      "sequencePath": "index", // a path to file with a main function
      "packageVolumeId": "eea8bc33-440f-4a17-8931-eb22a17d5d56"
    }
  },
  {
    "instances": [
      "02381acf-cb16-4cff-aa9b-f22f04ada94f"
    ],
    "id": "3ec02b93-4ca9-4d23-baab-048dab5ffda4",
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/checksum-sequence",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "3ec02b93-4ca9-4d23-baab-048dab5ffda4"
    }
  }
]

[ GET ] /api/v1/instances - show all Instances running on the Host

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "02381acf-cb16-4cff-aa9b-f22f04ada94f", // Instance ID
    "sequence": "3ec02b93-4ca9-4d23-baab-048dab5ffda4" // Sequence ID
  },
  {
    "id": "ab0272d8-c9b0-43f7-9e7e-bcac9ec0f21f",
    "sequence": "e4ca555c-ced1-4a13-b531-f43016eaf4ed"
  }
]

[ GET ] /api/v1/load-check - monitor CPU, memory and disk usage metrics on the Host machine

Parameters

No parameters

Responses

Successful operation code: 200

{
  "avgLoad": 0.08,
  "currentLoad": 5.190776257704936,
  "memFree": 4634816512,
  "memUsed": 8050364416,
  "fsSize": [
    {
      "fs": "/dev/sda1",
      "type": "ext4",
      "size": 20838993920,
      "used": 14939455488,
      "available": 5882761216,
      "use": 71.75,
      "mount": "/"
    },
    {
      "fs": "/dev/sda15",
      "type": "vfat",
      "size": 109422592,
      "used": 9621504,
      "available": 99801088,
      "use": 8.79,
      "mount": "/boot/efi"
    }
  ]
}

[ GET ] /api/v1/log - monitor CPU, memory and disk usage metrics on the Host machine

Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:04:47.094Z log (object:Host) Host main called.
2021-11-19T16:04:47.100Z info (object:SocketServer) Server on: /tmp/scramjet-socket-server-path
2021-11-19T16:04:47.104Z info (object:Host) API listening on: 127.0.0.1:8000
2021-11-19T16:05:08.228Z info (object:Host) New sequence incoming...
2021-11-19T16:05:08.229Z log (object:LifecycleDockerAdapterSequence) Docker sequence adapter init.
2021-11-19T16:05:08.229Z log (object:DockerodeDockerHelper) Checking image scramjetorg/pre-runner:0.12.2
2021-11-19T16:05:12.234Z info (object:LifecycleDockerAdapterSequence) Docker sequence adapter done.
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Volume created. Id:  c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Starting PreRunner { image: 'scramjetorg/pre-runner:0.12.2', maxMem: 128 }
2021-11-19T16:05:13.536Z log (object:DockerodeDockerHelper) Checking image scramjetorg/runner:0.12.2
2021-11-19T16:05:16.670Z info (object:SequenceStore) New sequence added: c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:16.672Z info (object:Host) Sequence identified: {
  container: {
    image: 'scramjetorg/runner:0.12.2',
    maxMem: 512,
    exposePortsRange: [ 30000, 32767 ],
    hostIp: '0.0.0.0'
  },
  name: '@scramjet/multi-outputs',
  version: '0.12.2',
  engines: {},
  config: {},
  sequencePath: 'index.js',
  packageVolumeId: 'c50fe4d3-89cc-4685-a82a-16cbc744733d'
}
2021-11-19T16:05:16.691Z debug (object:Host) Request date: 2021-11-19T16:05:08.239Z, method: POST, url: /api/v1/sequence, status: 202

Sequence operations

[ POST ] /api/v1/sequence - add new sequence

Parameters

Name	Type	Description	Required
`file`	`binary`	compressed package in tar.gz format	yes
`appConfig`	`json`	additional package.json config file	no

Responses

Accepted operation code: 202

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0" // sequence id
}

[ GET ] /api/v1/sequences - show list of sequences

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [
      "742d2713-7ab6-4cde-82f3-a7beabdd4e98"       // list of sequence instances
    ],
    "id": "bdef63db-d3a0-45c8-85db-e94ebb96097f",  // sequence id
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512
      },
      "name": "@scramjet/transform-hub",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
    }
  }
]

[ POST ] /api/v1/sequence/:id/start - start chosen sequence

Parameters

Name	Type	Description	Required
`appConfig`	`json`	additional package.json config file	no
`args`	`json`	additional arguments that instance should starts with	no

Responses

Successful operation code: 200

{
  "id": "681c856e-dfa4-46a1-951d-47b27345552e"
}

[ DELETE ] /api/v1/sequence/:id - delete a sequence by id

Parameters

No parameters

Responses

Successful operation code: 200

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0"
}

Conflict operation code: 409 - the instance is still running

{
  "error": "Can't remove sequence in use."
}

Instance basic operations

[ GET ] /api/v1/instances - list all instances

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "742d2713-7ab6-4cde-82f3-a7beabdd4e98",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "681c856e-dfa4-46a1-951d-47b27345552e",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "21f787ed-6b9e-4e9f-828e-afe428d84833",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  }
]

[ GET ] /api/v1/instance/:id - show data of chosen instance

Parameters

No parameters

Responses

Name	Code	Description
Successful operation	`200`	Returns JSON data
Not Found operation	`404`	For example if instance was already stopped

{
  "created": "2021-10-29T16:08:36.524Z",
  "started": "2021-10-29T16:08:38.701Z",
  "sequenceId": "b0c02fdc-b05f-4f26-9d68-43a702eb7b44"
}

[ POST ] /api/v1/instance/:id/_stop - end instance gracefully and prolong operations or not for task completion

Parameters

Name	Type	Description	Required
`timeout`	`number`	The number of milliseconds before the Instance will be killed. Default: 7000ms.	no
`canCallKeepalive`	`boolean`	If set to true, the instance will prolong the running. Default: false.	no

Responses

Successful operation code: 200

{
   "code": 0,
   "type": "string",
   "message": "string"
}

[ POST ] api/v1/instance/:id/_kill - end instance gracefully waiting for unfinished tasks

Parameters

No parameters

Responses

Accepted operation code: 202

No body returned

[ GET ] /api/v1/instance/:id/health - check status about instance health

Parameters

No parameters

Responses

Successful operation code: 200

{
  "cpuTotalUsage": 529325247,
  "healthy": true,
  "limit": 536870912,
  "memoryMaxUsage": 16117760,
  "memoryUsage": 14155776,
  "networkRx": 1086,
  "networkTx": 0,
  "containerId": "1c993c4ff774fac06185aa9554cf40c23b03e1479a7e0d14827708161b08ae51"
}

Instance advanced operation

[ POST ] /api/v1/instance/:id/_event - send event to the Instance

Parameters

Name	Type	Description	Required
`eventName`	`string`	Name of an event	true
`message`	`string`	JSON formatted event payload	false

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/event - get the data stream with the events from the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/once - get the last event sent by the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/input - send data to the input stream of the Instance to consume it

Parameters

No parameters

Responses

Name	Code	Description
Successful operation	`200`	-
Not Acceptable operation	`406`	Instance expects the input to be provided from the Topic API

[ GET ] /api/v1/instance/:id/output - get stream data from an instance and consume it through the endpoint

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/inout - Experimental. Send data to the input stream of the Instance to consume it. Get stream data from instance.

Parameters

No parameters

Responses

Name	Code	Description
Successful operation	`200`	-

[ POST ] /api/v1/instance/:id/stdin - process.stdin

Parameters

No parameters

Responses

Successful operation code: 200

[ GET ] /api/v1/instance/:id/stdout - process.stdout

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/stderr - process.stderr

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/log - stream all instance logs

Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:12:22.948Z log (Sequence) 42
2021-11-19T16:12:23.949Z log (Sequence) 41
2021-11-19T16:12:24.950Z log (Sequence) 40
2021-11-19T16:12:25.951Z log (Sequence) 39
2021-11-19T16:12:26.952Z log (Sequence) 38
2021-11-19T16:12:27.952Z log (Sequence) 37
2021-11-19T16:12:28.953Z log (Sequence) 36
2021-11-19T16:12:29.953Z log (Sequence) 35

Service Discovery: Topics

If a given topic does not exist, Transform-Hub creates it and stores the sent data in the newly created topic. The data is stored in the topic until the data is not consumed (either by the Topic API or by the Instances subscribing to this topic). When the data are sent to the topic they are written to the returned stream.

[ POST ] /api/v1/topics/:name - sends data to the topic

Parameters

No parameters

Request Headers

Header	Type	Description	Default	Required
`x-end-stream`	`boolean`	If set to `true`, then close topic stream after processing the request.	false	no
`content-type`	`text`, `application`	Specifies data type of this topic	`application/x-ndjson`	no

Supported types: text/x-ndjson, application/x-ndjson, application/x-ndjson, text/plain, application/octet-stream

Responses

Name	Code	Description
Successful operation	`200`	data was sent with the header indicating the end of data
Successful operation	`202`	data was sent without the header indicating the end of data (default)

[ GET ] /api/v1/topics/:name - get data from the topic

Parameters

No parameters

Responses

Topic data stream.

Successful operation code: 200

{
  "source": "Twitter",
  "id": "850006245121695778",
  "content": "Natural wetlands make up ~30% of global total CH4 emissions",
  "user": {
    "id": 1234994945,
    "name": "Climate Change Conference",
    "screen_name": "Climate Change",
  }
}

Some important links

Scramjet, the company behind Transform Hub
The Scramjet Framework - functional reactive stream processing framework
The Transform Hub repo on github
You can see the Scramjet Transform Hub API docs here
You can see the CLI documentation here, but si help should also be quite effective.
Don't forget to ⭐ this repo if you like it, subscribe to releases and keep visiting us for new versions and updates.
You can open an issue - file a bug report or a feature request here

License and contributions

This module is licensed under AGPL-3.0 license.

The Scramjet Transform Hub project is dual-licensed under the AGPL-3.0 and MIT licenses. Parts of the project that are linked with your programs are MIT licensed, the rest is AGPL.

Contributions

We accept valid contributions and we will be publishing a more specific project roadmap so contributors can propose features and also help us implement them. We kindly ask you that contributed commits are Signed-Off git commit --sign-off.

We provide support for contributors via test cases. If you expect a certain type of workflow to be officially supported, please specify and implement a test case in Gherkin format in bdd directory and include it in your pull request. More info about our BDD test you will find here.

Help wanted 👩‍🎓🧑👱‍♀️

The project need's your help! There's lots of work to do and we have a lot of plans. If you want to help and be part of the Scramjet team, please reach out to us, on discord or email us: opensource@scramjet.org.

Donation 💸

Do you like this project? It helped you to reduce time spent on delivering your solution? You are welcome to buy us a coffee ☕ Thanks a lot! 😉

You can sponsor us on github

There's also a Paypal donation link if you prefer that:

@scramjet/api-client

Scramjet Transform Hub API Client

Docs

More reading

Scramjet Transform Hub

Use cases

Quick guide

Host operations

Sequence operations

Instance basic operations

Instance advanced operation

Service Discovery: Topics

Some important links

License and contributions

Contributions

Help wanted 👩‍🎓🧑👱‍♀️

Donation 💸

Readme

Keywords

Package Sidebar

Install

Repository

Homepage

Weekly Downloads

Version

License

Unpacked Size

Total Files

Last publish

Collaborators

@scramjet/api-client

Scramjet Transform Hub API Client

Docs

More reading

Scramjet Transform Hub

Use cases

Quick guide

Host operations

Sequence operations

Instance basic operations

Instance advanced operation

Service Discovery: Topics

Some important links

License and contributions

Contributions

Help wanted 👩‍🎓🧑👱‍♀️

Donation 💸

Readme

Keywords

Package Sidebar

Install

Repository

Homepage

DownloadsWeekly Downloads

Version

License

Unpacked Size

Total Files

Last publish

Collaborators

Weekly Downloads