@scramjet/api-client
TypeScript icon, indicating that this package has built-in type declarations

1.0.1 • Public • Published

Scramjet Transform Hub API Client

GitHub license STH version GitHub stars npm Discord Donate

⭐ Star us on GitHub — it motivates us a lot! 🚀

Scramjet Transform Hub Logo

The package provides an API Client for use with Scramjet Transform Hub.

Usage:

import { HostClient, SequenceClient } from "@scramjet/api-client";

const host = new HostClient("http://192.168.0.34:8000/api/v1");

host.getSequence("3e989fe7-a3b7-401e-a269-2fa0a7ac70f4")

Docs

See the code documentation here: scramjetorg/transform-hub/docs/api-client/modules.md

More reading

In the link below you will find more information about our stream protocol and API usage.

See the code documentation here: scramjetorg/transform-hub/docs/read-more/stream-and-api.md

Scramjet Transform Hub

This package is part of Scramjet Transform Hub.

Scramjet Transform Hub is a deployment and execution platform. Once installed on a server, it will allow you to start your programs and keep them running on a remote machine. You will be able to start programs in the background or connect to them and see their output directly on your terminal. You will be able to pipe your local data to the program, as if it was running from your terminal. You can start your server in AWS, Google Cloud or Azure, start it on your local machine, install it on a Raspberry Pi or wherever else you'd like.

Use cases

There's no limit what you can use it for. You want a stock checker? A chat bot? Maybe you'd like to automate your home? Retrieve sensor data? Maybe you have a lot of data and want to transfer and wrangle it? You have a database of cities and you'd like to enrich your data? You do machine learning and you want to train your set while the data is fetched in real time? Hey, you want to use it for something else and ask us if that's a good use? Ask us via email or hop on our Scramjet Discord!

Quick guide


Host operations

[ GET ] /api/v1/version - show the Host version


Parameters

No parameters

Responses

Successful operation code: 200

{ "version" : "0.12.2" }
[ GET ] /api/v1/sequences - show all Sequences saved on the Host


Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [], // a list of all running Instances of this Sequence
    "id": "eea8bc33-440f-4a17-8931-eb22a17d5d56", // Sequence ID
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/hello-alice-out",
      "version": "0.12.2",
      "engines": {
        "node": ">=10"
      },
      "config": {},
      "sequencePath": "index", // a path to file with a main function
      "packageVolumeId": "eea8bc33-440f-4a17-8931-eb22a17d5d56"
    }
  },
  {
    "instances": [
      "02381acf-cb16-4cff-aa9b-f22f04ada94f"
    ],
    "id": "3ec02b93-4ca9-4d23-baab-048dab5ffda4",
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/checksum-sequence",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "3ec02b93-4ca9-4d23-baab-048dab5ffda4"
    }
  }
]
[ GET ] /api/v1/instances - show all Instances running on the Host


Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "02381acf-cb16-4cff-aa9b-f22f04ada94f", // Instance ID
    "sequence": "3ec02b93-4ca9-4d23-baab-048dab5ffda4" // Sequence ID
  },
  {
    "id": "ab0272d8-c9b0-43f7-9e7e-bcac9ec0f21f",
    "sequence": "e4ca555c-ced1-4a13-b531-f43016eaf4ed"
  }
]
[ GET ] /api/v1/load-check - monitor CPU, memory and disk usage metrics on the Host machine


Parameters

No parameters

Responses

Successful operation code: 200

{
  "avgLoad": 0.08,
  "currentLoad": 5.190776257704936,
  "memFree": 4634816512,
  "memUsed": 8050364416,
  "fsSize": [
    {
      "fs": "/dev/sda1",
      "type": "ext4",
      "size": 20838993920,
      "used": 14939455488,
      "available": 5882761216,
      "use": 71.75,
      "mount": "/"
    },
    {
      "fs": "/dev/sda15",
      "type": "vfat",
      "size": 109422592,
      "used": 9621504,
      "available": 99801088,
      "use": 8.79,
      "mount": "/boot/efi"
    }
  ]
}
[ GET ] /api/v1/log - monitor CPU, memory and disk usage metrics on the Host machine


Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:04:47.094Z log (object:Host) Host main called.
2021-11-19T16:04:47.100Z info (object:SocketServer) Server on: /tmp/scramjet-socket-server-path
2021-11-19T16:04:47.104Z info (object:Host) API listening on: 127.0.0.1:8000
2021-11-19T16:05:08.228Z info (object:Host) New sequence incoming...
2021-11-19T16:05:08.229Z log (object:LifecycleDockerAdapterSequence) Docker sequence adapter init.
2021-11-19T16:05:08.229Z log (object:DockerodeDockerHelper) Checking image scramjetorg/pre-runner:0.12.2
2021-11-19T16:05:12.234Z info (object:LifecycleDockerAdapterSequence) Docker sequence adapter done.
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Volume created. Id:  c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Starting PreRunner { image: 'scramjetorg/pre-runner:0.12.2', maxMem: 128 }
2021-11-19T16:05:13.536Z log (object:DockerodeDockerHelper) Checking image scramjetorg/runner:0.12.2
2021-11-19T16:05:16.670Z info (object:SequenceStore) New sequence added: c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:16.672Z info (object:Host) Sequence identified: {
  container: {
    image: 'scramjetorg/runner:0.12.2',
    maxMem: 512,
    exposePortsRange: [ 30000, 32767 ],
    hostIp: '0.0.0.0'
  },
  name: '@scramjet/multi-outputs',
  version: '0.12.2',
  engines: {},
  config: {},
  sequencePath: 'index.js',
  packageVolumeId: 'c50fe4d3-89cc-4685-a82a-16cbc744733d'
}
2021-11-19T16:05:16.691Z debug (object:Host) Request date: 2021-11-19T16:05:08.239Z, method: POST, url: /api/v1/sequence, status: 202

Sequence operations

[ POST ] /api/v1/sequence - add new sequence


Parameters

Name Type Description Required
file binary compressed package in tar.gz format yes
appConfig json additional package.json config file no

Responses

Accepted operation code: 202

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0" // sequence id
}
[ GET ] /api/v1/sequences - show list of sequences


Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [
      "742d2713-7ab6-4cde-82f3-a7beabdd4e98"       // list of sequence instances
    ],
    "id": "bdef63db-d3a0-45c8-85db-e94ebb96097f",  // sequence id
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512
      },
      "name": "@scramjet/transform-hub",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
    }
  }
]
[ POST ] /api/v1/sequence/:id/start - start chosen sequence


Parameters

Name Type Description Required
appConfig json additional package.json config file no
args json additional arguments that instance should starts with no

Responses

Successful operation code: 200

{
  "id": "681c856e-dfa4-46a1-951d-47b27345552e"
}
[ DELETE ] /api/v1/sequence/:id - delete a sequence by id


Parameters

No parameters

Responses

Successful operation code: 200

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0"
}

Conflict operation code: 409 - the instance is still running

{
  "error": "Can't remove sequence in use."
}

Instance basic operations

[ GET ] /api/v1/instances - list all instances


Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "742d2713-7ab6-4cde-82f3-a7beabdd4e98",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "681c856e-dfa4-46a1-951d-47b27345552e",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "21f787ed-6b9e-4e9f-828e-afe428d84833",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  }
]
[ GET ] /api/v1/instance/:id - show data of chosen instance


Parameters

No parameters

Responses

Name Code Description
Successful operation 200 Returns JSON data
Not Found operation 404 For example if instance was already stopped
{
  "created": "2021-10-29T16:08:36.524Z",
  "started": "2021-10-29T16:08:38.701Z",
  "sequenceId": "b0c02fdc-b05f-4f26-9d68-43a702eb7b44"
}
[ POST ] /api/v1/instance/:id/_stop - end instance gracefully and prolong operations or not for task completion​


Parameters

Name Type Description Required
timeout number The number of milliseconds before the Instance will be killed. Default: 7000ms. no
canCallKeepalive boolean If set to true, the instance will prolong the running. Default: false. no

Responses

Successful operation code: 200

{
   "code": 0,
   "type": "string",
   "message": "string"
}
[ POST ] api/v1/instance/:id/_kill - end instance gracefully waiting for unfinished tasks


Parameters

No parameters

Responses

Accepted operation code: 202

No body returned
[ GET ] /api/v1/instance/:id/health - check status about instance health


Parameters

No parameters

Responses

Successful operation code: 200

{
  "cpuTotalUsage": 529325247,
  "healthy": true,
  "limit": 536870912,
  "memoryMaxUsage": 16117760,
  "memoryUsage": 14155776,
  "networkRx": 1086,
  "networkTx": 0,
  "containerId": "1c993c4ff774fac06185aa9554cf40c23b03e1479a7e0d14827708161b08ae51"
}

Instance advanced operation

[ POST ] /api/v1/instance/:id/_event - send event to the Instance


Parameters

Name Type Description Required
eventName string Name of an event true
message string JSON formatted event payload false

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/event - get the data stream with the events from the Instance


Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/once - get the last event sent by the Instance


Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/input - send data to the input stream of the Instance to consume it


Parameters

No parameters

Responses

Name Code Description
Successful operation 200 -
Not Acceptable operation 406 Instance expects the input to be provided from the Topic API
[ GET ] /api/v1/instance/:id/output - get stream data from an instance and consume it through the endpoint


Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/inout - Experimental. Send data to the input stream of the Instance to consume it. Get stream data from instance.


Parameters

No parameters

Responses

Name Code Description
Successful operation 200 -
[ POST ] /api/v1/instance/:id/stdin​ - process.stdin


Parameters

No parameters

Responses

Successful operation code: 200

[ GET ] /api/v1/instance/:id/stdout - process.stdout


Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/stderr - process.stderr


Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/log - stream all instance logs


Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:12:22.948Z log (Sequence) 42
2021-11-19T16:12:23.949Z log (Sequence) 41
2021-11-19T16:12:24.950Z log (Sequence) 40
2021-11-19T16:12:25.951Z log (Sequence) 39
2021-11-19T16:12:26.952Z log (Sequence) 38
2021-11-19T16:12:27.952Z log (Sequence) 37
2021-11-19T16:12:28.953Z log (Sequence) 36
2021-11-19T16:12:29.953Z log (Sequence) 35

Service Discovery: Topics

If a given topic does not exist, Transform-Hub creates it and stores the sent data in the newly created topic. The data is stored in the topic until the data is not consumed (either by the Topic API or by the Instances subscribing to this topic). When the data are sent to the topic they are written to the returned stream.

[ POST ] /api/v1/topics/:name​ - sends data to the topic


Parameters

No parameters

Request Headers

Header Type Description Default Required
x-end-stream boolean If set to true, then close topic stream after processing the request. false no
content-type text, application Specifies data type of this topic application/x-ndjson no

Supported types: text/x-ndjson, application/x-ndjson, application/x-ndjson, text/plain, application/octet-stream

Responses

Name Code Description
Successful operation 200 data was sent with the header indicating the end of data
Successful operation 202 data was sent without the header indicating the end of data (default)
[ GET ] /api/v1/topics/:name​ - get data from the topic


Parameters

No parameters

Responses

Topic data stream.

Successful operation code: 200

{
  "source": "Twitter",
  "id": "850006245121695778",
  "content": "Natural wetlands make up ~30% of global total CH4 emissions",
  "user": {
    "id": 1234994945,
    "name": "Climate Change Conference",
    "screen_name": "Climate Change",
  }
}

Some important links

License and contributions

This module is licensed under AGPL-3.0 license.

The Scramjet Transform Hub project is dual-licensed under the AGPL-3.0 and MIT licenses. Parts of the project that are linked with your programs are MIT licensed, the rest is AGPL.

Contributions

We accept valid contributions and we will be publishing a more specific project roadmap so contributors can propose features and also help us implement them. We kindly ask you that contributed commits are Signed-Off git commit --sign-off.

We provide support for contributors via test cases. If you expect a certain type of workflow to be officially supported, please specify and implement a test case in Gherkin format in bdd directory and include it in your pull request. More info about our BDD test you will find here.

Help wanted 👩‍🎓🧑👱‍♀️

The project need's your help! There's lots of work to do and we have a lot of plans. If you want to help and be part of the Scramjet team, please reach out to us, on discord or email us: opensource@scramjet.org.

Donation 💸

Do you like this project? It helped you to reduce time spent on delivering your solution? You are welcome to buy us a coffee ☕ Thanks a lot! 😉

You can sponsor us on github

  • There's also a Paypal donation link if you prefer that:

paypal

Readme

Keywords

none

Package Sidebar

Install

npm i @scramjet/api-client

Weekly Downloads

11

Version

1.0.1

License

MIT

Unpacked Size

160 kB

Total Files

19

Last publish

Collaborators

  • patu
  • scramjetorg
  • michalcz
  • stoq