official SDK for the schnedale-broke
Via NPM:
npm install schnedale
Import it:
const client = require('schnedale');
- Clients can subscribe and push data to pipelines.
- Pipeline naming convention:
pipeline:tag
- the tag represents the stage of the pipeline. For pipelines having only one stage,:raw
can be used. E.g. an article pipeline has different stages:created, taged, analyzed, scored
, which is why you might use the pipeline namesarticles:created, articles:tagged, articles:analyzed, articles:scored
. The pipeline prefix isarticles
in this case.
Create a connection to the schnedale-broker by defining the broker's IP address and port number. Once a connection is established, the callbnack is triggered.
- ip: IPv4 address of schnedale-broker
-
port: usually
2227
- callback: triggered when connection is established
const IP = '127.0.0.1';
const PORT = 2227; // default port
client.connect(IP, PORT, () => {
console.log('connected to schnedale-broker');
})
Subscribe to a pipeline by defining its name and the callback function receiving data published to pipeline.
- pipeline: the pipeline's name (usually pipeline:tag, e.g. articles:raw)
- callback: function receiving data which is published to pipeline
// once client is connected
const pipeline = 'articles:transformed';
client.subscribe(pipeline, (data) => {
console.log(data);
})
Push data to a pipeline.
Note: Published messages are acknowledged by the server. However, there is no SDK support for that feature.
- pipeline: the pipeline's name (usually pipeline:tag, e.g. articles:raw)
- data: object, string or number
// once client is connected
const pipeline = 'articles:raw';
const payload = {
hello: 'world'
}
client.publish(pipeline, payload)
Transformers read data from one pipeline, do something with it and publish the transformed data to another pipeline.
Note: The pipeline's prefix should be the same, however, the tags should be different
- ingressPipeline: name of the pipeline where the data is coming in
- egressPipeline_ name of the pipeline where the transformed data should be published to
- transformerFunction: function receiving data from the ingressPipeline and returning the transformed data
// once client is connected
// articles:raw -> transformation -> articles:tagged
const ingressPipeline = 'articles:raw';
const egressPipeline = 'articles:tagged';
client.transform(ingressPipeline, egressPipeline, (data) => {
data.tags = ['tag1', 'tag2'];
return data;
})