node-red-biglib

    1.3.4 • Public • Published

    node-red-biglib

    "Big Lib" is the core library for "Big Nodes"

    Installation

    npm install node-red-biglib

    Principles for Big Nodes

    See biglib for details on Big Nodes. Big Lib and subsequent Big Nodes are a family of nodes built for my own purpose. They are all designed to help me build a complete process for production purposes. For that I needed nodes able to:

    • Flow big volume of data (memory control, work with buffers)
    • Work with a flow of blocks (buffers) (multiple payload within a single job)
    • Tell what they are doing with extended use of statuses (color/message)
    • Use their second output for flow control (start/stop/running/status)
    • Reuse messages in order to propagate _msgid, topic
    • Depends on state of the art libraries for parsing (csv, xml, xlsxs, line, ...)
    • Acts as filters by default (1 payload = 1 action) or data generators (block flow)

    All functionnalities are built under a library named biglib and all Big Nodes rely on it

    The Big Nodes family

    This library is used for big nodes developpement.

    Here is the family

    alt tag

    Dependencies

    byline simple line-by-line stream reader

    filesize provides an easy way to get a human readable file size String

    moment Parse, validate, manipulate, and display dates in JavaScript

    API

    Work in progress. Please look at the following Big Nodes:

    • big file for reading files in multiple ways/format/encoding
    • big csv for parsing CSV
    • big line for parsing by lines
    • big fixed for fixed format parsing
    • big status for visual statuses
    • big exec for executing commands
    • big ssh for executing remote command over SSH
    • big mongo fork from node-red-contrib-mongodb2 for executing MongoDB requests with cursor support
    • big db2 in progress
    • big xlsx for parsing XLSX files
    • big splitter fork from node-red-contrib-splitter with big control support
    • big tail used for converting multiple payloads in one message with line and size constraints

    Sample use of Big Nodes

    Workflow of actions with statuses and last messages capture

    It uses several nodes which are needed for this to work:

    npm install node-red-contrib-bigline
    npm install node-red-contrib-bigtail
    npm install node-red-contrib-bigexec
    npm install node-red-contrib-bigsplitter
    npm install node-red-contrib-bigstatus
    npm install node-red-contrib-traffic
    [{"id":"c5ca4d63.3a35b","type":"bigtail","z":"894b4f7b.76b4b","name":"","size":"2","add_cr":true,"size_kbyte":"","x":620,"y":60,"wires":[["449b97f1.bb6468"],[]]},{"id":"dd148b1f.22eb78","type":"bigline","z":"894b4f7b.76b4b","name":"","filename":"","format":"utf8","keepEmptyLines":false,"x":480,"y":60,"wires":[["c5ca4d63.3a35b"],["c5ca4d63.3a35b"]]},{"id":"99748aff.668b78","type":"debug","z":"894b4f7b.76b4b","name":"tail stdout","active":true,"console":"false","complete":"payload","x":1000,"y":80,"wires":[]},{"id":"449b97f1.bb6468","type":"traffic","z":"894b4f7b.76b4b","name":"","property_allow":"control.ok","filter_allow":"true","ignore_case_allow":false,"negate_allow":false,"send_allow":false,"property_stop":"control.state","filter_stop":"start","ignore_case_stop":false,"negate_stop":false,"send_stop":false,"default_start":false,"differ":false,"x":850,"y":80,"wires":[["99748aff.668b78"]]},{"id":"9323a207.6cdc6","type":"bigline","z":"894b4f7b.76b4b","name":"","filename":"","format":"utf8","keepEmptyLines":false,"x":480,"y":220,"wires":[["d991b04e.266e5"],["d991b04e.266e5"]]},{"id":"d991b04e.266e5","type":"bigtail","z":"894b4f7b.76b4b","name":"","size":"3","add_cr":true,"x":620,"y":220,"wires":[["4ac88b64.b53774"],[]]},{"id":"4ac88b64.b53774","type":"traffic","z":"894b4f7b.76b4b","name":"","property_allow":"control.ok","filter_allow":"true","ignore_case_allow":false,"negate_allow":true,"send_allow":false,"property_stop":"control.state","filter_stop":"start","ignore_case_stop":false,"negate_stop":false,"send_stop":false,"default_start":false,"differ":false,"x":850,"y":200,"wires":[["83cde6d9.7c3218"]]},{"id":"83cde6d9.7c3218","type":"debug","z":"894b4f7b.76b4b","name":"tail stderr","active":true,"console":"false","complete":"payload","x":1000,"y":200,"wires":[]},{"id":"109cae54.ef6352","type":"inject","z":"894b4f7b.76b4b","name":"no file","topic":"","payload":"non/existent","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":160,"wires":[["5e919a71.a16e64"]]},{"id":"dbc04ed6.243fb","type":"bigstatus","z":"894b4f7b.76b4b","name":"","locale":"","x":850,"y":140,"wires":[["cfd54729.302ab8"]]},{"id":"5e919a71.a16e64","type":"bigexec","z":"894b4f7b.76b4b","name":"cat","command":"cat","commandArgs":"","minError":1,"minWarning":1,"cwd":"","shell":"","extraArgumentProperty":"","envProperty":"","format":"utf8","limiter":true,"payloadIs":"argumentNoStdin","x":290,"y":140,"wires":[["dd148b1f.22eb78"],["dd148b1f.22eb78","449b97f1.bb6468","9323a207.6cdc6","4ac88b64.b53774","dbc04ed6.243fb"],["9323a207.6cdc6"]]},{"id":"cfd54729.302ab8","type":"filter","z":"894b4f7b.76b4b","name":"ok?","property":"control.ok","filter":"true","ignorecase":true,"x":990,"y":140,"wires":[["d05541bf.2faac"],["fbb38f7b.044c7"],[]]},{"id":"d05541bf.2faac","type":"debug","z":"894b4f7b.76b4b","name":"next in the flow","active":true,"console":"false","complete":"payload","x":1160,"y":120,"wires":[]},{"id":"fbb38f7b.044c7","type":"debug","z":"894b4f7b.76b4b","name":"KO, other action","active":true,"console":"false","complete":"payload","x":1160,"y":160,"wires":[]},{"id":"6d4a788a.92b588","type":"comment","z":"894b4f7b.76b4b","name":"Change to suit your conf","info":"","x":170,"y":100,"wires":[]},{"id":"321868ab.cde798","type":"comment","z":"894b4f7b.76b4b","name":"Sample use of big nodes","info":"","x":130,"y":40,"wires":[]},{"id":"34616222.cb9e9e","type":"inject","z":"894b4f7b.76b4b","name":"test file","topic":"","payload":"test.csv","payloadType":"str","repeat":"","crontab":"","once":false,"x":130,"y":120,"wires":[["5e919a71.a16e64"]]}]

    alt tag

    Author

    • Jacques W

    License

    This code is Open Source under an Apache 2 License.

    You may not use this code except in compliance with the License. You may obtain an original copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0
    

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. Please see the License for the specific language governing permissions and limitations under the License.

    Feedback and Support

    Please report any issues or suggestions via the Github Issues list for this repository.

    For more information, feedback, or community support see the Node-Red Google groups forum at https://groups.google.com/forum/#!forum/node-red

    Keywords

    Install

    npm i node-red-biglib

    DownloadsWeekly Downloads

    665

    Version

    1.3.4

    License

    Apache-2.0

    Unpacked Size

    44 kB

    Total Files

    4

    Last publish

    Collaborators

    • jacques44