@dcos/recordio

    0.1.8 • Public • Published

    RecordIO Build Status


    👩‍🔬 Please be aware that this package is still experimental — changes to the interface and underlying implementation are likely, and future development or maintenance is not guaranteed.


    This package provides a function to read records in the RecordIO format from the input string.

    Usage

    import { read } from "@dcos/recordio";
     
    const [records, rest] = read(input);

    Function read returns a tuple whose first element is an array of records and the second element is the rest part of the input that is either empty or contains partial records.

    RecordIO format

    Prepends to a single record its length in bytes, followed by a newline and then the data:

    The BNF grammar for a RecordIO-encoded streaming response is:

    records         = *record
    
    record          = record-size LF record-data
    
    record-size     = 1*DIGIT
    record-data     = record-size(OCTET)
    

    record-size should be interpreted as an unsigned 64-bit integer (uint64).

    For example, a stream may look like:

    121\n
    {"type": "SUBSCRIBED","subscribed": {"framework_id": {"value":"12220-3440-12532-2345"},"heartbeat_interval_seconds":15.0}20\n
    {"type":"HEARTBEAT"}675\n
    ...
    

    Further documentation can be found in the Apache Mesos documentation.

    Testing

    The implementation is tested with different UTF-8 character sets to verify that it reads the correct number of bytes from the input string.

    You can use the following python snippet to create test records from the provided messages array.

    messages = [u"foo", u"bar"]
     
    for message in messages:
        chars = len(message)
        size = len(message.encode("utf-8"))
        print("Message")
        print("> Chars: %s" % chars)
        print("> Size: %s" % size)
        print("> Message: %s" % message)
        print("> Record: %s\\n%s" % (sizemessage))

    Some of the tests use character sets from the UTF-8 decoder capability and stress test developed by Markus Kuhn to ensure that parsing of character with different byte sequences works properly.

    Keywords

    none

    Install

    npm i @dcos/recordio

    DownloadsWeekly Downloads

    3

    Version

    0.1.8

    License

    Apache-2.0

    Unpacked Size

    949 kB

    Total Files

    449

    Last publish

    Collaborators

    • brandonc
    • tattdcodemonkey
    • tnguyen0
    • natmegs
    • mperrotti
    • dschmidt
    • mesosphere-frontend-ci
    • mesosphere-it
    • orlandohohmeier
    • specone
    • nlight
    • juliangieseke