Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »

    repl_queuepublic

    repl_queue

    The idea of repl is use local queue to preserve some request needed to invoke remote api and those request compare to synchronize request, which require response immediately. Async request could response as message accepted, then process those requests later. It also very useful in batch tasks. Also if remote is out of service or network is blocked, then those requests will hold inside local queue, after remote host is up and ready to serve, then the request will rely to remote by correct order(fifo). It is similar concept as email queue.

    Version 0.0.1

    support binary format
    revamp fifo kill 
    

    source directory

    lib

    basic libraries
    

    bin

    some commandline tools(see README in bin/README.md)
    

    test

    some testing data
    

    test/test_writer

     txlogw benchmark
    

    Log writer

    repl queue is consist by reader and writer.
    The goal of writer is very simple. 
    receive http post request then insert into sqlite.
    Because it's so simple, It could accept 1000req/sec. (commit 1's per second)
    

    Log reader

    Node.js could watch file change event by fs.watch()
    If writer sync data into db file, then it will trigger
    reader start to consume message. 
    
    standard consumer 
        rely consumer over http
            to remote socketio
            to local sqlite db
    

    Config example in tests/example/config.js

    config = {
        type: 'fifo',
        path: '../test/example',
        max_volumes: 5,
        server: {
            listen: 9090,
            socketio_handler_enable: true,
            rest_handler_enable: true 
        },
        reader:{
            '1':{ // index
                consumer_function: [rely_rest, 'http://localhost:9091/repl/example'],
                filter : filter_module.filter('ID', filter_module.mod_rule(2 ,0))
            },
            '2':{// index
                consumer_function: [rely_rest, 'http://localhost:9091/repl/example'],
                filter : filter_module.filter('ID', filter_module.mod_rule(2 ,1))
            },
            '3':{// index
                consumer_function: [ExampleReader, '']
            },
            '4':{
                consumer_function: [dist, '../test/test_dist'],
            }
        },
        writer: {
            acl: ip_acl('127.0.0.1')
        }
    };
    

    sqlite schema volume_N.db

    Why I choise sqlite to implement queue. It just because it's most easy 
    to implement. But still possible to implement by other key value storage.
    
    create table if not exists QUEUE_VOLUME (
        ID INTEGER PRIMARY KEY ASC,
        CMD TEXT,
        DATA TEXT,
        CREATED INTEGER
    )
    

    sqlite schema metta.db

    create table if not exists QUEUE_META (
        ID text PRIMARY KEY,
        VOLUME int,
        LAST_RECORD int
    )
    

    Keywords

    none

    install

    npm i repl_queue

    Downloadsweekly downloads

    4

    version

    0.0.1

    license

    none

    repository

    githubgithub

    last publish

    collaborators

    • avatar