Nutritious Polygonal Meatball

    neo4j-batch-index-stream

    0.0.1 • Public • Published

    Neo4j batch index stream

    A writable stream for batch indexing labeled nodes and relations into Neo4j. For example from a csv file or piped output. Before streaming, it will check if a connection to neo4j can be made, and optionally indexes can be set to speed up the inserting process.

    Usage

    Given the following test.csv file:

    Fabien|likes|Nick
    Nick|dislikes|Fabien
    Jochem|likes|Nick
    Lydia|likes|Fabien
    

    And the following example script, that uses neo4j-batch-index-stream. You can then run less test.csv | node.js main.js to pipe the contents of test.csv into main.js, which converts the csv into objects that can be batch inserted in neo4j.

    //// main.js
     
    var through2 = require('through2');
    var split2 = require('split2');
    var Neo4jStream = require('neo4j-batch-index-stream');
     
    var buildRecords = through2({ objectMode: true }, function(chunk, enc, callback) {
        var line = chunk.toString().trim();
     
        if (line && line.length) {
            var parts = line.split("|");
     
            var Left = {
                "label": "Person",
                "name" : parts[0],
            }
     
            var Right = {
                "label" : "Person",
                "name" : parts[2],
            };
     
            var relation = {
                "relation" : parts[1],
                "start"    : Left,
                "end"      : Right,
            }
     
            this.push(Left);
            this.push(Right);
            this.push(relation);
        }
     
        callback();
    });
     
     
    var config = require("./secret-config.json");
    var username = config.username;
    var password = config.password; // your own password ;
     
     
    var stream = new Neo4jStream(username, password, {
        "index_key"     : "name",
        "highWaterMark" : 10000
    });
     
    stream.index([
        ["Person", "name"]
    ]);
     
     
    process.stdin
    .pipe(split2())
    .pipe(buildRecords)
    .pipe(stream)
    .on('error', function(error) {
        console.log(error);
    })
    .on('finish', function() {
        console.log("DONE");
    })

    Result

    Neo4j database with people and relations

    Possible errors

    Error: Error: connect ECONNREFUSED 127.0.0.1:7474

    Neo4j is not running or not available.


    Error: {
      "errors" : [ {
        "message" : "Invalid username or password.",
        "code" : "Neo.ClientError.Security.AuthorizationFailed"
      } ]
    }
    

    You either have the default neo4j/neo4j password or gave incorrect credentials.

    Contributing

    If you have neo4j running, you can do a simple batch insert with npm test.

    If you feel something is missing, you can open an issue stating the problem and desired result. If code is unclear give me a @mention. Pull requests are welcome.

    Install

    npm i neo4j-batch-index-stream

    DownloadsWeekly Downloads

    0

    Version

    0.0.1

    License

    MIT

    Last publish

    Collaborators

    • tessmore