x12-parser
    TypeScript icon, indicating that this package has built-in type declarations

    1.1.1 • Public • Published

    Build Status

    jsdoc documentation: https://tastypackets.github.io/x12-parser/ Current d.ts are auto-generated from JS and jsdoc until all files are converted to TypeScript.

    Description

    An X12 parser built using the NodeJS stream API, which can handle very large files and significantly improves processing time by reducing RAM usage. This parser was built using only JS and native NodeJS APIs, the only dependencies in use are dev dependencies for documentation and testing.

    Why are streams important?

    EDI files can be very large and may be transmitted with multiple transactions inside one file. With the unknown size of these files it's very easy for them to consume lots of RAM.

    Below are some rough examples, later this should be updated with more precise measurements.

    File Size on Disk Number of Lines No Grouping (RAM) Group on CLP (RAM) Group on ISA (RAM)
    15.5MiB 565,832 47MiB 53MiB 294MiB
    Objects exiting stream - 565,832 20,166 1

    Example usage

    yarn add x12-parser or npm install x12-parser

    The X12parser is a Node Transform Stream, so you must pipe a read stream to it. The simplest and most common way to do this is using fs.createReadStream in NodeJS, however you should be able to use other read streams as long as they are sending the data unmodified from the source file. If you need to control the encoding you can pass defaultEncoding to the X12parser, for example X12parser('utf8').

    The X12parser will then push every segment as an object, you can either pip() this data to another stream for further processing or you can directly consume the data events from the X12parser.

    Every segment will be an object with a name key and multiple int keys, incrementing based on their element number. For example if there was a HAI segment with only 1 element it would look like this: {"name": "HAI", "1": "HI!"}.

    // Imports
    const { X12parser } = require("x12-parser");
    const { createReadStream } = require("fs");
    
    // Create a new parser
    const myParser = new X12parser();
    myParser.on("error", (err) => {
      console.error(err);
    });
    
    // Create a read stream from a file
    const ediFile = createReadStream("./testFile.edi");
    ediFile.on("error", (err) => {
      console.error(err);
    });
    
    // Handle events from the parser
    ediFile.pipe(myParser).on("data", (data) => {
      console.log(data);
    });
    
    // Output example:
    // { --- 1st Obj emitted
    //     '1': '00',
    //     '2': '',
    //     '3': '00',
    //     '4': '',
    //     '5': 'ZZ',
    //     '6': 'EMEDNYBAT',
    //     '7': 'ZZ',
    //     '8': 'ETIN',
    //     '9': '100101',
    //     '10': '1000',
    //     '11': '^',
    //     '12': '00501',
    //     '13': '006000600',
    //     '14': '0',
    //     '15': 'T',
    //     '16': ':',
    //     name: 'ISA'
    // }
    // { --- 2nd Obj emitted
    //     '1': 'HP',
    //     '2': 'EMEDNYBAT',
    //     '3': 'ETIN',
    //     '4': '20100101',
    //     '5': '1050',
    //     '6': '6000600',
    //     '7': 'X',
    //     '8': '005010X221A1',
    //     name: 'GS'
    // },

    Notes

    This lib does not enforce 2-character segment & component names, it simply increments the number. For example, ISA01 would be ISA1 in the output object and ISA10 would still be ISA10. Similarly, components such as SVC01-01 would be SVC1-1.

    If this becomes an issue for processing the 2-character enforcement can be added, however it was dropped to reduce processing of each segment name. Please let me know if you have any feedback on this and I may change it in v2.

    Grouping

    WARNING: This will be fully rewritten and replaced in v2. Current version was expiermental and not intended for production use.

    The grouping stage is another transform stream, it is designed to help group unbounded loops in the X12 files. You must provide it a list of groups and what indicates the start / stop of a loop. If the loop is unbounded the stop is null.

    Every group will be accumulated until another group starts, at which point the group will be pushed down the stream. This should be kept in mind when grouping, because making large groups could cause memory issues and reduce performance of streams. If a loop is unbounded it will be terminated by either another peer loop or it will be terminated by the closer of a parent loop.

    The grouper can accept an array of schemas or a single schema. The grouper will try to select the schema version that matches GS08. If no schemas match GS08 it will look for a schema with the defaultSchema key set to true, if there are none it will then select the first schema in the array of schemas.

    Below is an example of grouping everything inside the ISA, however this is not a good idea for production. There is an unknown number of nested elements and it may have memory issues. It would be better in this case to group on something like the CLP lines.

    Grouper Example

    const { X12parser, X12grouper, Schema } = require("x12-parser");
    const { createReadStream } = require("fs");
    
    const schema = {
      start: "ISA", // What segment starts the group
      end: "IEA", // What segment ends the group
      name: "Envelope", // What is the name of the group
      groups: [
        // Nested groups
        {
          start: "BPR",
          terminators: ["N1"],
          name: "headers",
        },
        {
          start: "N1",
          terminators: ["LX"],
          name: "1000",
        },
        {
          start: "LX",
          name: "2000",
          terminators: ["SE"],
          groups: [
            {
              start: "CLP",
              name: "2100",
              groups: [
                {
                  start: "SVC",
                  name: "2110",
                },
              ],
            },
          ],
        },
      ],
    };
    
    const myParser = new X12parser();
    const mySchema = new Schema("005010X221A1", schema);
    const myGrouper = new X12grouper(mySchema);
    
    const testFile = createReadStream("./test/testFiles/835/profee.edi");
    
    testFile
      .pipe(myParser)
      .pipe(myGrouper)
      .on("data", (data) => {
        console.log(data);
      });

    Install

    npm i x12-parser

    DownloadsWeekly Downloads

    960

    Version

    1.1.1

    License

    MIT

    Unpacked Size

    31 kB

    Total Files

    15

    Last publish

    Collaborators

    • tastypackets