2.0.0 • Public • Published

    darp² (Distributed Automated Remote Processing Pipeline)

    Build Status

    darp² is an experimental tool for distributed automated remote processing pipelines for point clouds to be used in conjunction with DAT. Its purpose is to allow processing of data, that is remotely stored in a distributed network without the need to create a local copy of the data first. Instead only necessary data is requested from its peers The data is streamed through a processing pipeline, that writes the result into a directory, that is immediately available as a Dat archive again.

    darp² can be used as a CLI (Command Line Interface; aka. terminal) tool or as a Node.js library.

    CLI usage

    darp² is distributed using npm, the package manager for Node.js. Please install the latest version of Node.js from the official homepage.

    To install darp² you then type the following command:

    $ npm install darp2 --global

    Once that succeeds the darp2 command should be available inside your terminal.


    $ darp2 --help
    darp2 -i dat://... -g "**/*" -c "ls -lash \"\${DARP2_TEMP_FOLDER}\" > ls.out"
      --version          Show version number                                                   [boolean]
      --input, -i        Input DAT key                                               [string] [required]
      --run-script, -r   Script to run on the task.                                             [string]
      --cmd, -c          Command to run on after downloading the dat.                           [string]
      --work-folder, -w  Folder used for the output. Passed as `DARP2_WORK_FOLDER` and
                         `DARP2_UPLOAD_FOLDER` env variable to the cmd.                         [string]
      --temp-folder, -t  Folder used to download the dat. Passed as `DARP2_TEMP_FOLDER` and
                         `DARP2_DOWNLOAD_FOLDER` env variable to the cmd.                       [string]
      --nolog            Dont write the log file to the out dat.              [boolean] [default: false]
      --log-path         File to write the log into.              [string] [default: ".dat-through.log"]
      --glob, -g                                                              [string] [default: "**/*"]
      --glob-nobrace     Do not expand `{a,b}` and `{1..3}` brace sets.       [boolean] [default: false]
      --glob-noglobstar  Disable `**` matching against multiple folder names. [boolean] [default: false]
      --glob-dot         Allow patterns to match filenames starting with a period, even if
                         the pattern does not explicitly have a period in that spot.
                         Note that by default, `a/**/b` will **not** match `a/.d/b`, unless `dot`
                         is set.                                              [boolean] [default: false]
      --glob-noext       Disable "extglob" style patterns like `+(a|b)`.      [boolean] [default: false]
      --glob-nocase      Perform a case-insensitive match.                    [boolean] [default: false]
      --glob-nonull      When a match is not found by `minimatch.match`return a list containing
                         the pattern itself if this option is set.  When not set, an empty list
                         is returned if there are no matches.                 [boolean] [default: false]
      --glob-matchBase   If setthen patterns without slashes will be matched
                         against the basename of the path if it contains slashes.  For example,
                         `a?b` would match the path `/xyz/123/acb`, but not `/xyz/acb/123`.
                                                                              [boolean] [default: false]
      --glob-nocomment   Suppress the behavior of treating `#` at the start of a pattern as a
                         comment.                                             [boolean] [default: false]
      --glob-nonegate    Suppress the behavior of treating a leading `!` character as negation.
                                                                              [boolean] [default: false]
      --glob-flipNegate  Returns from negate expressions the same as if they were not negated. (Ie, true
                         on a hit, false on a miss.)                          [boolean] [default: false]
      --concurrency, -p  Specifies how many files and folders should be max. processed in parallel.
                                                                                   [number] [default: 1]
      --help             Show help                                                             [boolean]

    PDAL Example

    This example shows how darp² can take a dat link and then run a processing pdal processing pipeline to pointcloud files and share the output of the processing pipeline with a new dat link:

    $ darp2 \
      --input dat://74117a268a95438ceabcde6541dc9f5f00f5c21f192bd14f17258e5ca9ff194b \
      --cmd "pdal pipeline $(pwd)/processing_pipeline.json --readers.las.filename=\"\${DARP2_TEMP_FOLDER}/lidar_raleigh_nc_spm_height_feet.laz\""

    This example specifies the input as the dat available under the dat://74117a..b link. The dat link should start with dat:// followed by the 64-character hex string.

    It runs a pdal processing pipeline that is defined by the processing_pipeline.json file (You can download it here)

    The output will generate a new dat link ready for sharing.


    MIT License


    npm i darp2

    DownloadsWeekly Downloads






    Unpacked Size

    47.9 kB

    Total Files


    Last publish


    • leichtgewicht