darp² (Distributed Automated Remote Processing Pipeline)
darp²
is an experimental tool for distributed automated remote processing pipelines for point clouds to be used in conjunction with DAT. Its purpose is to allow processing of data, that is remotely stored in a distributed network without the need to create a local copy of the data first. Instead only necessary data is requested from its peers The data is streamed through a processing pipeline, that writes the result into a directory, that is immediately available as a Dat archive again.
darp²
can be used as a CLI (Command Line Interface; aka. terminal) tool or as a Node.js library.
CLI usage
darp²
is distributed using npm
, the package manager for Node.js. Please install the latest version of
Node.js from the official homepage.
To install darp²
you then type the following command:
$ npm install darp2 --global
Once that succeeds the darp2
command should be available inside your terminal.
Options
$ darp2 --help darp2 -i dat://... -g "**/*" -c "ls -lash \"\${DARP2_TEMP_FOLDER}\" > ls.out" Options: --version Show version number [boolean] --input, -i Input DAT key [string] [required] --run-script, -r Script to run on the task. [string] --cmd, -c Command to run on after downloading the dat. [string] --work-folder, -w Folder used
PDAL Example
This example shows how darp²
can take a dat link and then run a processing pdal processing pipeline to pointcloud files and share the output of the processing pipeline with a new dat link:
$ darp2 \ --input dat://74117a268a95438ceabcde6541dc9f5f00f5c21f192bd14f17258e5ca9ff194b \ --cmd "pdal pipeline $(pwd)/processing_pipeline.json --readers.las.filename=\"\${DARP2_TEMP_FOLDER}/lidar_raleigh_nc_spm_height_feet.laz\""
This example specifies the input as the dat available under the dat://74117a..b
link. The dat link should start with dat://
followed by the 64-character hex string.
It runs a pdal processing pipeline that is defined by the processing_pipeline.json
file (You can download it here)
The output will generate a new dat link ready for sharing.