@sweepbright/sbimport

sbimport

Installation

This tool is built using Node.js. You can install it globally using npm or yarn as follows:

For npm

npm install -g @sweepbright/sbimport

For yarn

yarn global add @sweepbright/sbimport

Once the process terminates, you can start using it. Let's try

sbimport --help

Another approach is to use npx to immediately use the tool.

npx @sweepbright/sbimport --help

Usage

Currently, sbimport support two commands:

• init: to initialise the import folder
• import: to import the datasets in an existing import folder.

Let's see in more details what this commands do.

init command

The init command expect a folder name as an argument. When executed, it will ask you the API Key and the environment. Provide them and it’s done!

You can now inspect the folder the tools created. If you have tree installed you should get a similar output.

sbimport init ImportCustomerX

tree ImportCustomerX

ImportCustomerX
├── company.config.yaml
├── contacts
├── logs
└── properties
    └── files
        └── README.txt

5 directories, 2 files

The folder is organised by entity that can be imported. Currently, we support import of properties (properties subfolder) and contacts (contacts subfolder).

We also have:

• company.config.yaml: the configuration file that contains the information specified during the init (we reserve the possibility to expand this in the future with more options).
• logs: The folder where all the logs will be persisted.

import command

The import command expect a folder name as an argument and support a few options. To have a full overview of the available option run

sbimport import --help

The import command will attempt to import every record it finds in the import folder provided as an argument. These behaviour can be changed by using specific options. Having the possibility to select what to import is useful when you just need to import specific records.

The sbimport keeps track of what has been process so that consecutive execution of the same dataset will not generate unnecessary import operations. It does that by storing an hash for each record and asset in the datasets and comparing it during subsequent executions. When the comparison fail we consider the record to be imported.

Some options are:

• --entities: Allow to import just the selected entity type.
• --force: By default the sbimport remembers what was processed to avoid importing the same data. To prevent that use this option.

Examples

To import everything in the import folder test

sbimport import test

To import just properties datasets in the import folder test

sbimport import test --entities property

To import properties datasets and relative images in the import folder test

sbimport import test --entities property --entities propertyAssetTo import just a specific contacts datasets between the multiple ones in the import folder `test`

Preparing the datasets

The sbimport tool expect data in JSONL format. In short, files using JSONL expect to have a valid JSON object per line.

To convert a normal JSON file to the JSONL version that the tool expect, you could use jq as follows:

jq -c '.[]' < dataset.json > dataset.jsonl

Once your datasets are ready, move them to the respective folders:

• Properties datasets in properties folder
• Contacts datasets in the contacts folder.

Preparing properties files

To support SweepBright properties file options, we decided to organise the import folder properties files using the following hierarchy (it’s also described in the Readme file inside the files folder)

This directory is used to store the properties files.

The structure is as follows:

{property-reference}/
    ├── documents/
    │   ├── private/
    │   └── public/
    ├── images
    │   ├── private/
    │   └── public/
    └── plans
        ├── private/
        └── public/

Replace {property-reference} with your property ID.

For example, assuming you are importing one property with ID a3155152-3cb3-4878-b1e6-39466844328c, and this property has:

• doc1.pdf public and doc2.pdf private
• img1.png public and img2.png private
• and no plans

The files folder should look like that

files
└── a3155152-3cb3-4878-b1e6-39466844328c
    ├── documents
    │   ├── private
    │   │   └── doc2.pdf
    │   └── public
    │       └── doc1.pdf
    ├── images
    │   ├── private
    │   │   └── img2.png
    │   └── public
    │       └── img1.png
    └── plans
        ├── private
        └── public