@politico/postal-service

POLITICO Service for moving data from APIs to S3.

Why This?

CORS is the devil. This tool is used to create a "shipping lane". Shipping lanes take data from an API endpoint and put it on S3... and then it keeps doing that over and over again to keep the S3 file up to date. Because it's a node process, you don't have to deal with CORS errors when fetching from the API.

This is meant as a tool for development and should not be used in a production environment. If you need up-to-date data on S3 for a production build, you should set up a proper publishing pipeline.

Quick Start

Installation:

$ npm install -g @politico/postal-service

"Shipping" Data

To start a new shipping lane run...

$ postal ship <URL_FOR_DATA> <S3_BUCKET_KEY>

Intervals

By default, ship will go on forever until the process is cancelled. You can set an explicit number of intervals by using --intervals or -i and setting it equal to the number of intervals you want to run. You can also change the delay between each interval (defaults to 5 seconds) by using --delay or -d and setting it equal to a number of milliseconds.

Buckets

By default, ship will publish to staging (since this is a development tool). If you want to publish to the production bucket set the --production flag or use the --bucket or -b flag and set it equal to a bucket.