usgs-stream
A node.js application to asynchronously transform a stream of USGS tab-delimited (rdb) water data to JSON.
Note: The USGS Instantaneous Values Web Service is the official way to do this sort of thing.
Installation
npm install usgs-stream
Start Server
$ usgs_stream
usgs-stream listening to http://127.0.0.1:5000/
or
$ node app.js
usgs-stream listening to http://127.0.0.1:5000/
Usage
Retrieve data using the URL format
http://127.0.0.1:5000/[site number]
Example: Request data from Point of Rocks, MD (site 01638500)
curl http://127.0.0.1:5000/01638500
or open the URL in your browser (see a live example).
A JSON object is returned (see below) -- a straightforward conversion of the original tab-delimited data.
Time Parameters
The date and time range retrieved can be selected in two ways.
Option 1: Period (default)
- period=PxD: Number of days x (before now) that are returned [default: x = 7 days]
Example: Return data for the last 5 days
http://127.0.0.1:5000/01638500?period=P5D
If it's 3 pm on Saturday, this will return data from 3 pm last Monday.
Option 2: Date range
- startDT=YYYY-MM-DD
- endDT=YYYY-MM-DD
Example: Return data from March 3 to March 12, 2015
http://127.0.0.1:5000/01638500?startDT=2015-03-03&endDT=2015-03-12
This will return data from midnight on March 3 to 23:59 on March 12 in the local time of the USGS site.
The time can be provided using the ISO-8601 format YYYY-MM-DDTHH:MM
in the local time zone, or YYYY-MM-DDTHH:MMZ
for UTC.
JSON Output
The returned JSON object has the following format.
-
header
: Metadata header-
header.title
: Site title -
header.retrieved
: Date the data was retrieved -
header.fields
: Data fields-
header.fields.desc
: Description of each field -
header.fields.type
: Data type (date, month, numeric, or string)
-
-
header.qualifications
: Data value qualification codes that appear in the data set -
header.raw
: Original header
-
-
data
: An array of data rows, where each row has the following properties-
data.agency_cd
: Agency collecting the data or maintaining the site -
data.site_no
: USGS site identification number -
data.datetime
: Local date and time in ISO-8601 format -
data.tz_cd
: Local time zone -
data.nn_nnnnn
: Measured data value (described inheader.fields
) -
data.nn_nnnnn_cd
: Qualification code for this data point (described inheader.qualification
) -
data.utc
: UTC date and time in ISO-8601 format
-
Scripting
usgs-stream can be used in scripts to convert a readable USGS water data stream to a writable JSON stream.
#!/usr/bin/env node
var usgs = require('usgs-stream');
var fs = require('fs');
var rdb = fs.createReadStream('data.rdb')
var json = fs.createWriteStream('data.json')
usgs.transform(rdb, json);