Fast imports from OSM XML into a fresh osm-p2p-db
This is useful if you have a very large OSM dataset that would otherwise be too
slow to insert directly into
How much faster?
- osm-p2p-server: 10.5s
- osm-p2p-db-importer: 7.3s
- osm-p2p-server: 720+ seconds (stopped recording)
- osm-p2p-db-importer: 73.3s
How does it work?
A fresh import is a special case that lets us make a key assumption:
- No historic data (no data in the import is an earlier revision of other data)
There is a lot of cost in ensuring consistency across the layers of
osm-p2p-*, which can be potentially skipped if we know we don't need to worry
about inconsistent data coming in.
This module skips multiple layers, accepting OSM XML data and generating LevelDB
batch operations directly, without the
layers of processing in between. This isn't very stable in the long-term (since
the things under the blankets of these modules may change), but for now it makes
for some very quick data insertions into a fresh DB.
var importer =var fs =var xml = fsvar dbPath = '/tmp/osm-p2p-ex'
var importer =
importer(osmDir, xmlStream, opts, done)
osmDir: path to a directory where the
osm-p2p-dbwill be located.
xmlStream: a readable stream of OSM XML data.
opts: options object. Currently accepts
done: a callback function, receiving an error
importer.toLevel(db, xmlStream, done)
Like the above, except operating on a user-specified LevelUP instance,
This is useful if you aren't using a filesystem-backed LevelDOWN as your
USAGE: osm-p2p-db-importer [--slow] DBDIR [XMLFILE] Import OSM XML into an osm-p2p-db located at DBDIR. XMLFILE is a file containing OSM XML. If not specified, OSM XML is read from standard input. --slow, -s: skip fragile optimizations; import using the normal osm-p2p-db process.
Anything that would be an acceptable upload to OSM v0.6's
POST /api/0.6/changeset/:id/upload should work here.
Caveat: Changeset ID
Note that this module doesn't create a changeset for you. You will need to create one first, an ensure your XML has that changeset set.
With npm installed, run
As a Library
$ npm install osm-p2p-db-importer
As a Command Line Utility
$ npm install --global osm-p2p-db-importer