elasticsearch-tools
A collection of elasticsearch command line tools for doing things like bulk importing/exporting and exporting/importing mappings.
It was created because some of the existing import/export tools ran too slow on my machine. Using the new bulk API seemed to speed things up dramatically. The other tools I used also weren't exporting _parent and _routing fields.
Installation
npm install -g es-tools
After installing, you will have access to the following command line tools:
Exporting
Importing
Usage: es-source
[elasticsearch-tools ]$ es-source --help Usage: es-source [options] Options: -h, --help output usage information -u, --host [url] elasticsearch url to connect to [http://localhost:9200] -s, --scrollId [string] provided
Usage: es-sink
[elasticsearch-tools ]$ es-sink --help Usage: es-sink [options] Options: -h, --help output usage information -v, --version output the version number -u, --host [string] the elasticsearch host to connect to -b, --batchSize [num] the number of operations to send per bulk request -c, --consistency [one|quorum|all] elasticsearch consistency setting
Usage: es-export-bulk
Options
es-export-bulk --help Usage: es-export-bulk [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to -m, --max <number> the maximum number of items to export. different than the scroll size --transformMeta <js> a javascript --transformSource <js> a javascript --transformMetaInit <js> a javascript
Examples
export 1 hour of data from local db
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --body '{"query":{"range":{"timestamp":{"gte":"2014-08-13T11:00:00.000Z","lte":"2014-08-13T12:00:00.000Z"}}}}'
export "myIndex" from local db
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --index myIndex
add a key/value to all exported documents
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = "neat"'# the return statement is optional es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = "neat";return data;'
delete the key "foo" from all exported documents
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'delete data.foo'
don't include _parent in meta data
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'delete data.index._parent'
change the index name that we export
es-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'data.index._index = "newIndex"'
Usage: es-export-mappings
Options
es-export-mappings --help Usage: es-export-mappings [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --index <index> ES OPTION: String, String[], Boolean — A comma-separated list of index names --type <type> ES OPTION: String, String[], Boolean — A comma-separated list of document types --ignoreUnavailable <ignoreUnavailable> ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable --allowNoIndices <allowNoIndices> ES OPTION: Boolean — Whether to ignore
Examples
export mappings from local db
es-export-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json
Usage: es-export-settings
Options
es-export-settings --help Usage: es-export-settings [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --index <index> ES OPTION: String, String[], Boolean — A comma-separated list of index names --ignoreUnavailable <ignoreUnavailable> ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable --allowNoIndices <allowNoIndices> ES OPTION: Boolean — Whether to ignore
Examples
export settings from local db
es-export-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json
Usage: es-export-aliases
Options
es-export-aliases --help Usage: es-export-aliases [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --index <index> ES OPTION: String, String[], Boolean — A comma-separated list of index names --local <local> ES OPTION: Boolean — Return local information, do not retrieve the state from master node --name <name> ES OPTION: String, String[], Boolean — The name of the settings that should be included
Examples
export aliases from local db
es-export-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json
Usage: es-import-bulk
Options
es-import-bulk --help Usage: es-import-bulk [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to -m, --max <items> the max number of lines to process per batch
Examples
import data to local db from file
es-import-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/rafflev1.json
Usage: es-import-mappings
Options
es-import-mappings --help Usage: es-import-mappings [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --ignoreConflicts <ignoreConflicts> ES OPTION: Boolean — Specify whether to ignore conflicts
Examples
import mappings to local db
es-import-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json
Usage: es-import-settings
Options
es-import-settings --help Usage: es-import-settings [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --masterTimeout <masterTimeout> ES OPTION: Date, Number — Specify timeout
Examples
import settings to local db
es-import-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json
Usage: es-import-aliases
Options
es-import-aliases --help Usage: es-import-aliases [options] Options: -h, --help output usage information -v, --version output the version number -u, --url <url> the elasticsearch url to connect to -f, --file <file> the file to write data to --timeout <timeout> ES OPTION: Date, Number — Explicit operation timeout --masterTimeout <masterTimeout> ES OPTION: Date, Number — Specify timeout
Examples
import aliases to local db
es-import-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json
Other Elasticsearch Tools
Imports / Exports
License
Copyright (c) 2014 skratchdot
Licensed under the MIT license.