High performance distributed data processing and machine learning.
- Multiple data sources: filesystems, databases, cloud (S3, azure)
- Multiple data formats: CSV, JSON, Columnar (Parquet)...
- 50 high level operators to build parallel apps
- Machine learning: scalable classification, regression, clusterization
- Run interactively in a nodeJS REPL shell
- Docker ready, simple local mode or full distributed mode
- Very fast, see benchmark
npm install skale
Word count example:
var sc = ;sc;
In local mode, worker processes are automatically forked and communicate with app through child process IPC channel. This is the simplest way to operate, and it allows to use all machine available cores.
To run in local mode, just execute your app script:
or with debug traces:
SKALE_DEBUG=2 node my_app.js
In distributed mode, a cluster server process and worker processes must be started prior to start app. Processes communicate with each other via raw TCP or via websockets.
To run in distributed cluster mode, first start a cluster server
On each worker host, start a worker controller process which connects to server:
./bin/worker.js -H server_host
Then run your app, setting the cluster server host in environment:
SKALE_HOST=server_host node my_app.js
The same with debug traces:
SKALE_HOST=server_host SKALE_DEBUG=2 node my_app.js
- Contributing guide
- Gitter for support and discussion
- Mailing list for discussion about use and development