A fast and efficient Node.js storage engine for stock market tick data and couple of modules for trading software development.
An extremely fast and efficient Node.js storage engine for raw stock market tick data as well as a couple of handy modules for trading software development.
node-stock is used by my proprietary blackbox trading company for all our robots. The code has been in production for more than a year now, so it is considered mature.
The code is well coverted by unit tests.
The main thing: a tick storage engine.
- ExtraNumber - incredibly small yet vital module for internal price representation in node-stock.
- ExtraDate - date and time calculations specifically helpful for algorithmic trading on NYSE, NASDAQ and CME;
- WorkingDays - NYSE and NASDAQ working days calculator;
- TimePeriod - defines a list of human-readable day time periods and a few calculations with it;
- ExtraLog - console.log and formatter functions tailored to node-stock internal formats of price and date/time;
- CandlesCalculator - Fast OLHC calculator over raw tick data;
All prices in node-stock and derived software are stored in integer in 1/100 of a cent. I.e. $23,45 becomes 234500. To convert a conventional price to this format, use
parseInt(price*10000). To show a human-readable version, use
Number.humanReadablePrice() in ExtraNumber:
require'stock/ExtraNumber';var price = 23.56;price = parseIntprice*10000;console.log"Price = %s" pricehumanReadablePrice;// or:require'stock/ExtraLog';ExtraLog.log"Price = %p" price;
Daystamp. A daystamp is a date in form of YYYYMMDD. It can be Number or String, so your code must handle both cases. See
ExtraDate, it contains all you need for daystamp calculations.
Day minute. It's the number of a minute since 00:00 in the current day. I.e. 9:30 am becomes 570 (9*60+30=570). BTW, 24hr time is used thorough the project and so should you.
dbPath. Path to the tickers database, a folder. See below for database structure description.
Time zones management. You absolutely should run your software in the time zone of the stock exchange you are trading with. There are two ways of doing that:
TZ environment variable:
TZ=America/New_York node something.js
process.env.TZ right at the start of your script before making anything else, especially before calling any Date methods:
processenvTZ='America/New_York';console.log"Hello at %s" ;// or:require'stock/ExtraDate';require'stock/ExtraLog';processenvTZ='America/New_York';ExtraLog.log"Hello at %D" Dateunixtime;
- Node.js >= 0.6.0; s
- node-date-utils https://github.com/JerrySievert/node-date-utils or
npm install date-utils;
- node-compress-buffer https://github.com/egorFiNE/node-compress-buffer or
npm install compress-buffer;
- optimist https://github.com/substack/node-optimist.git or
npm install optimist;
- nodeunit for unit testing;
Note: nodeunit must be launched in New York time zone:
TZ=America/New_York nodeunit test/
npm install stock.
A: It's not. If you actually start developing your own ticks database you will inevitably end up with a similar format. For a funny reference: this format is about forty times smaller and way faster than text files.
A: Unfortunately, no. My license doesn't allow reselling of tick data. However, if you need some tick data strictly for R&D - please get in touch with me by email, we can work something out.
However, I have made one particular NYSE ticker publicly available so that you can play around with data. Should be enough for basic testing and development. It's anonymized, but the data is correct and intact; please don't try to figure out which company it is. You can find nyse.zip in Downloads area at Github.
node-stock adds methods to
Date (see ExtraDate) and
Number (see ExtraNumber).
This practice is wrong. Please do not inherit it in your project. There are only extremely rare cases when it's okay to do so, and I consider these two modules an example of those.
Database path (
dbPath) is a folder which contains tickers folders:
/mnt/storage/tickers/ /mnt/storage/tickers/AAPL/ /mnt/storage/tickers/ORCL/ /mnt/storage/tickers/YHOO/ …
Each ticker folder contains TickStorage files, each named with a daystamp of that day:
/mnt/storage/tickers/YHOO/20110103.ticks /mnt/storage/tickers/YHOO/20110104.ticks …
Each file is in TickStorage format.
You don't need to traverse this folder structure manually: there are two handy modules that will do that for you:
Symbol. They abstract the actual file storage from you.
Here's a handy sinopsis:
Symbols = require'stock/Symbols';Symbol = require'stock/Symbol';TickStorage = require'stock/TickStorage';require'stock/ExtraDate';require'stock/ExtraNumber';var dbPath = '/Users/egor/tickers';// initialise the class that works with tickers database (a folder of tickers folders)var symbols = dbPath;// now actually load the list of tickers (folders) available at that pathif !symbolsloadconsole.log"Cannot load tickers database!";return;// now let's iterate over all tickers availablevar symbol;while symbol=symbolsnext// mind that here "symbol" is a instantiated "Symbol" class// actually load list of days (files) at for that ticker (folder)if !symbolloadconsole.log"Cannot load days for %s" symbolsymbol;return;// prepare tick storage for the first day of that tickervar tickStorage = dbPath symbolsymbol symbolfirstDay;// actually load the raw tick data for this symbol at that dayif !tickStorageloadconsole.log"Cannot load ticks for %s/%s" symbolsymbol symbolfirstDay;return;// iterate over market ticks and calculate total volumevar tick totalVolume=0;while tick=tickStoragenextTickif tickisMarkettotalVolume+=tickvolume;console.log"%s total volume = %d" symbolsymbol totalVolume;
node-stock is triple-licensed.
If you are using node-stock in-house and do not redistribute the software, you can use it under LGPL v2.1. Essentially this means "do whatever you want as long as it's in-house". Feel free to provide commercial services or make zillions of money with this software. Just drop me an email so I can share your joy.
If you are redistributing node-stock, please consider it GPLv2 licensed.
If neither option is good for you, I can sell you a commercial license, which also includes my personal support. Contact me by email.
node-stock is created by Egor Egorov, email@example.com. I welcome your comments and suggestions, please feel free to drop me an email.