Distributed URL safe short ID generator.
This module was inspired by icicle and snowflake. The ideas is to be able to generate non-colliding, URL friendly, and relatively short IDs that could be used in any application that requires to create URIs for arbitrary resources.
Looking around for what is available, I failed to find anything that would be simple and easy to implement. As a result, this module was born.
The id is a 64bit unsigned integer with 42 bits used for current timestamp in milliseconds, 10 bits used for shard id, and final 12 bits are used for revolving sequence.
Official repository is at http://gotfix.com/pixnr/short-duid-js
This module is roughly 5 times slower compared to its native implementation https://gotfix.com/pixnr/short-duid but does not require any compilation, including dependencies. If speed is what you need, please use short-duid. Except few convenience methods, they are both API compatible and can act as a drop-in replacement for each other.
npm install short-duid-js --save
This module is very simple to use, first you will need to initialize it and then start using its instance.
Instantiates short-duid and sets parameters for the life of instance.
shard_id- ID of this instance of short-duid, should be unique and not shared with other instances in the cluster; from 0 to 1023. This parameter will be converted into signed 32 bit integer and masked to fit in 12 bits.
salt- Salt that is used by hashid encoder/decoder, should be constant and shared across all nodes in the cluster. Do not change this parameter once used in production, or you will have collisions in the alphanumeric IDs. Good way to generate salt on Linux:
dd if=/dev/random bs=1 count=102400 2>/dev/null| sha256sum
epoch_start- Number of milliseconds since unix epoch (1970, Jan 1 00:00:00 GMT). This should be some date in the near past and should never be changed further into the future once in production. Example: 1433116800000; //Mon, 01 Jun 2015 00:00:00 GMT. This parameter will be converted to unsigned 64bit integer.
Method to retrieve array of DUIDs in alphanumeric form. Length of the array is specified by
[ "XLz0E3MvkEL" ]
count- Number of alphanumeric DUIDs to return, from 0 to 8192.
Essential same method as
_instance_.getDUID but instead of hashid converted integer, will return unique ID in a numeric form as string.
[ "12534941854212112" ]
count- Number of numeric DUIDs to return, from 0 to 8192.
Method to get currently set shard ID of ShortDUID
numbercurrent shard ID of ShortDUID
Method to get currently set custom epoch starting time in milliseconds of ShortDUID
stringcurrently set custom epoch of ShortDUID
_instance_, since it is unsigned 64bit integer, we return it as string.
Method to get current salt of ShortDUID
_instance_. Salt is used to generate alphanumeric DUIDs and also in
stringcurrently set salt that is used in hashid conversion of ShortDUID
"this is my salt"
Method to hash(encode) array of unsigned 64bit integers (in
stringhashid of array of unsigned 64bit integers
Decode previously encoded array of numbers with hashid method.
['1', '2', '3']
hashid_string- Hashid in a string form. Example:
This API is mainly used by unit tests and should not be required for normal usage of the module. Use it at your own risk.
Method to get current time since unix epoch in milliseconds as seen by the module, not adjusted for custom epoch. This method can be useful in testing and also in capturing reference time to ensure timer stability across restarts.
stringof numbers, current time since unix epoch in milliseconds as seen by the module.
Method to help simulate
system_clock drift, can accept positive or negative integers.
stringnumber of milliseconds to drift ShortDUID's internal clock
milliseconds(optional) number of milliseconds to drift system_clock by, can be a positive or negative integer.
Simplest example to execute all of the major methods of the module.
var duid = ;var duid_instance = duid;console;console;
More complete example that will create API server with help of koajs and reply to queries.
'use strict';var pm2 = ;var cpus = length;var procs = Math;pm2;
'use strict';var koa = ;var router = ;var app = moduleexports = ;var duid = ;//Define app name and port for koa-clustervar cpus = length;appname = "ShortDUID";appnode_id = 0;appnid = processenvNODE_APP_INSTANCE ? processenvNODE_APP_INSTANCE : processpid % cpus; //nodejs instance IDappshard_id = appnode_id + appnid;appport = 65000;appsalt = "this is my super secret salt";appepoch_start = 1433116800 * 1000; //Mon, 01 Jun 2015 00:00:00 GMT//Instantiate short-duidvar duid_instance = appshard_id appsalt appepoch_start;//Setup routesrouter;//Setup middlewareapp;app;
And then you should install application with
npm install --save and start the server by
node index.js. You can check the logs and also list the processes with
./node_modules/.bin/pm2 list. Getting fresh id can be done by curl:
For more examples please see
examples folder, which I plan to keep adding to. You are free to contribute more examples.
So far I know of none, if you are using it in your project and do not mind sharing this information, please drop me a note at firstname.lastname@example.org, and I will add you to this list.
git clone https://gotfix.com/pixnr/short-duid-js.git && cd short-duid-js && npm install --save-dev
npm test && npm run bench
$ npm test> email@example.com test /builds/pixnr/short-duid-js> ./node_modules/mocha/bin/mocha --harmony --reporter spec ./test/Short DUID#getEpochStart()✓ should return set epoch start, for instance #1: 1433116800000✓ should return set epoch start, for instance #2: 1433116800000✓ instance #1 and instance #2 should return same epoch start: 1433116800000✓ should reset custom epoch to zero if given one larger than real epoch✓ should accept custom epoch that is even 1 millisecond in the past#getSalt()✓ should return set salt, for instance #1: 39622feb2b3e7aa7208f50f45ec36fd513baadad6977b53295a3b28aeaed4a54✓ should return set salt, for instance #2: 39622feb2b3e7aa7208f50f45ec36fd513baadad6977b53295a3b28aeaed4a54✓ instance #1 and instance #2 should return same salt: 39622feb2b3e7aa7208f50f45ec36fd513baadad6977b53295a3b28aeaed4a54#getShardID()✓ should overflow if shard id is set to integer that does not fit in 10 bits: 1024 --> 0✓ should overflow if shard id is set to integer that does not fit in 10 bits: 1025 --> 1✓ should return set shard id for id that fits within 10 bits: 1023 --> 1023✓ should return set shard id for id that fits within 10 bits: 0 --> 0✓ should return set shard id for instance #1: 123✓ should return set shard id for instance #2: 12✓ should return different shard ids for instance #1 and instance #2#getDUID()✓ Asked for 1 DUIDs, correctly returns 1 DUIDs✓ Asked for 0 DUIDs, correctly returns 0 DUIDs✓ Asked for 8192 DUIDs, correctly returns 8192 DUIDs (467ms)✓ Asked for 8193 DUIDs, correctly returns 1 DUIDs✓ should have no duplicates in the returned arrays, 8192 IDs each, and combined. (1162ms)#getDUIDInt()✓ Asked for 1 Int DUIDs, correctly returns 1 Integer DUIDs✓ Asked for 0 Int DUIDs, correctly returns 0 Integer DUIDs✓ Asked for 8192 Int DUIDs, correctly returns 8192 Integer DUIDs (423ms)✓ Asked for 8193 Int DUIDs, correctly returns 1 Integer DUIDs✓ should have no duplicates in the returned arrays, 8192 IDs each, and combined. (133ms)DUID with drifting time✓ should return same drift time as given as parameter✓ should generate ID with -5890 millisecond drift into the past from now( 6012689289 ), 25219046593507329 should be numerically smaller than 25219071440564225✓ should consistently generate unique IDs even when time is drifting backwards constantly (523ms)28 passing (3s)$ cat /proc/cpuinfo | grep "^model name" | uniqmodel name : Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz$ npm run-script bench> firstname.lastname@example.org bench /builds/pixnr/short-duid-js> /usr/bin/env node benchmarks/test.jssingle DUIDInt generation x 212,771 ops/sec ±6.31% (84 runs sampled)batch of 10 DUIDInt generation x 22,137 ops/sec ±5.96% (91 runs sampled)single DUID generation x 21,336 ops/sec ±3.27% (92 runs sampled)batch of 10 DUID generation x 2,169 ops/sec ±3.11% (94 runs sampled)single DUID generation (1 character salt) x 21,081 ops/sec ±3.84% (95 runs sampled)batch of 10 DUID generation (1 character salt) x 2,118 ops/sec ±3.42% (92 runs sampled)
All are welcome to submit pull requests and patches
The MIT License (MIT)
Copyright (c) 2015 Ian Matyssik email@example.com
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.