Locality-sensitive hashing (LSH) reduces the dimensionality of high-dimensional data and hashes input items so that similar items map to the same “buckets” with high probability (the number of buckets being much smaller than the universe of possible input items).
$ npm install lsh --save
To figure out how to use this package please take a look at test files
const RandomWords =const Lsh =const config =storage: 'memory'shingleSize: 5numberOfHashFunctions: 120const lsh = Lshconst numberOfDocuments = 100const documents =// generate random documents containing 100 words eachfor let i = 0; i < numberOfDocuments; i += 1documents// add documents just created to LSH with their idfor let i = 0; i < numberOfDocuments; i += 1lsh// search for a specific document with its id and custom bicketSize// you can also perform a query using a string by passing text instead of id// bucket size are dynamic. feel free to change it to find proper oneconst q =id: 1// text: 'this is a sample text to search for',bucketSize: 6const result = lsh// this will print out documents which are candidates to be similar to the one we are looking forconsole
It's incredibly fast. No matter how many documents you have, time complexity is always constant. Its Dependencies are injected by using @adonisjs/fold which allows us to extend the package freely. There is only one storage and I'll be appreciated if you would implement the other storages such as Redis and submit a pull request.
In this package, MurmurHash, a non-cryptographic hash function which is faster than cryptographic hash function, is used.
You can run the test suite using the following commands:
All contributions are welcome. Please ensure that the tests are passing when submitting a pull request. If you're adding new features to this package, please include tests.
0.1.0 - Initial release, basic functionality and supports memory storage.