Mines the npm registry for user and package data used by valuepack.
Supports valuepack, the community driven rating system for nodejs modules on npm in order to help in selecting the right one.
Read more about its goals.
var mineNpm = ;;
The entire public API is exposed via the index file:
exportsstoreNpmPackages = ;exportsstoreNpmUsers = ;exportsstreamNpmPackages = ;exportsstreamNpmUsers = ;
You can play with scripts inside
The following environment variables are considered by the scripts:
VALUEPACK_DATAthe directory in which json data fetched from the npm registry is stored (defaults to
VALUEPACK_MINE_DBthe path at which the leveldb data is stored (defaults to
Make sure to include them every time you execute a script or add the following to your
export VALUEPACK_DATA=~/.valuepack/dataexport VALUEPACK_MINE_DB=~/.valuepack/valuepack-mine.db
An init script is provided that fetches all needed data from npm and stores it as json and in a leveldb database at
Do one of the following:
npm run init
cd scripts && ./init.sh
# fetch users./fetch-npm-users.js# fetch packages./fetch-npm-packages.js
In order to store all data in leveldb, please run:
Note: rebuilding the database from local json files takes about 2mins on a MacBookAir.
store-* scripts take parameters.
--readread out values of the store instead of adding them
--keyspull out and print keys only
--valuespull out and print values only
Therfore if you want to query data you should always pass the
--ownerslist owner indexes instead of packages
--keywordlist keyword indexes instead of packages
--githublist github login indexes instead of users
# get package keyword count./store-npm-packages.js --read --keyword --keys | wc -l# get package count./store-npm-packages.js --read --keys | wc -l# query how many users did provide a github account./store-npm-users.js --read --github --keys | wc -l# query how many packages dominictarr published./store-npm-packages.js --read --owner --keys | grep dominictarr | wc -l