Wondering what’s next for npm?Check out our public roadmap! »

    wikibase-dump-filter

    5.0.5 • Public • Published

    wikibase-dump-filter

    Filter and format a newline-delimited JSON stream of Wikibase entities.

    Typically useful to create a formatted subset of a Wikibase JSON dump.

    Some context: This tool was formerly known as wikidata-filter. Wikidata is an instance of Wikibase. This tool was primarly designed with Wikidata in mind, but should be usable for any Wikibase instance.

    This project received a Wikimedia Project Grant.


    wikibase           wikidata

    License Node JavaScript Style Guide

    NPM

    Summary

    Install

    this tool requires to have NodeJs installed.

    # Install globally
    npm install -g wikibase-dump-filter
    # Or install just to be used in the scripts of the current project
    npm install wikibase-dump-filter

    Changelog

    See CHANGELOG.md for version info

    Download dump

    Wikidata dumps

    Wikidata provides a bunch of database dumps, among which the desired JSON dump. As a Wikidata dump is a very laaarge file (September 2020: 55GB compressed), it is recommended to download that file first before doing operations on it, so that if anything crashes, you don't have to start the download from zero (the download time being usually the bottleneck).

    wget -C https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.bz2
    cat latest-all.json.bz2 | bzcat | wikibase-dump-filter --claim P31:Q5 > humans.ndjson

    Your own Wikibase instance dump

    You can generate a JSON dump using the script dumpJson.php. If you are running Wikibase with wikibase-docker, you could use the following command:

    cd wikibase-docker
    docker-compose exec wikibase /bin/sh -c "php ./extensions/Wikibase/repo/maintenance/dumpJson.php --log /dev/null" > dump.json
    cat dump.json | wikibase-dump-filter --claim P1:Q1 > entities_with_claim_P1_Q1.ndjson

    How-to

    This package can both be used as a command-line tool (CLI) and as a NodeJS module. Those 2 uses have there own documentation page but the options stay the same, and are documented in the CLI section

    Donate

    We are developing and maintaining tools to work with Wikidata from NodeJS, the browser, or simply the command line, with quality and ease of use at heart. Any donation will be interpreted as a "please keep going, your work is very much needed and awesome. PS: love". Donate

    See Also


    You may also like

    inventaire banner

    Do you know Inventaire? It's a web app to share books with your friends, built on top of Wikidata! And its libre software too.

    License

    MIT

    Install

    npm i wikibase-dump-filter

    DownloadsWeekly Downloads

    4

    Version

    5.0.5

    License

    MIT

    Unpacked Size

    46.1 kB

    Total Files

    35

    Last publish

    Collaborators

    • avatar