Wikidata JSKOS
This node module provides a web service, a command line client, and a library to access Wikidata in JSKOS format. The data includes Wikidata items as concepts and concept schemes (read) and mappings between Wikidata and other authority files (read and write).
Table of Contents
Background
Wikidata is a large knowledge base with detailed information about all kinds of entities. Mapping its data model to JSKOS data format allows simplified reuse of Wikidata as authority file. This implementation is used in the Cocoda web application but it can also be used independently.
The mapping between Wikidata and JSKOS format includes:
- Wikidata items expressed as authority records (JSKOS Concepts)
- Selected Wikidata items covering information about authority files (JSKOS Concept Schemes)
- Selected Wikidata statements linking Wikidata to other authority files (JSKOS Mappings)
In addition a search service is provided for selecting a Wikidata item with typeahead.
Editing Wikidata mapping statements to other authority files requires authentification via OAuth. The following authority files have been tested succesfully:
- Basisklassifikation (BK)
- Regensburg Classification (RVK)
- Integrated Authority File (GND)
- Nomisma
- Iconclass
Other systems (not including DDC) may also work but they have not been converted to JSKOS yet, so they are not provided for browsing in Cocoda.
Install
Clone and Install
git clone https://github.com/gbv/wikidata-jskos.gitcd wikidata-jskosnpm install
Optionally make the command line tool wdjskos
available:
npm link
Configuration
You can customize the application settings via a configuration file, e.g. by providing a generic config.json
file and/or a more specific config.{env}.json
file (where {env}
is the environment like development
or production
). The latter will have precendent over the former, and all missing keys will be defaulted with values from config.default.json
.
Please consult config.default.json
for possible configuration options. Some notes:
- To use a custom Wikibase instance, you can set the subkeys of the
wikibase
property. Bothinstance
andsparqlEnpoint
are necessary. By default, Wikidata is used. - wikidata-jskos supports saving, editing, and deleting mappings in Wikidata. To enable this, you will need to provide
auth.algorithm
andauth.key
(algorithm and key to decode the JWT), as well asoauth.consumer_key
andoauth.consumer_secret
(for your registered OAuth consumer). - Please provide a
baseUrl
when used in production. If no baseUrl is provided,http://localhost:${port}/
will be used.
Usage
Run Server
For development serve with hot reload and auto reconnect at http://localhost:2013/:
npm run start
Deployment
For deployment there is a config file to use with pm2:
pm2 start ecosystem.config.json
Web Service
An instance is available at https://coli-conc.gbv.de/services/wikidata/. The service provides selected endpoints of JSKOS API.
Authentication
The following endpoints require an authenticated user:
Authentication works via a JWT (JSON Web Token). The JWT has to be provided as a Bearer token in the authentication header, e.g. Authentication: Bearer <token>
. It is integrated with login-server and the JWT is required to have the same format as the one login-server provides. Specifically, the OAuth token and secret for the user need to be provided as follows:
There are more properties in the JWT, but those are not used by wikidata-jskos. Note that the JWT needs to be signed with the respective private key for the public key provided in the configuration. Also, the OAuth user token and secret need to come from the same OAuth consumer provided in the config.
GET /status
Returns a JSKOS API status object. See JSKOS Server for details.
GET /concept
Look up Wikidata items as JSKOS Concepts by their entity URI or QID.
-
URL Params
uri=[uri]
URIs for concepts separated by|
.language
orlanguages
: comma separated list of language codes. -
Success Response
JSON array of JSKOS Concepts
Only some Wikidata properties are mapped to JSKOS fields. The result also
contains broader
links determined by an additional SPARQL request.
GET /mappings
Look up Wikidata mapping statements as JSKOS Concept Mappings between
Wikidata items (query parameter from
) and external identifiers (query
parameter to
). At least one of both parameters must be given.
-
URL Params
from=[uri|notation]
specify the source URI or notationto=[uri|notation]
specify the target URI or notationfromScheme=[uri|notation]
only show mappings from this concept scheme (URI or notation)toScheme=[uri|notation]
only show mappings to this concept scheme (URI or notation)language
orlanguages
enables inclusion of entity labels. A comma separated list of language codes is used as preference list.mode=[mode]
specify the mode forfrom
,to
, one ofand
(default) andor
direction=forward|backward|both
searches mappings fromfrom
toto
(default), reverse, or in both directionslimit=[number]
maximum number of mappings to return (not fully implemented)offset=[number]
start number of mappings to return (not fully implemented)
Concept Schemes are identified by BARTOC IDs (e.g. http://bartoc.org/en/node/430`).
-
Success Response
JSON array of JSKOS Concept Mappings
-
Examples
?from=http://www.wikidata.org/entity/Q42
?to=http://d-nb.info/gnd/119033364
Mapping relation types (P4390) are respected, if given, see for example mapping from Wikidata to http://d-nb.info/gnd/7527800-5.
GET /mappings/voc
Look up Wikidata items with Wikidata properties for authority control as JSKOS Concept Schemes with used for mappings. These schemes need to have a BARTOC-ID (P2689), and be subject item (P1629) of an external identifier property with statements P1921 and P1793 (regular expression).
-
URL Params
None.
-
Success Response
JSON array of JSKOS Concept Schemes
GET /mappings/:_id
Returns a specific mapping for a Wikidata claim/statement.
-
Success Response
JSKOS object for mapping.
-
Error Response
If no claim with
_id
could be found, it will return a 404 not found error. -
Sample Call
curl https://coli-conc.gbv.de/services/wikidata/mappings/Q11351-9968E140-6CA7-448D-BF0C-D8ED5A9F4598
POST /mappings
Saves a mapping in Wikidata. Requires authentication.
Note that if an existing mapping in Wikidata is found with the exact same members, that mapping will be overwritten by this request.
-
Success Reponse
JSKOS Mapping object as it was saved in Wikidata.
PUT /mappings/:_id
Overwrites a mapping in Wikidata. Requires authentication.
-
Success Reponse
JSKOS Mapping object as it was saved in Wikidata.
DELETE /mappings/:_id
Deletes a mapping from Wikidata. Requires authentication.
-
Success Reponse
Status 204, no content.
GET /suggest
OpenSearch Suggest endpoint for typeahead search.
Command line tool
The command line client wdjskos
provides roughly the same commands as accessible via
the web service.
Mapping schemes are cached in the caching directory of wikidata-cli.
wdjskos concept
Look up Wikidata items as JSKOS Concepts.
wdjskos concept Q42
wdjskos mappings
Look up JSKOS Concept Mappings.
wdjskos mappings Q42 | jq .to.memberSet[].uri
wdjskos mappings - http://viaf.org/viaf/113230702
A single hyphen (-
) can be used to nullify argument from
or to
,
respectively. Mappings can be limited to a target scheme. These are equivalent:
wdjskos --scheme P227 mappings Q42
wdjskos --scheme 430 mappings Q42
wdjskos --scheme http://bartoc.org/en/node/430 mappings Q42
wdjskos schemes
Look up JSKOS Concept Schemes with Wikidata properties for authority control.
wdjskos find
Search a Wikidata item by its names and return OpenSearch Suggestions response.
wdjskos mapping-item
Convert a JSKOS mapping to a Wikidata item.
wdjskos mapping-item mapping.json
wdjskos --simplfiy mapping-item mapping.json
API
The node library can be used to convert Wikidata JSON format to JSKOS (mapEntity) and to convert JSKOS mappings to Wikidata JSON format (mapMapping).
mapEntity
jskos = wds
Entity data can be retrieved via Wikidata API method wbgetentities and from Wikidata database dumps. See JavaScript libraries wikidata-sdk and wikidata-filter for easy access and processing.
Map selected parts of a Wikidata entity
All methods return a JSKOS item.
jskos = wds// { uri: "http://www.wikidata.org/entity/...", notation: [ "..." ] } jskos = wds// { prefLabel: { ... } } jskos = wds// { altLabel: { ... } } jskos = wds// { scopeNote: { ... } } jskos = wds// { occurrences: [ { ... } ], subjectOf: [ { url: ... }, ... ] } jskos = wds// ... // convert claims with mapping propertiesjskos = wds jskos = wds// ...
Map simplified Wikidata entities
Each method has a counterpart to map simplified Wikidata entities.
jskos = wdsjskos = wdsjskos = wds...
mapMapping
Convert a JSKOS mapping into a Wikidata claim. Only respects JSKOS fields
from
, to
, uri
, and type
(if given) and only supports 1-to-1 mappings
from a single Wikidata item to a concept in another concept scheme.
this is work in progress!
Contributing
See https://github.com/gbv/wikidata-jskos.
For debugging set environment variable DEBUG
, also possible via file .env
:
DEBUG=wds:*
See lib/utils/debug.js
for details.