Neatly Packaged Magic

    pairio

    0.3.3 • Public • Published

    pairio

    Pairio is a system for associating hashes of inputs (keys) with hashes of outputs (values) so that results of processing pipelines may be cached both locally and on the cloud. Essentially it is a key/value store where the keys and values are meant to be 40-character SHA-1 hashes of objects or files.

    Installation of the python client

    pip install pairio
    

    or to install from source clone the repository and then run:

    python setup.py install
    

    or for a development installation, clone the repository and then run:

    python setup.py develop
    

    Basic usage on local machine

    To use pairio on the local computer, simply run

    from pairio import client as pa
    key='some-string-40-or-fewer-characters'
    val='another-string-40-or-fewer-characters'
    pa.set(key,val)
    

    Then later retrieve the value

    val=pa.get(key)
    print(val)
    

    Often the key will not be a string, but will instead be a python dict. For example

    pa.set({'operation':'add','arg1':12,'arg2':30},'42')
    

    In this case pairio will use the hash of the JSON string of the key to store the value. Thus the result of this operation could be retrieved later by passing in the same dict key:

    result=pa.get({'operation':'add','arg1':12,'arg2':30})
    print(result)
    

    On primary use case is when the value is the SHA-1 hash of a file. For example

    pa.set({'operation':'bash-stdout','script':'echo "Hello, pairio."'},'a09fd13fd92800aeafe475e9113efe216788d934')
    

    Here, the value is the SHA-1 hash of the file whose contents is "Hello, pairio."

    Looking up values in the cloud

    Values can be retrieved from the cloud by configuring pairio to look for pairs stored under pairio user accounts. For example, to retrieve the value associated with the dict {'name':'test'} in the 'magland' cloud collection, do

    val=pa.get({'name':'test'},user='magland',local=False)
    print(val)
    

    If found, you might see the "a-test-string", depending on what is actually in the database. If not found, you will see None.

    This can also be achieved by configuring pairio to always look for key/value pairs under the 'magland' cloud collection

    pa.setConfig(users=['magland'])
    val=pa.get({'name':'test'},local=False)
    print(val)
    

    In general, pairio will first look on the local machine. If they key is not found, it will then look in the cloud under the collections owned by each of the users specified in the pa.setConfig(users=users) command. If still not found, it will return None.

    Storing pairs in the cloud

    In order to store key/value pairs in the cloud, you must have a pairio account for some pairio server. Contact the authors if you would like an account on the default pairio server at pairio.org. Once you have a user name and token, you can configure pairio to save key/value pairs in your cloud collection as follows:

    pa.setConfig(user='your-user-name',token='your-pairio-token')
    

    Alternatively, you can set the PAIRIO_USER and PAIRIO_TOKEN environment variables.

    Then subsequent calls to pa.set() will set the key/value pairs to both the local machine and to the remote cloud collection associated with your user name.

    Configuration options

    You can specify the default behavior of the pa.get() and pa.set() operations as follows:

    pa.setConfig(
      users=['list','of','user','names'], # Names of user cloud collections to search when getting values
      user='user-name', # The user name for setting key/value pairs in the cloud
      token='user-token', # The token corresponding to the user name
      local=True, # Whether or not to store key/value pairs locally when set() is called
      remote=True, # Whether or not to store key/value pairs remotely when set() is called
      url='url-to-pairio-server' # Specify a remote pairio server. By default it will point to the server at pairio.org
    )
    

    Hosting a pairio server

    You can host your own pairio server. The source code for the NodeJS server is found in the pairioserver/ directory.

    TODO: finish this section.

    See server API docs at the top of this file: https://github.com/magland/pairio/blob/master/pairioserver/pairioserver.js

    Keywords

    none

    Install

    npm i pairio

    DownloadsWeekly Downloads

    14

    Version

    0.3.3

    License

    Apache-2.0

    Unpacked Size

    62.5 kB

    Total Files

    15

    Last publish

    Collaborators

    • magland