kademlia-dht
kademlia-dht is a javascript, network-agnostic implementation of the Distributed Hash Table storage container abstraction, employing the Kademlia algorithms and data structures.
From a local point of view, a DHT is similar to a classic hash table. It
provides two main operations, set
and get
allowing you, respectively,
to store a key/value pair, and to retrieve a value from the key. The
"distributed" aspect comes from the fact the pairs are stored accross a
network of interconnected nodes (eg. over the Internet), making it suitable
to provide information to a large number of users. Typical usage of DHTs
include file-sharing (eg. Bitorrent).
kademlia-dht is implemented with Node.js, but does not depend on system resources, like the network. Instead, the implementation of the network layer (called by Kademlia the Remote Procedure Calls) is left to the user or higher-level libraries. That means this DHT implementation is theorically adaptable to the browser JS without too much hassle.
Note: this implementation is, for now partially complete. It basically lacks two things:
- the handling of time-driven behaviors: key/value pairs expiration, bucket refresh, replication, and pairs republish.
- an implementation of the Rpc usable out-of-the-box (it will be done as a separate library) even though it's possible to use your own already;
Installation
npm install kademlia-dht
Example usage
This example creates two nodes on a mock network, stores a value on one node, then get it on the other node.
'use strict'; var kad = ; // Store a value on one side and get it back on the other side.// { dht1;} // Spawn a node. A node is composed of two elements: the local Dht and the Rpc.// { kadMockRpc;} ;
Output:
boop === boop
MockRpc
is provided for demonstration and tests. It works by simulating a
basic network of interconnected nodes. Every instance of MockRpc
can send
messages to any other instances of the same node.js process. For a real
use case, you must implement your own Rpc or use a higher-level library. The
Rpc interface is explained below.
This example is available as example/demo.js
.
Notes
A Dht cannot ensure a complete consistency of the returned data. The fact new nodes arrive and other exit the network at any time causes variations in the lookup algorithm: all nodes resulting from the lookup are not necessarily up-to-date all the time. You must take into account this variability when processing returned values.
Moreover, a Dht can hardly ensure the authenticity of the values. For instance, a technique known as the Sybil attack allows attackers to cluster around a specific key, accepting any store operation, but then returning a malicious value or no value at all instead of the expected one. This kind of attack is inherent to the structure of such a distributed system, even though some tentative implementations try to counteract via trust-based systems.
The following documents have been the basis of this Kademlia implementation:
- http://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf
- http://xlattice.sourceforge.net/components/protocol/kademlia/specs.html
API
Class: Dht
new Dht(rpc, id)
Create directly the node from an Id. You should never use this directly.
Dht.spawn(rpc, seeds, callback)
rpc
Rpc The RPC object to use to send messages to the network.seeds
Array An array of known endpoints (eg. IPs).callback(err, dht)
Function Called when the Dht is ready.
Create a brand new Dht. It will try to connect to the seeds, and grab knowledge about its nearest nodes. You cannot use a Dht that's not connected to some network, hence the need for seeds.
The Dht takes ownership of the rpc, and it should not be used for anything else in your application.
dht.set(key, value, callback)
key
Stringvalue
Anycallback
Function Called with arguments(err)
once the key/value pair has been replicated onto the network.
dht.get(key, callback)
key
Stringcallback(err, value)
Function Called once the value is found.
If the key cannot be found on the network, this is not an error. The value
will merely be null
.
dht.peek(key)
key
String
Return immediately the associated value if known locally. You can use this method to avoid the callback overhead, but only very few key/value pair are available with this technique.
dht.close()
Close the Dht and the inner node. No other operation is then possible.
Class: Rpc
This describe the excepted interface for the rpc
object provided to
Dht.spawn()
.
rpc.send(message, endpoint, payload, cb)
message
String Type of message, always one of: "ping", "store", "findNode" or "findValue".endpoint
Any A value representing the 'endpoint' of the node. Generally it will be a pair{ipAddress, port}
, when the Rpc is implemented over TCP or UDP.payload
Object An object containing information attached to the message.cb(err, result)
Function To be called once the message got a reply.
This function must implement a timeout mechanism, and callback with an error if so happens. This ensures the Dht does not hang up on a request and properly take into account unresponsive nodes.
rpc.receive(message, handler)
message
String Type of message to handle.handler(payload)
Function Function locally handling the specified message.
Register a local handler.
The receive handler must be a synchronous operation. The handling function returns a result object that must be forwarded to the node initiating the request. The handler may throw, in which case an error should be forwarded to the remote node; make sure, though, that no local information leaks such as stacktraces, because they can contain personal or sensitive information.
rpc.close()
Close the Rpc endpoint and prevent any further call to message handlers.