AnyFetch provider
NodeJS toolkit for creating anyFetch providers.
Introduction
If you want to add a new service to AnyFetch (as a document entry point), you should use this tiny toolkit.
This let you bridge a given service to the anyFetch api by mounting a server receiving calls from both sides (the service and AnyFetch API).
Use Provider boilerplate to generate a new project stub.
You will need a redis and a mongodb server running.
For a simple and practical example you can take a look at the linkedin provider.
Installation and documentation
npm install anyfetch-provider
Here is a sample usage:
// See syntax belowvar server = AnyFetchProvider; server;
Parameters
Too lazy to read the doc? Why not check out real use-case from open-source code! For a simple use case, take a look on this file from the Google Contacts provider. For more advanced use-case with file upload, see Dropbox provider.
connectFunctions
The first parameter to pass to AnyFetchProvider.createServer()
is a set of functions handling the OAuth part on the provider side (the OAuth flow on Anyfetch side is automatically handled for you by the lib).
var connectFunctions = // IN : // * callback url to ping after grant // OUT : // * err // * url to redirect to // * data to store (if any) { serviceLib; } // IN : // * GET params from the incoming request (probably the result from a consent screen), // * Params returned by previous function redirectToService // OUT : // * err // * account name, current account identifier on the service (email address from the user for instance) // * service data to permanently store { serviceLib; };
The first function, redirectToService
will be invoked when a user asks to connect his AnyFetch account with your provider. It is responsible for providing an URL to your provider consent page.
It takes as parameter a callbackUrl
(of the form https://your.provider.address/init/callback
). It is up to you to ensure the user is redirected to this URL after giving consent on the redirectUrl
page.
The second parameter of the cb
can be data that needs to be stored. The lib will reinject this data on its next call to retrieveTokens
(in storedParams
).
This behavior can be useful when you need to store
code
orrequestTokens
. In most of the case however, you're safe to leave this empty.
The second function, retrieveTokens
, will be invoked when the user has given his permission.
It takes as parameter a reqParams
object, which contain all GET params sent to the previous callbackUrl
. It will also get access in storedParams
to data you sent to the callback in redirectToService
.
You're responsible for invoking the cb
with any error, the account name from the user and all the final data you wish to store internally—in most case, this will include at least a refresh token, but this can be anything as long as it's a valid JavasScript object.
Warning: Final data can't contain a documentsPerUpdate property. it would be overwritten by the documents_per_update HTTP parameter.
updateAccount
This function (which must be directly exported in the file sent to AnyFetchProvider.createServer()
) will be invoked whenever the user asks to update his account with new data from your provider.
In order to do so, a cursor
parameter is sent—you'll return it at the end of the function, updated, to match the new state (either an internal cursor sent from your provider, or the current date, whichever suits you).
// IN :// * serviceData returned by retrieveTokens// * last cursor returned by this function, or null on the first update// * Queues to use// OUT :// * err// * new cursor// * new serviceData to replace previous ones (if any)module { serviceLib;};
It takes as parameter the data you sent to retrieveTokens
, the cursor
that was sent during the last invokation of updateAccount
or null
if this is the first run.
You can then start pushing tasks onto the different queues—more on that on next section.
To emulate the update call, you can curl on localhost:8000 this way:
http://localhost:8000/update?access_token=ANYFETCH_ACCESS_TOKEN&documents_per_update=NUMBER_OF_DOCUMENTS&force=1
workers
Workers (exported in the file sent to AnyFetchProvider.createServer()
) are functions responsible for handling the tasks returned by updateAccount
. Keep in mind they are shared for all users of your lib, and should therefore not rely on any external state or context.
workers
must be an object where each key is a specific worker with specific options. For nearly all use-cases, you'll only need two workers: one for additions (sending new and updated documents from your provider to AnyFetch) and one for deletions (deleting documents onto AnyFetch).
A worker is a simple function taking two parameters: a job, and a cb
to invoke with an error if any.
// IN :// * job, with preconfigured keys:// * task: data to process// * anyfetchClient: pre-configured client, see https://github.com/AnyFetch/anyfetch.js// * serviceData: as returned by retrieveTokens, or updated by updateAccount (third optional parameter for cb)// * cache: a LRU cache, see https://github.com/isaacs/node-lru-cache// OUT :// * errvar workers = { jobanyfetchClient; } { jobanyfetchClient; }; moduleexports = workers;
The job
parameter contains 4 keys:
task
: the task sent by yourupdateAccount
function,anyfetchClient
: a pre-configured anyfetch client, with simple functions to send documents and files to the API.serviceData
: the data you've registered for this access token.cache
: a pre-configured LRU cache
cb
does not take any additional params after the error.
Note: Those workers are internally called as OS subprocesses by the lib, to ensure the failure of one worker does not bring down the whole server; it also avoids memory leaks from your code, since the workers are "cleaned" between each user.
config
The last parameter to AnyFetchProvider.createServer()
is an object containing your application keys. You can find them on the AnyFetch manager.
var config = // Anyfetch app id appId: "your_app_id" // Anyfetch app secret appSecret: "your_app_secret" // Server currently running providerUrl: "https://your.provider.address/";
Faster?
You can set the concurrency—the number of parallel tasks per user—that will be running to unstack all tasks. Default is 1, but you can increase this value using the config.concurrency
property:
// Set concurrency. Defaults to 1 when unspecified.configconcurrency = 10;
You can also set the number of users processing in parallel by setting the config.usersConcurrency
property.
Note: as stated before, workers are doing their job in subprocesses. High concurrency means many small process running in parallel. The total number of tasks running in parallel will be
concurrency × usersConcurrency
.
Going further...
Adding endpoints
AnyFetchProvider.createServer()
returns a restify server. This is very similar to express, and you can simply add endpoints; for instance:
var server = AnyFetchProvider; server; server;
Configuring Mongo and Redis
By default, the lib will read values from process.env.MONGO_URL
and process.env.REDIS_URL
to connect to external services. You can override this behavior using config.redisUrl
and config.mongoUrl
.