Create providers for Anyfetch.
NodeJS toolkit for creating anyFetch providers.
If you want to add a new service to AnyFetch (as a document entry point), you should use this tiny toolkit.
This let you bridge a given service to the anyFetch api by mounting a server receiving calls from both sides (the service and AnyFetch API).
Use Provider boilerplate to generate a new project stub.
You will need a redis and a mongodb server running.
For a simple and practical example you can take a look at the linkedin provider.
npm install anyfetch-provider
Here is a sample usage:
// See syntax belowvar server = AnyFetchProvidercreateServerconnectFunctions __dirname + '/lib/workers.js' __dirname + '/lib/update.js' config;serverlisten;
Too lazy to read the doc? Why not check out real use-case from open-source code! For a simple use case, take a look on this file from the Google Contacts provider. For more advanced use-case with file upload, see Dropbox provider.
The first parameter to pass to
AnyFetchProvider.createServer() is a set of functions handling the OAuth part on the provider side (the OAuth flow on Anyfetch side is automatically handled for you by the lib).
var connectFunctions =// IN :// * callback url to ping after grant// OUT :// * err// * url to redirect to// * data to store (if any)redirectToService:serviceLibgenerateRedirectUrlredirectUrl += "&callback=" + encodeURIcallbackUrl;var dataToStore =foo: 'bar';cbnull redirectUrl dataToStore;;// IN :// * GET params from the incoming request (probably the result from a consent screen),// * Params returned by previous function redirectToService// OUT :// * err// * account name, current account identifier on the service (email address from the user for instance)// * service data to permanently storeretrieveTokens:serviceLibgenerateAccessTokenreqParamscodeserviceLibretrieveUserEmailaccessTokencbnull userEmailaccessToken: accessTokenaccount: storedParamsaccount;;;;
The first function,
redirectToService will be invoked when a user asks to connect his AnyFetch account with your provider. It is responsible for providing an URL to your provider consent page.
It takes as parameter a
callbackUrl (of the form
https://your.provider.address/init/callback). It is up to you to ensure the user is redirected to this URL after giving consent on the
The second parameter of the
cb can be data that needs to be stored. The lib will reinject this data on its next call to
This behavior can be useful when you need to store
requestTokens. In most of the case however, you're safe to leave this empty.
The second function,
retrieveTokens, will be invoked when the user has given his permission.
It takes as parameter a
reqParams object, which contain all GET params sent to the previous
callbackUrl. It will also get access in
storedParams to data you sent to the callback in
You're responsible for invoking the
cb with any error, the account name from the user and all the final data you wish to store internally—in most case, this will include at least a refresh token, but this can be anything as long as it's a valid JavasScript object.
Warning: Final data can't contain a documentsPerUpdate property. it would be overwritten by the documents_per_update HTTP parameter.
This function (which must be directly exported in the file sent to
AnyFetchProvider.createServer()) will be invoked whenever the user asks to update his account with new data from your provider.
In order to do so, a
cursor parameter is sent—you'll return it at the end of the function, updated, to match the new state (either an internal cursor sent from your provider, or the current date, whichever suits you).
// IN :// * serviceData returned by retrieveTokens// * last cursor returned by this function, or null on the first update// * Queues to use// OUT :// * err// * new cursor// * new serviceData to replace previous ones (if any)moduleexports =serviceLibretrieveDeltacursor// Push new tasks onto the workerscreatedFilesforEachqueuesadditionspushtask;;deletedFilesforEachqueuesdeletionspushtask;;// Save new cursorcbnull toString;;;
It takes as parameter the data you sent to
cursor that was sent during the last invokation of
null if this is the first run.
You can then start pushing tasks onto the different queues—more on that on next section.
To emulate the update call, you can curl on localhost:8000 this way:
Workers (exported in the file sent to
AnyFetchProvider.createServer()) are functions responsible for handling the tasks returned by
updateAccount. Keep in mind they are shared for all users of your lib, and should therefore not rely on any external state or context.
workers must be an object where each key is a specific worker with specific options. For nearly all use-cases, you'll only need two workers: one for additions (sending new and updated documents from your provider to AnyFetch) and one for deletions (deleting documents onto AnyFetch).
A worker is a simple function taking two parameters: a job, and a
cb to invoke with an error if any.
// IN :// * job, with preconfigured keys:// * task: data to process// * anyfetchClient: pre-configured client, see// * serviceData: as returned by retrieveTokens, or updated by updateAccount (third optional parameter for cb)// * cache: a LRU cache, see// OUT :// * errvar workers =jobanyfetchClientpostDocumentjobtask cb;jobanyfetchClientdeleteDocumentByIdjobtaskid cb;;moduleexports = workers;
job parameter contains 4 keys:
task: the task sent by your
anyfetchClient: a pre-configured anyfetch client, with simple functions to send documents and files to the API.
serviceData: the data you've registered for this access token.
cache: a pre-configured LRU cache
cb does not take any additional params after the error.
Note: Those workers are internally called as OS subprocesses by the lib, to ensure the failure of one worker does not bring down the whole server; it also avoids memory leaks from your code, since the workers are "cleaned" between each user.
The last parameter to
AnyFetchProvider.createServer() is an object containing your application keys. You can find them on the AnyFetch manager.
var config =// Anyfetch app idappId: "your_app_id"// Anyfetch app secretappSecret: "your_app_secret"// Server currently runningproviderUrl: "";
You can set the concurrency—the number of parallel tasks per user—that will be running to unstack all tasks. Default is 1, but you can increase this value using the
// Set concurrency. Defaults to 1 when unspecified.configconcurrency = 10;
You can also set the number of users processing in parallel by setting the
Note: as stated before, workers are doing their job in subprocesses. High concurrency means many small process running in parallel. The total number of tasks running in parallel will be
concurrency × usersConcurrency.
AnyFetchProvider.createServer() returns a restify server. This is very similar to express, and you can simply add endpoints; for instance:
var server = AnyFetchProvidercreateServerconnectFunctions updateAccount workers config;serverget'/hello'ressend"Hello " + reqparamsname;next;;serverlisten;
By default, the lib will read values from
process.env.REDIS_URL to connect to external services. You can override this behavior using