@contentstack/datasync-manager

Contentstack is a headless CMS with an API-first approach. It is a CMS that developers can use to build powerful cross-platform applications in their favorite languages. Build your application frontend, and Contentstack will take care of the rest. Read More.

Contentstack DataSync Manager

Contentstack DataSync lets you sync your Contentstack data with your database, enabling you to save data locally and serve content directly from your database. It is a combination of four powerful modules that is DataSync Webhook Listener, DataSync Manager, DataSync Asset Store Filesystem, DataSync Content Store — Filesystem and MongoDB.

The DataSync Manager is one of the four important components of Contentstack DataSync. When any publish, unpublish, or delete operations are performed on assets or content, the DataSync Manager fetches the data and sends it to Content Store. It uses Contentstack's Sync APIs to sync data from Contentstack with your preferred database — Filesystem and MongoDB in our case.

Prerequisite

nodejs v8+

Usage

This is how DataSync Manager works with DataSync boilerplate:

const assetStore = require('@contentstack/datasync-asset-store-filesystem')
const contentStore = require('@contentstack/datasync-content-store-filesystem')
const listener = require('@contentstack/webhook-listener')
const syncManager = require('@contentstack/datasync-manager') // <--
const config = require('./config')

// Set asset, content store, listener and config to Datasync Manager
syncManager.setAssetStore(assetStore)
syncManager.setContentStore(contentStore)
syncManager.setListener(listener)
syncManager.setConfig(config)

// start DataSync manager
syncManager.start()
  .then(() => {
    console.log('Contentstack sync started successfully!')
  })
  .catch(console.error)

You can replace @contentstack/datasync-content-store-filesystem used above, with @contentstack/datasync-content-store-mongodb and switch content store databases.

Configuration

Here's a list of contentstack's configuration keys

Key Name	Default	Description
apiKey		Required. Your stack's API key
deliveryToken		Required. Your environment's delivery token
sync_token		Token from where you'd like to start the process
pagination_token		Token from where you'd like to start the process
MAX_RETRY_LIMIT	6	Number of times the API call would retry, if the server fails

Here's a list of configuration keys for contentstack datasync-manager:

Property	Default	Description
cooloff	3000	Number of `ms` the app would wait before making a subsequent `Sync API` request
enableAssetReferences	true	This is not implemented, you can help: #33
enableContentReferences	true	This is not implemented, you can help: #34
limit	100	Number of items fetched in each `Sync API` request made.
maxsize	2097152	Maximum file size of files (ex: .ledger file)
queue.pause_threshold	10000	The min-max internal queue size of contentstack-sync-manager. The DataSync manager would pause making `Sync API` requests once the internal queue size exceeds `pause_threshold` count.
queue.resume_threshold	5000	The min-max internal queue size of contentstack-sync-manager. The DataSync manager will resume `Sync API` requests once the queue size goes below `resume_threshold` count.
saveFilteredItems	true	When enabled, the app will log items that have been filtered out. Items can be filtered out due to the following: Missing required keys Language filters applied via config
filters	content_type_uid locale action	Filters allow you to narrow down the items that'd be returned by the SYNC API response. Ex: `{filters: {content_type_uid: ['blogs']}, locale: ['en-us'], action: ['publish']}`

And here's an example to get you started:

{
  // exmaple to override default values
  contentstack: {
      apiKey: '',
      deliveryToken: '',
      sync_token: '',
      MAX_RETRY_LIMIT: 5
    }
  },
  syncManager: {
      cooloff: 2000,
      limit: 80,
      filters: {
        content_type_uid: ['blogs'],
        locale: ['es-es', 'en-us'],
        action: ['publish', 'unpublish']
      }
    }
  }
}

Support and Feature requests

If you have any issues working with the library, please file an issue here at Github. You can send us an e-mail at support@contentstack.com if you have any support or feature requests. Our support team is available 24/7 on the intercom. You can always get in touch and give us an opportunity to serve you better!

Licence

This repository is published under the MIT license.

@contentstack/datasync-manager