LoadingCache

LoadingCache is a generic utility that simplifies data access with a consistent API for any data source, such as databases or services, reducing requests to those sources by automatically reading-through data into caches with a pluggable implementation.

When the LoadingCache receives a request for some data, it first checks for the requested keys in its cache(s). Only the data missed in the cache(s) is loaded from the data source, and is then merged with the cached data to produce the final result. In the background, each cache is populated with the resulting data it missed, which will be accessed in subsequent requests to reduce over time the volume of data needing to be loaded from the data source.

Having all this handled behind the scenes in a consistent way means that your data-accessing layer can be much cleaner and clearer, while being backed by a robust and optimized read-through multi-level cache implementation.

Getting Started

First install LoadingCache using npm.

npm install loading-cache

To get started, create a LoadingCache. Each LoadingCache instance is backed by one or many cache stores which are setup and managed by the application.

When used with a web-server like express, instances can be created per request with the same cache store(s), or a single instance can be shared across requests.

Importing

import { LoadingCache } from 'loading-cache'
// or
const { LoadingCache } = require('loading-cache')

Usage

Create a LoadingCache by providing a loader function as well as a cache object or an array of cache objects (and optionally a third options object).

import { LoadingCache } from 'loading-cache'

const loadingCache = new LoadingCache(
  async (keys) => await myFetchAll(keys), // define a way to retrieve data, e.g. fetching from a data source over the network
  // instruct how to interact with the cache store (which could be an in-memory data structure, a Redis client, etc.)
  {
    // either by implementing these functions...
    getFromStore: (keys) => {
      /* get the values for each key from the store */
    },
    putInStore: (keys, values) => {
      /* put each key-value pair in the store */
    },
  }, // ...or using one of the built-in adapters (continue reading below)
)

// handle errors from caching the loaded values in the background
loadingCache.on('error', (e) => console.log('Loading Cache Error', e))

Then get some values from the LoadingCache. It will first check for the requested keys in the provided cache(s) before calling your loader function with only those missing keys. After the get() or getMany() methods are called once with a given key, the resulting value is cached to eliminate redundant loads.

In the example below, key 'A' is loaded then cached after the call to get(). Later on when the keys 'A' and 'B' are requested, only 'B' is loaded while 'A' is retrieved directly from cache.

const valueForA = await loadingCache.get('A')

// later on
const [valueForA, valueForB] = await loadingCache.getMany(['A', 'B'])

The get() and getMany() methods are effectively memoized functions storing in the provided cache(s) the values loaded for keys requested by your application.

Loader function

A loader function accepts an Array of keys and returns a Promise which resolves to an Array of values.

There are a few constraints this function must uphold:

• it must return a Promise (thenable)
• the Array of values must be the same length as the Array of keys
• each index in the Array of values must correspond to the same index in the Array of keys

Cache object

A cache object has two properties; getFromStore and putInStore, which are methods that enable the LoadingCache to carry out the necessary cache operations against the underlying cache store(s) it uses. These methods must implement the interaction with whichever cache store implementation you intend to use with the LoadingCache, for example a simple Map data-structure, or a Redis client.

The signature for these methods as well as applicable guarantees and constraints are documented here.

LoadingCache instances within an application can therefore be used with any cache store implementations, by adapting the store as a cache object with its methods implementing the cache operations in whichever way is desired.

The following sections shows usage with some common cache store implementations using the functions exported from different submodules under the loading-cache/adapters path.

Sections:

• redis
• ioredis
• Map-like stores
• cache-manager multi-key store engines
• Other stores

redis

Using a Redis client created with the redis npm package.

import { redisAdapter } from 'loading-cache/adapters/redis'

import { createClient } from 'redis'

// set up a Redis client
const redisClient = createClient()
await redisClient.connect()

new LoadingCache(
  myFetchAll,
  redisAdapter(redisClient), // translates cache operations into `mGet()` and `mSet()` calls on the Redis client
)

The redisAdapter() function accepts options to customize serialization/deserialization from Redis. See the reference.

ioredis

Using a Redis client created with the ioredis npm package.

import { ioredisAdapter } from 'loading-cache/adapters/ioredis'

import { Redis } from 'ioredis'

// setup a Redis client
const redis = new Redis();

new LoadingCache(
  myFetchAll,
  ioredisAdapter(redis), // translates cache operations into `mget()` and `mset()` calls on the Redis client
)

The redisAdapter() function accepts options to customize serialization/deserialization from Redis. See the reference.

Map-like stores

With the mapAdapter(), you can use a custom cache object with whatever behavior you prefer, so long as it implements the methods get() and set() of the Map API.

import { mapAdapter } from 'loading-cache/adapters/map'

new LoadingCache(
  myFetchAll,
  mapAdapter(new Map()), // translates cache operations into multiple `get()` and `set()` calls on the map
)

The example below uses an LRU (least recently used) cache to limit total memory to hold at most 100 cached values via the lru_map npm package.

import { mapAdapter } from 'loading-cache/adapters/map'

import { LRUMap } from 'lru_map'

new LoadingCache(
  myFetchAll,
  mapAdapter(new LRUMap(100)),
)

This last example uses an in-memory implementation to limit the number of keys, and also set a 10s TTL (time to live) on the cache entries via the cache-manager npm package.

import { mapAdapter } from 'loading-cache/adapters/map'

import { caching } from 'cache-manager'

// set up the memory cache
const memoryCache = await caching(
  'memory',
  {
    max: 100,
    ttl: 10_000, // ms
  },
)

new LoadingCache(
  myFetchAll,
  mapAdapter(memoryCache.store),
)

Note Some cache-manager implementations provide multi-key operations, which can be leveraged with a different store adapter, see the next section.

cache-manager multi-key store engines

The base cache-manager in-memory implementation provides only single-key operations, and so it suffices to use the Map adapter as seen in the previous section.

Some implementations of the store engine for the cache-manager package provide multi-key operations. The example below uses Redis to back the cache store via the cache-manager-redis-store npm package.

import { cacheManagerAdapter } from 'loading-cache/adapters/cache-manager'

import { caching } from 'cache-manager'
import { redisStore } from 'cache-manager-redis-store'

// set up the Redis cache
const redisCache = cacheManager.caching({ store: await redisStore({}) })

new LoadingCache(
  myFetchAll,
  cacheManagerAdapter(redisCache.store), // translates cache operations into `mget()` and `mset()` calls on the store
  // if not defined on the store implementation, it falls back to multiple `get()` and `set()` calls
)

Note Unless your use case calls for it, you can bypass using cache-manager as an intermediary and directly setup a Redis client to use as a cache store for the LoadingCache, see the usage sections for Redis with either node-redis or ioredis.

See How is this different from cache-manager? for a comparison.

Other stores

For any other cache stores for which there is no adapter provided by this package, you can implement your own, simply by providing as the cache to the LoadingCache an object with the function properties getFromStore and putInStore. You can refer to the Cache type [todo use docs link against github to the generated docs for this type] for the expected method signatures, as well as review the implementations of the built-in adapters under src/adapters.

The getFromStore method must uphold the same constraints as those applicable to the loaderFn, as documented here. Any index in the returned array that is either (exclusively) undefined or null is considered to be missing from the cache, and the corresponding key for that index will be loaded from the loaderFn.

The putInStore method is guaranteed to receive an Array of keys and an Array of values such that:

• the Array of values is the same length as the Array of keys
• each index in the Array of values corresponds to the same index in the Array of keys

The Array of values received by putInStore can include any values returned by the loaderFn (e.g. null or undefined), so it must be able to cope with these.

Note If you think a store implementation warrants an adapter be included with this package, consider contributing one or filing an issue.

Background cache updating

In most cases, consumers of the loading cache are only interested in the returned data, and do not need to wait for the data to be updated in the cache(s) as part of the read-through process. As such, the get() and getMany() methods always resolve as soon as the data is available (all keys are retrieved from the cache(s) or loaded from the loaderFn), and update the cache(s) in the background.

Warning Since the cache update(s) rely on the data after it has been returned, the returned data must not be mutated before the cache update(s) complete. Otherwise, the data that is stored in the cache might differ from the data that was received by the requesting code, which could create inconsistencies within your application when the cached data is later retrieved. If you intend to mutate the returned data, you must make sure to do one of the following:

• wait for all the cache update(s) to complete, see Observing the cache update(s)
• only mutate a (deep) copy of the returned data
• disable cache update(s) for the request, see the reference for the noCacheUpdate option

On the other hand, the prime() and primeMany() methods perform their cache updates in the "foreground". That is, they only resolve after any and all cache update(s) complete.

Note To "prime" or "warm up" a cache is to pre-populate it with some keys, typically those that are expected to be accessed shortly thereafter.

await loadingCache.prime('A')

const dataForA = await loadingCache.get('A')

// elsewhere in your application
const dataForA = await loadingCache.get('A')

After the call to prime() resolves, the value loaded for 'A' will have been put in the cache, therefore all following calls to get() will retrieve key 'A' from the cache.

In contrast, without first calling prime(), the key 'A' will be put in the cache sometime after the first call to get() resolves, by which point the other call to get() might have already initiated loading the key from the loaderFn.

Observing the cache update(s)

In some cases, it may be necessary to be notified when the cache(s) are updated, for example to perform some cleanup or notify some clients. All methods on the LoadingCache accept an onCacheUpdated callback that is called after each cache update (with any error thrown or reason rejected as the first argument, or null), and an onComplete callback that is called after all of the cache update(s).

const result = await loadingCache.get('A',
  {
    onCacheUpdated: (err) => {
      if (err) {
        console.error("Loading Cache failed cache update for key 'A'", err)
        return
      }

      console.log("cache updated with key 'A'") // logged second
    },
    onComplete: () => {
      console.log("completed all cache update(s) for key 'A'") // logged third
    }
  })

console.log("got value for key 'A':", result) // logged first

Note The onCacheUpdated callback may be called fewer times than the number of provided caches, in cases where the requested key(s) were all retrieved from a subset of the cache(s) without needing to lookup any from the others (or the loaderFn).

The onComplete callback can be used to wait for all the cache update(s) to complete before continuing.

const result = await new Promise(async (resolve, reject) => {
  const result = await loadingCache.get('A',
    {
      onComplete: () => {
        resolve(result)
      }
    })
    .catch(reject)
})

console.log("got value for key 'A':", result)

To allow for a consistent API, the prime() and primeMany() methods accept the same callbacks. The method will resolve after effectuating all calls to the onCacheUpdated callback, after which the onComplete callback will be called before the next event loop tick (placed in a process.nextTick()).

Failure to update the cache

Since the calls to get() and getMany() resolve without waiting for any cache update(s) to complete, the returned promise cannot be used to communicate cache update(s) resolution status (namely, to communicate a failure). The onCacheUpdated callback hook is provided as a means for your application to handle failures that occured as a result of a call to the putInStore function on a cache, such that cache updating can safely take place in the background. The argument to the callback is the thrown error or rejected reason from the failed call.

Failures when updating a cache are also emitted as an error event to allow for centralized error handling, see Events.

To ensure that all cache updating errors are handled, it is recommended to register an error event listener when setting up the LoadingCache instance, as later calls to the LoadingCache methods may not necessarily handle them.

Warning It is important that the onCacheUpdated callback not throw an error (or reject). Since there is no possible avenue by which these can be safely handled, this will result in an unhandled rejection.

Multiple cache levels

The LoadingCache can be supplied more than one cache, in order to implement a more complex cache hierarchy. Construct a multi-level LoadingCache by supplying an Array of caches, ordered from upper to lower level in the hierarchy.

The LoadingCache will check for the requested keys in the first cache, continuing to search through the next level(s) of cache and eventually requesting from the loaderFn only those keys that were missed in all previous levels. Cache(s) in the lower level(s) may not need to be accessed if all requested keys were found in the previous cache(s).

A general strategy is to have the upper cache level(s) be optimized for speed, only storing the most frequently accessed keys, and relegate access of less-frequently accessed keys to the lower cache level(s).

The following example showcases a two-level cache hierarchy:

• the upper-level is a faster LRU (least recently used) cache which limits total memory to hold at most 100 cached values via the lru_map npm package
• the lower-level a Redis cache using a client created via the redis npm package

import { mapAdapter } from 'loading-cache/adapters/map'
import { redisAdapter } from 'loading-cache/adapters/redis'

import { LRUMap } from 'lru_map'

import { createClient } from 'redis'

// set up the Redis client
const redisClient = createClient()
await redisClient.connect()

const multiLevelLoadingCache = new LoadingCache(
  myFetchAll,
  [
    mapAdapter(new LRUMap(100)),
    redisAdapter(redisClient),
  ],
)

See the LoadingCache multi-level caching spec for more details.

Events

The LoadingCache class is a Node.js EventEmitter.

It emits a single event, the error event, whenever a cache update triggered as a result of a get(), getMany(), prime(), or primeMany() call failed. The argument to the event listener is the thrown error or rejected reason from the call to the putInStore function on a cache. The event could be emitted more than once as a result of a LoadingCache method call, if using multiple cache levels and more than one cache update failed.

Warning By default, LoadingCache emits error events silently (only emits when there is at least one listener) so that your application won't crash if it is not listening to the error event. See the EventEmitter docs for more details.

Configuration Options

The LoadingCache accepts a number of configuration options as the third argument to the constructor. See the reference.

Why Use This?

Abstracts the process of getting entries from a cache (including key, value serialization and deserialization), loading entries from a backing data source, and merging the hits and misses.

Where an application might set up custom machinery to handle caching and loading from various data sources, each consumed via their own API, using LoadingCache instances encapsulates the underlying data source interaction and caching mechanisms, providing a common way to retrieve data from the various data sources.

Simply put, it allows you to setup a LoadingCache instance once, and use it around your app to reap the benefits of caching, centralizing all the complexity into the single point of initialization.

How is this different from cache-manager?

How is this different from dataloader?

• LoadingCache supports asynchronous (as well as synchronous) cache operations, which enables plugging in any cache implementation (namely ones caching data outside the application's runtime memory, e.g. Redis). DataLoader only accepts fully synchronous cache implementations.

• LoadingCache is designed to get and put many keys in the cache at once. With Redis, for example, this can significantly save on networking overhead and noticeably improve performance, even with relatively small key sets. (For caches that don't support multiple-key operations, adapting is simply a matter of wrapping the single-key operations with a function looping over each key, as is done in the adapter for Map-like stores.) DataLoader only performs single-key operations on its cache.

• LoadingCache supports multi-level caching, with each level able to use a different cache implementation. A two-level LoadingCache might retrieve a set of keys for example by first looking up in a fast cache within the process' memory (e.g. an LRU cache), falling back if needed to a slower cache over the network (e.g. a Redis cache), before finally loading any remaining missed keys from a backend. The cache updating process automatically takes care of putting in each cache level only those missed keys which were retrieved from subsequent cache levels or the backend, which avoids unecessary cache operations. DataLoader only accepts a single cache.

• LoadingCache is able to update its cache(s) in the background, which ensures requests for keys can resolve as soon as possible. DataLoader updates its cache before the values are returned.

If these differences are not relevant to your use case, consider using the dataloader package instead.

Optimization

Reducing cache and backend round-trips

Where possible, opt to use the multi-key variants of the LoadingCache methods – getMany() and primeMany() – when working with multiple keys at once, rather than repeatedly calling their single-key counterparts with each key.

Using these methods ensures the minimum possible number of requests are issued to the cache(s) and loaderFn, and optimizes the data processing overhead. In general, retrieving a key requires (in the worst case) a round-trip to each cache and to the backend. If the keys are retrieved together, these round-trips are only incurred a single time with all the keys at once.

Batching

It is common for applications to experience large bursts of demand for some common data, usually as a result of some user trend (e.g. a traffic surge) or as caused by client behavior, failure recovery (e.g. coinciding retries), or a periodic cron job.

This is especially true if handling a single request can result in multiple calls for the same data, as may be the case when implementing a GraphQL API.

Retrieving this data in an application is usually performed via asynchronous function calls, which are then queued on the event loop. At any given tick in the event loop, if it is possible to coalesce those calls which exist simultaneously in the queue, it presents an opportunity to batch together requests for data from the same source, as well as deduplicate requests for the same data.

The dataloader npm package enables coalescing concurrent requests for the data into a single batch, reducing load on the data provider. The DataLoader class provided by the package takes in a batchFn which receives batched keys from the calls to load data from its methods.

Wrapping the loaderFn with a DataLoader

One way to integrate this pattern into the LoadingCache is to provide as the loaderFn the loadMany() method on a DataLoader instance, which itself takes in as the batchFn the implementation for retrieving the batched keys. Notice that the signatures for the DataLoader batchFn and loadMany() method, and the LoadingCache loaderFn and getMany() method, are all the same, which makes it easy to interchange them based on performance needs.

Since caching is already handled by the LoadingCache instance, make sure to disable caching on the constructed DataLoader instance by setting the cache option to false, to avoid duplication of the cache.

import { LoadingCache } from 'loading-cache'

import { DataLoader } from 'dataloader'

const dataLoader = new DataLoader(
  myFetchAll,
  { cache: false },
)

const loadingCache = new LoadingCache(
  dataLoader.loadMany.bind(dataLoader),
  /* ... */,
)

// assume the cache store is initially empty

await loadingCache.get('A') // loads 'A' from the loaderFn
// myFetchAll called once with ['A']

await Promise.all([
  loadingCache.getMany(['A', 'B']), // retrieves 'A' from the cache, then loads 'B' from the loaderFn
  loadingCache.get('C'), // loads 'C' from the loaderFn
])
// myFetchAll called a second time with ['B', 'C']

The Promise.all fires two calls to the loaderFn, once to load 'B' and once to load 'C'. The calls will be batched by the DataLoader (since the loaderFn is the DataLoader's loadMany() method), resulting in a single call to the underlying myFetchAll() implementation.

Fronting the LoadingCache with a DataLoader

For read-heavy applications, it may be advantageous to ensure the cache lookups themselves are batched, by directly fronting the LoadingCache getMany() method with a DataLoader.

In this case, the deduplicateKeys option should be set when creating the LoadingCache, since the DataLoader instance fronting it will not deduplicate keys when caching is disabled. Since calls to the LoadingCache will already be batched, there is no need to also batch calls to the loaderFn (unless of course that same loaderFn will be called in places outside the LoadingCache.)

const loadingCache = new LoadingCache(
  myFetchAll,
  /* ... */,
  { deduplicateKeys: true },
)

const dataLoader = new DataLoader(
  loadingCache.getMany.bind(loadingCache),
  { cache: false },
)

Accessing the LoadingCache in this way via the DataLoader encapsulates its API, which means the second options parameter to getMany() cannot be specified by callers.

In practice, any performance gains from batching cache requests across calls to getMany() would likely be marginal, assuming cache lookups are inexpensive.

Avoiding repeat requests for keys missing in the backend

A request for keys from a backend will typically only receive values for those keys that the backend was able to match to some corresponding data. The same caching regime for data present in the backend can typically also be used to cache the absence of the data in the backend. In this way, the cache can be employed to keep track of which data is missing in the backend, in addition to its primary role of storing the data from the backend.

Since the loaderFn must return a value for all the loaded keys, some value must be returned for those keys for which no data was received from the backend. Caching the absence of a key can be done by making sure that the loaderFn returns a value that gets put in the cache for that key such that when it is later retrieved, it can be interpreted by the application to discern the key's absence in the backend. In this way, once a key is requested for which there was no data from the backend, ensuing requests for that key do not result in a query the backend for data that is known to be missing. This ensures applications can efficiently handle cases where required data is not found.

A simple approach would be to return null from the loaderFn for missing keys and store this as-is in the cache (if the cache is accessed directly, a null for a key would need to be handled differently than undefined; the former indicating that the key is absent from the backend, and the latter that it is absent from the cache itself).

Another approach compatible with string key-value stores, assuming a backend that returns JSON values, would be to have the loaderFn handle missing keys by returning an empty object as the JSON-parsed value, which can then be serialized and deserialized as any other value.

Note the same principle can be applied to backend errors for queried keys, by caching a value that indicates the error that was returned by the backend for that key.

Deduplicating keys

The getMany() (or primeMany()) methods always deduplicate the requested keys, which ensures the number of keys needing to be looked up in the cache(s) and loaded from the loaderFn is kept to a minimum.

Common Patterns

Freezing results to enforce immutability (when using an in-memory store)

Since LoadingCache caches the results of the loaderFn to the provided store, in order to ensure predictable behaviour when using an in-memory store, these values should be treated as immutable by your application. You can create a higher-order function to enforce immutability with Object.freeze():

function freezeResults(loaderFn) {
  return async (keys) => {
    const values = await loaderFn(keys)
    return values.map(Object.freeze)
  }
}

const myLoadingCache = new LoadingCache(
  freezeResults(myFetchAll),
  /* ... */,
)

Chunking Keys

If you wish to limit the number of keys that can be requested from the cache or loaded from the loaderFn in a given call, you can opt to retrieve the full set of keys in chunks of the desired size. This can be done to prevent flooding resources in extreme cases, or to accomodate for bandwidth limitations.

This is not a facility provided by the LoadingCache out of the box, but can be easily built into the loaderFn implementation or added to the getFromStore method of the cache. The example below makes use of the chunk function from the lodash npm package.

import _ from 'lodash'

function loadInChunks(loaderFn, chunkSize) {
  return async (keys) => {
    const keyChunks = _.chunk(keys, chunkSize)
    const loadedChunks = await Promise.all(keyChunks.map(loaderFn))
    return loadedChunks.flat()
  }
}

const loadingCache = new LoadingCache(
  loadInChunks(myFetchAll, 100),
  /* ... */,
)

Restricting access to the LoadingCache instance

By design, methods of the LoaderCache have side-effects in the form of mutations against the provided cache store. You may want to restrict certain parts of your application from making changes to the cache, while still being able to retrieve data.

The implementation below wraps the getMany() method to provide a means to get data from the LoadingCache but with cache updates disabled. This could for example be useful for a service which is likely to retrieve a subset of data that is not relevant to the rest of the application, and would otherwise waste cache occupancy.

const loadingCache = new LoadingCache(/* ... */)

new MyService({
  getMyData: (keys) => loadingCache.getMany(keys, { noUpdateCache: true }),
})

Common Data Sources

Try the integrations in the /examples for use with a specific back-end.

Reference

A complete reference of the public API formed by the exported members of this package is available under /docs.