s3-disk-cache

Cached S3 Wrapper

S3 File Access Abstraction providing Memory and Disk Caching Layer. Useful e.g. in lambda functions if you want to reduce the amount of s3 access for serving (semi-)static files.

What it does

• Access abstraction to access JSON and GZipped data on AWS S3
• Two Layer caching (memory and disk)
• Allows you to define cache constraints like TTL

Getting Started

Install

$ npm install --save s3-cached

Request S3 Files

const s3 = require('s3-cached')({
  bucket: 'YOUR_BUCKET_NAME',
  s3Options: {
    accessKeyId: 'YOUR_ACCESS_KEY_ID',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'
  }
});
 
s3.getJsonObjectCached('large.json').then((json) => {
  // do something with the json data
}).catch((err) => {
  // there has been an error
});

Available functions

• getBinaryObjectCached: retrieve file content, caching additional modifications possible through parameter
• getTextObjectCached: retrieve file content as string and return as promise
• getJsonObjectCached: retrieve file content as string, parse as json and return as promise
• getDeflatedObjectCached: retrieve file content, deflate and return as promise
• getKeysCached: retrieve all file names in bucket with given prefix

Note that you can specify the ttl and/or custom bucket on a per file basis by calling e.g. s3.getJsonObjectCached(FILE_NAME, { ttl, bucket }). For exact method signatures please check the code.

Options

bucket

Type: string
Required

Specify the Bucket name you want to retrieve data from.

s3Options

Type: object
Default: -

Passed into aws-wrap-sdk for AWS.S3() initialization.

logger

Type: logger
Default: null

Passed into aws-wrap-sdk.

ttl

Type: integer
Default: 600

Define how long a cached file is kept by default. This can be overwritten on a per-file basis by passing a second parameter into the function.

memoryLimit

Type: integer
Default: 100

Define how many cached entities can be hold in memory at the same time. If more entities are present, the earliest are discarded from memory cache.

diskMaxSize

Type: integer
Default: 469762048

Maximum amount of disk space in bytes used by disk cache. Earliest files are discarded from file cache if more space is used.

diskTmpDirectory

Type: string
Default: /tmp

Location to store temporary data for disk cache.