manger

cache feeds

manger - cache feeds

The manger Node package caches RSS and Atom formatted XML feeds using LevelUP. It provides an interface to query entries by feed and time.

An optional String().

String() | null | undefined

One metadata object per XML feed.

  • author str()
  • copyright str()
  • feed str()
  • id str()
  • image str()
  • language str()
  • link str()
  • payment str()
  • subtitle str()
  • summary str()
  • title str()
  • ttl str()
  • updated str()

A related resource of an entry().

  • href str()
  • length str()
  • type str()

entry()

An individual entry.

  • author str()
  • enclosure enclosure() | null | undefined
  • duration str()
  • feed str()
  • id str()
  • image str()
  • link str()
  • subtitle str()
  • summary str()
  • title str()
  • updated str()

A query to get a feed or entries of a feed in a time range between Date.now() and since.

  • url String()
  • since Date() | null | undefined
  • etag String() | null | undefined An entity tag
  • force Boolean()| false Force update ignoring cache

Options for a Manger instance.

  • cacheSize Number() | 8 * 1024 * 1024 Passed to levelup()
  • objectMode Boolean() | false
  • name String() The name of the file system directory for the database
  • opts opts()

The manger module exports a single function that returns a new cache object (an instance of Manger). To access the Manger class require('manger').

var manger = require('manger')
var cache = manger('/tmp/manger.db')

If options has objectMode set to true, results are read as Object types, otherwise they are Buffer or String moulding valid JSON, depending of which stream of the API is used.

manger leverages the lexicographical key sort order of LevelDB. The keys are designed to stream feeds or entries in time ranges between now and some point in the past.

The distinction between feed and entries may not be clear. A feed models the metadata of an RSS or Atom feed (title, author, published, etc.), while entries are the actual items in the feed. These are detached to not repeatedly transmit feed metadata—after all manger tries to save round-trips.

A Transform stream that transforms queries or URLs to entries.

  • write(Buffer() | String() | query())
  • read() Buffer() | String() | entry()

A stream that transforms queries or URL strings to feeds.

  • write(query() | String())
  • read() Buffer() | String() | feed()

A Readable stream of URLs of all feeds currently cached.

  • read() Buffer() | str()
  • x Number() | 5 The number of streams to engage concurrently

Updates all ranked feeds and returns a stream that emits feed URL strings of updated feeds. This, of course, might be a resource heavy operation! Feeds are updated ordered by their popularity, using the rank index, therefore flushCount has to be invoked before this method can take any effect.

  • read() str()

A stream of URLs sorted by rank (highest first).

  • read() Buffer() | String() | null

Resets the ranks index.

  • cb Function(error, count) | null | undefined The callback
    • error Error() | null | undefined The possible error
    • count Number() Total number of cached feeds

manger keeps an in-memory count of how many times feeds have been accessed. This function flushes the counter to disk, updating the ranks index.

Applies callback cb without arguments if a feed with this url is cached.

  • url String() The URL of the feed
  • cb Function(error) The callback
    • error Error() | null | undefined The possible error

Attempts to remove a feed matching the url from the cache and applies callback without error if this succeeds.

  • url String() The URL of the feed
  • cb Function(error) | null | undefined The callback
    • error Error() | null | undefined The possible error

The manger module decorates the exported Manger constructor with two convenience functions for querying the cache.

A failable factory function returning a valid query() or null | undefined.

This stream transforms JSON to queries which can be piped to feeds() and entries() streams. The expected JSON input format:

[
  { "url": "http://feeds.5by5.tv/directional" },
  { "url": "http://www.newyorker.com/feed/posts",
    "since": 1433083971124 },
  { "url": "https://www.joyent.com/blog/feed",
    "since": "May 2015" },
  ...
]

Where "since" can be anything Date() is able to parse.

With npm do:

$ npm install manger

MIT License