@segment/localstorage-retry

    1.3.0 • Public • Published

    localstorage-retry

    Circle CI

    Provides durable retries with a queue held in localStorage (with graceful fallbacks to memory when necessary).

    How It Works

    Each page maintains its own list of queued and in-progress tasks, while constantly refreshing its ack time. If a queue goes more than 10s without updating its ack, another page will remove it, claim all queued tasks, and retry all in-progress tasks.

    API

    new Queue(name, [opts], processFunc(item, done(err, res)))

    You can omit the opts argument to initialize the queue with defaults:

    var Queue = require('@segment/localstorage-retry');
    
    var queue = new Queue('my_queue_name', function process(item, done) {
      sendAsync(item, function(err, res) {
        if (err) return done(err);
        done(null, res);
      });
    });
    
    queue.on('processed', function(err, res, item) {
      if (err) return console.warn('processing %O failed with error %O', item, err);
      console.log('successfully sent %O with response %O', item, res);
    });
    
    queue.start();

    Options

    The queue can be initialized with the following options (defaults shown):

    var options = {
      minRetryDelay: 1000,   // min retry delay in ms (used in exp. backoff calcs)
      maxRetryDelay: 30000,  // max retry delay in ms (used in exp. backoff calcs)
      backoffFactor: 2,      // exponential backoff factor (attempts^n)
      backoffJitter: 0,      // jitter factor for backoff calcs (0 is usually fine)
      maxItems: Infinity     // queue high water mark (we suggest 100 as a max)
      maxAttempts: Infinity  // max retry attempts before discarding
    };
    
    var queue = new Queue('my_queue_name', options, (item, done) => {
      sendAsync(item, (err, res) => {
        if (err) return done(err);
        done(null, res);
      });
    });
    
    queue.start();

    .addItem(item)

    Adds an item to the queue

    queue.addItem({ a: 'b' });

    .getDelay (attemptNumber) -> ms

    Can be overridden to provide a custom retry delay in ms. You'll likely want to use the queue instance's backoff constants here.

    this.backoff = {
      MIN_RETRY_DELAY: opts.minRetryDelay || 1000,
      MAX_RETRY_DELAY: opts.maxRetryDelay || 30000,
      FACTOR: opts.backoffFactor || 2,
      JITTER: opts.backoffJitter || 0
    };

    Default implementation:

    queue.getDelay = function(attemptNumber) {
      var ms = this.backoff.MIN_RETRY_DELAY * Math.pow(this.backoff.FACTOR, attemptNumber);
      if (this.backoff.JITTER) {
        var rand =  Math.random();
        var deviation = Math.floor(rand * this.backoff.JITTER * ms);
        if (Math.floor(rand * 10) < 5) {
          ms -= deviation;
        } else {
          ms += deviation;
        }
      }
      return Number(Math.min(ms, this.backoff.MAX_RETRY_DELAY).toPrecision(1));
    };

    .shouldRetry (item, attemptNumber, error) -> boolean

    Can be overridden to provide custom logic for whether to requeue the item. You'll likely want to use the queue instance's maxAttempts variable (which is overridable via constructor's opts argument).

    Default:

    queue.shouldRetry = function(item, attemptNumber, error) {
      if (attemptNumber > this.maxAttempts) return false;
      return true;
    };

    You may also want to selectively retry based on error returned by your process function or something in the item itself.

    Override Example:

    queue.shouldRetry = function(item, attemptNumber, error) {
      // max attempts
      if (attemptNumber > this.maxAttempts) return false;
    
      // based on something in the item itself
      if (new Date(item.timestamp) - new Date() > 86400000) return false;
    
      // selective error handling
      if (error.code === '429') return false;
    
      return true;
    }

    .start

    Starts the queue processing items. Anything added before calling .start will be queued until .start is called.

    queue.start();

    .stop

    Stops the queue from processing. Any retries queued may be picked claimed by another queue after a timeout.

    queue.stop();

    Emitter

    You can listen for processed events, which are emitted with each invocation of the processFunc and passed any error or response provided along with the item itself.

    If a message is discarded entirely because it does not pass your shouldRetry logic upon attempted re-enqueuing, the queue will emit a discard event.

    If the queue is reclaiming events from an abandonded queue, and sees duplicate entries, we will keep the first, and discard the rest, emitting a duplication event for each.

    processed

    queue.on('processed', function(err, res, item) {
      if (err) return console.warn('processing %O failed with error %O', item, err);
      console.log('successfully sent %O with response %O', item, res);
    });

    discard

    queue.on('discard', function(item, attempts) {
      console.error('discarding message %O after %d attempts', item, attempts);
    })

    duplication

    queue.on('duplication', function(item, attempts) {
      console.error('discarding message %O due to duplicate entries', item, attempts);
    })

    License

    Released under the MIT License

    Keywords

    none

    Install

    npm i @segment/localstorage-retry

    DownloadsWeekly Downloads

    13,378

    Version

    1.3.0

    License

    SEE LICENSE IN LICENSE

    Unpacked Size

    56.5 kB

    Total Files

    19

    Last publish

    Collaborators

    • rfarivar
    • pmaid
    • song4you
    • peterdemartini
    • emmy.byrne
    • ashkon
    • vincen7tran
    • ariel.silvestri
    • xinghaohuang
    • dean-huynh
    • cdignam-segment
    • bkambo
    • abhinavsureka
    • rikezatsegment
    • arunlalam-segment
    • cjradek
    • neeharikakondipati
    • simpixelated
    • chihchun-twilio
    • sangeeta.singh
    • acharles14
    • jyim008
    • seghungtran
    • alayvora
    • tv0
    • pchen-twilio
    • hema-segment
    • sudojatin
    • mkhan-twilio
    • mayberex
    • anandjha
    • pmunin
    • oscb
    • krousseau
    • brandongregoryscott
    • knksmith57
    • sachinwathore
    • fhalim-segment
    • cfree
    • mettledrum
    • 5t0k4st1k
    • maerf0x0
    • gnijor
    • ifonseka1
    • aniket.gupta
    • hjoonpm
    • lauramunozjimenez
    • celine-segment
    • parsa-segment
    • hmorgan94
    • pmcanseco-segment
    • ynguyen
    • mshwery
    • debajitr
    • masira
    • gpsamson
    • amillet89
    • cholt002
    • av-segment
    • aghotikar
    • vikrant-segment
    • ankur.agarwal
    • larryatsegment
    • lbrink
    • shivpoojan-segment
    • ariel_segment
    • zkuzmic-segment
    • scruwys1
    • sowjanyaedara
    • chrischalstrom
    • rossedfort
    • eesegment
    • lew-gordon
    • kyliepedersen
    • jinapark
    • skntwilio
    • segmentio
    • segment-admin
    • dominicbarnes
    • shobhita-agarwal
    • deanhuynh
    • jlineaweaver
    • ladanazita
    • anoonan
    • peripheral
    • achille-roussel
    • rajulvadera
    • nettofarah
    • dalchandc
    • lpediredla
    • albert.segment
    • segment-danielstjules
    • n2parko
    • segment-andy-yeo
    • sahilp
    • psankaranarayana
    • vdemedes
    • emilio-gomez-lavin
    • andreiko_ru
    • xagos
    • leifdreizler
    • alan-segment
    • tyson_segment
    • bgamwell
    • jfabre-segment
    • uditmehta
    • hellooimkat
    • brienne.mcnally
    • sanscontext
    • eculver
    • aultimus
    • salolivares
    • erikdw
    • chenxiangzhang
    • fauzyy
    • jaimal
    • davidbirdsong
    • ktrinh
    • calthomson
    • yunqiaohuang
    • dk1027
    • nielssegment
    • cdrycroft
    • yabrira
    • julee05
    • kamebkj
    • ryan_segment
    • laurenmreeder
    • tidothegreat
    • mericsson
    • prayansh-segmentt
    • stargaesser
    • andyguwc
    • jeremylarkin
    • bsneed
    • danieljackins
    • tomeliaz
    • segment-seth
    • bihl
    • shamil.ataev
    • james9446
    • priscilla.giatti
    • kellylu
    • smokeybears
    • jmbuckner
    • benhorowitz
    • yolken-segment
    • nlsun
    • thomas-pelletier-segment
    • brian-segment
    • segment-michael
    • cohara87
    • drew-thompson
    • segment-jsingh
    • vanesng
    • cvillela
    • pauljaeinyoo
    • mniehe
    • stephment
    • sharadbhadouria
    • emilyc
    • maggieyu
    • anoonan16
    • durgani
    • julio.farah
    • hblanks-segment
    • rsata
    • shyaan
    • juhaelee
    • akleiner2
    • kathryn-taylor
    • andrius-segment
    • pooyaj
    • llbsgmt
    • lab176
    • valerieernst
    • ucarion
    • solon.aguiar
    • rjenkinsseg
    • segment-ulysse
    • khinkalilover
    • maloneya
    • zackurey
    • kelcook
    • arta.razavi
    • odoren
    • alistairbarrell
    • anna.choi
    • whamo12
    • amaloney
    • gilomer
    • kiara.daswani
    • marcelopv
    • yannieyip
    • eric.rogner
    • marinhero
    • lcamposg
    • corey.ching
    • steve_at_segment
    • tepahk
    • sorrel.j
    • cjo2
    • fc-segment
    • kdharaiya
    • warrengreen
    • jon.anderson-at-segment.com
    • markzegarelli
    • stysegment
    • ajhenry
    • stacy.song
    • niveda.balananthan
    • collinvandyck
    • krlv
    • rexatsegment
    • nickaguilar
    • tri.truong
    • csayuso
    • joyce-shi
    • bradenbecker
    • neha.sanghrajka
    • reneewang
    • dan.lasky
    • sam.tapia
    • gbbastos
    • vikramkumar19
    • mpriyad25
    • peter.walker
    • jeremy.parker
    • khamidou-segment
    • smidges
    • daltonscharff