swstats
Sliding window statistics for Nodejs and browser.
Time and Size sliding windows, capable of calculating incremental statistics, such as count, sum, avg, stdev, freq, etc...
Features
- Nodejs and Browser compatible
- Time or size windows (not both)
- Global time or custom specified timestamp
- Numeric or category stats
- Stats are calculated incrementally, using online algorithms
- Core category stats: sum, frequency, mode
- Core numeric stats: count, sum, min, max, avg, stdev
- Plugable custom stats functions
- Time windows are slided each second
Installation
npm install swstats
Quick start
Nodejs:
const SWindow = ; // Create a time window of 10 seconds. Each value pushed to// the window will be stored in a unique slot (step:1)// Window will slide on each secondvar sw = 10000step:1;var i=0; ; ;
or browser:
The results will be something like:
count: 9 sum: 4383476926850223 avg: 048705299187224704 max: 09865126881294235 min: 00023880687887007923 stdev: avg: 048705299187224704 sqsum: 26772528231211963 sum: 4383476926850223 stdev: 024546266317028345 ... count: 994 sum: 49503542685194975 avg: 04980235682615189 max: 09865126881294235 min: 00023880687887007923 stdev: avg: 04980235682615189 sqsum: 3321783737578395 sum: 49503542685194975 stdev: 029352342336095866
Time Window API
new SWindow.TimeStats(time,[options])
Creates a new time window of time duration. The window will slide on each second. options is optional, and can take the following parameters:
- step: Time step for each time slot. When a new value is added to the window, it can be stored in a new slot, or use the last one, depending on the time that value was inserted. If step is 10, that means that if a new value is inserted after 10 ms. from the last one, a new slot will be created for this value. Otherwise, the value will be added and grouped to the last slot. By default, time step is 1000 ms. High values will prevent the window increasing and consuming memory over time for long windows, but some stats will lose accuracy (such as stdev).
- timestamp Defines how time is measured. Can take the following values:
- TimeStats.TS.ABSOLUTE This is the default value. When a value is pushed to the window, it takes the current machine timestamp. In this mode, windows are naturally slided in real-time.
- TimeStats.TS.RELATIVE With this mode, you can push values that has an associated timestamp that is independent of the machine time. Windows are slided based on the max and min inserted timestamps. This mode is less performant, but allows you to insert values out of time order.
- type: Value types for this window. Can take the values "numeric" or "category". By default, windows are numeric.
- ops: Statistic operations to perform on the window values. If you don't need all the core functions, or want to add a new custom operation, you can pass an array of operation names. By default, for numeric stats, ops are ["count","sum","avg","stdev"], and for category ["sum","freq","mode"]
- stats: Object with pre-initialized stats values.
Example:
const SWindow = ; // Time window of 10 secondsvar tw1 = 10000 step:1000 // Values will be accumulated in slots of 1 second type:"numeric" // Numeric values ops:"sum" // We only want to sum values stats : sum : 233 // Initial value where sum will start ; // Time window of 10 seconds with relative timestampvar tw2 = 10000 timestamp:TimeStatsTSRELATIVE step:1000 // Values will be accumulated in slots of 1 second type:"numeric" // Numeric values;
timeWindow.push(val)
Adds a new value to the window. It will consume a new slot or be added to the current one, depending on the step value. If timestamp mode is absolute, simply push the value, but if relative is activated, you must push an object with value and timestamp:
tw1;tw2;
timeWindow.push([vals])
Adds multiple values at once, in the same manner as push(val).
timeWindow.clean()
Resets window and stats.
timeWindow.pause()
Pauses the window, so no slide occurs, but no new values will be added.
timeWindow.resume(shift)
Resumes a paused window. If shift is true, the timestamps of each slots will be shifted relative to the time the window is resumed, otherwise they keep their original timestamps. This will affect how the window is slided on the next second.
destroy()
Kill the window, so no new values will be added, and no slide will occur. You can still access to its stats.
timeWindow.length
Gets the current size of the window (number of slots)
timeWindow.stats
Gets the current calculated stats
count: 9 sum: 4638591015963949 max: 08745422003577794 min: 004623850021742104 avg: 05153990017737722 stdev: avg: 05153990017737722 sqsum: 3044764166996456 sum: 4638591015963949 stdev: 026957558983867996
timeWindow.window
Gets the current calculated stats per slot
t: 1517499437649 sum: David: 571 John: 404 freq: David: 05856410256410256 John: 041435897435897434 mode: 'David' threshold: David: false John: false t: 1517499438649 sum: John: 398 David: 580 freq: John: 04069529652351738 David: 05930470347648262 mode: 'David' threshold: John: false David: false t: 1517499439649 sum: David: 449 John: 274 freq: David: 06210235131396957 John: 03789764868603043 mode: 'David' threshold: David: true John: false
Size Window API
new SWindow.SizeStats(size,[options])
Creates a new size window with size slots. The window will slide when the maximum slots have been reached. options is optional, and can take the following parameters:
- type: "numeric" or "category"
- ops: Same a s TimeStats.
- stats: Object with pre-initialized stats values.
Example:
const SWindow = ; // Size window of 100 valuesvat sw = 100 type:"category" // Category values ops:"freq" // We only want the value frequency stats : freq : // Initial value where freq will start "john" : 03 "david" : 07 ;
sizeWindow.push(val)
Adds a new value to the window. It will consume a new slot.
sizeWindow.push([vals])
Adds multiple values at once. In case of category values, unlike the temporary window, all the values pushed in one call will be added together in the same slot:
sizeWindow;
will consume only one slot, whereas:
sizeWindow;sizeWindow;
will consume two slots.
sizeWindow.clean()
Resets window and stats.
sizeWindow.length
Gets the current size of the window (number of slots)
sizeWindow.stats
Gets the current calculated stats
timeWindow.window
Gets the current calculated stats per slot
Custom Statistics API
SWindow.register(type,name,deps,fn,def)
It is possible to implement and plug a custom function to calculate any stats you need. To register a new stats function, you must call the register function with the following parameters:
- type: "numeric" or "category"
- name: Name of the operation
- deps: Array of dependencies. If the custom function depends on previous stats to be performed, you can pass the names in this array.
- fn: The stats function. It will be described later.
- default: If true, new created windows without the ops options specified, will perform by default this stats operation.
The stats function fn
fn(currval,newitems,olditems,allitems,newstats,oldstats)
The stats function passed to the register method takes the following arguments:
- currval: The current stats value for your function, prior to the next calculation when an item has been added.
- newitems: Array of new values added to the window since the last function call.
- olditems: Array of removed items from the window since the last function call.
- allitems: Array of all the values that are currently in the window (allitems includes newitems and excludes olditems)
- newstats: Current calculated stats by the functions called before this.
- oldstats: Previous calculated stats.
The items in the newitems, olditems and allitems arrays, have the following format:
- For numeric values:
t : 1496843741554 // Timestamp in unix format (ms) v : 1234 // Total value for this slot min : 24 // Minimum of the accumulated values max : 45 // Maximum of the accumulated values l : 4 // Values accumulated in this slot
- for category values:
t : 1496843741554 // Timestamp in unix format (ms) v : // Total values for this slot "john" : 4 "david" : 6
Note: Timestamp only appears in time windows.
Examples:
const SWindow = ; // Calculates the varianceSWindow; // A Weighted sum, where value decreases as more items are pushed// to the windowSWindow; var sw = 10 ops : "variance""decsum" stats : decsum : ratio : 099; forlet i=0;i<100;i++ sw; sw;console;
Results:
decsum: ratio: 099 weight: 013397967485796175 sum: 1053536530978331 avg: 75 stdev: avg: 75 sqsum: 625 sum: 75 stdev: 25 variance: 625