compressed-json-collection
A compressed collection used to manage and cache large sets of similar JSON objects.
What this is
- A collection with basic query abilities
- A collection that generates and maintains a compressed state of the collection for passing over http or similar as JSON.
- A basic cache for large amounts of similar data objects that should be exposed as JSON.
What this is not
- A collection that has a smaller memory footprint than a normal javascript array
Usage (typescript example)
// So you have some data:; // Provide a definition for the properties on the data items that should be included// in the compressed collection. Here you define how and if each property should be // compressed.; // Create a compressed collection with some (optional) initial data and the definition; // Possibly add some datacollection.add;collection.add; // Possibly remove some data... collection.removedata; // ...by referencecollection.removeByIndex3; // ...by indexcollection.removeByIndex6, 10; // ...from index to index // Retrieve the whole compressed collection...; // ...or query for a compressed subset of the data:; // Decompress compressed data ();
The definition explained
How the collection works is defined with the ICompressedJsonCollectionDefinition<T>
passed to the constructor of the CompressedCollection
.
The definition interface:
insertionHandler
The insertion handler is an optional function or object (e.g. class) that is used to evaluate each item that is added to the collection.
The insertionHandler function option: (item: T) => boolean
The function form of the insertion handler will receive the item being evaluated for insertion and is expected to return true if the item should be inserted or false if it should be rejected.
Example:
// Only allow items with a non null date property 'time' with a date succeeding 'timeThreshold';;
The insertionHandler object/class option: ICompressedJsonCollectionHandler<T>
The object/class insertionHandler
is an advanced option that comes with great freedom. It can be used to create advanced collection that processes the data on insertion in more advanced ways, like having a buffer, performing advanced preprocessing or dynamically re-encode parts of the collection on input.
Unlike the function version of the insertion handler the class/object version returns objects that should be inserted in to the collection. The handler get passed all items being inserted to the collection and should return items that should be added. If multiple items are added att the same time all of them will be sent to the handler in one array, unlike the function version that will be passed one at a time.
Example:
;
sort
The definition sort function is an optional way of turning the collection in to an ordered collection, each item added to the collection will be sorted in to place on insertion.
Example:
NOTE: Ordering can have a huge impact when performing diff based compression. When encrypting series of GPS positions for example, better compression is achieved if sorting the positions in a useful way (e.g. based on time or distance).
properties
The properties of the definitions define the properties to be included in the resulting compressed collection. The types available are defined by the interfaces:
RunLengthEncodingDefinition
The run length encoding type is useful for properties that has a limited set of possible values, like string constants or enums converted to strings or integers for example.
KeyFrameDiffEncodingDefinition
The key frame encoding type is useful for dates or numeric parameters. It achieves high compression rates when used with a fixed number of decimal digits (all numbers will be converted to decimalDigits
number of decimal digits), and is especially good for parameters with a limited numeric range. If the target parameter is a data set the type to 'date', the type 'number' is default.
RawEncodingDefinition
The raw type is useful for including parameters that should not be compressed in the resulting collection. The parameter will not be compresed or alered in anny way.