muschema
An extensible system for defining schemas, which describe structures of the data.
Schemas allow run-time type reflection, object pooling, and more importantly, binary serialization. And in mudb
, all message interfaces are specified by schemas.
Compared to protobuf, muschema
is better in that it supports delta encoding and is easier to customize (and worse in the sense that it only works in JavaScript).
TypeScript and Node.js friendly!
example
Here is a contrived example showing how all of the methods of the schemas work.
const MuFloat64 MuInt32 MuString MuDictionary MuStruct} = const MuWriteStream MuReadStream = // define an entity schemaconst EntitySchema = x: y: dx: dy: hp: 10 name: 'entity' // define an entity set schemaconst EntitySetSchema = EntitySchema // create a new entity set object using the schemaconst entities = EntitySetSchema // create a new entity and add it to the schemaconst player = EntitySchema playerx = 10playery = 10playerdx = -10playerdy = -20playername = 'winnie' entities'pooh' = player // make a copy of all entitiesconst otherEntities = EntitySetSchema // modify player entityotherEntitiesfoohp = 1 // compute a patch and write it to streamconst out = 32const hasPatch = EntitySetSchema let otherEntitiesCopy = EntitySetSchemaif hasPatch // read the patch from stream and apply it to // a copy of entities const inp = out otherEntitiesCopy = EntitySetSchema // pool objectsEntitySetSchema
table of contents
1 install
npm i muschema
2 api
2.1 interface
Each schema should implement the MuSchema
interface:
identity
the default value of the schemamuType
a string of type name for run-time reflectionmuData
(optional) additional run-time information, usually the schema of membersalloc()
creates a new value from scratch, or fetches a value from the object poolfree(value)
recycles the value to the object poolequal(a, b)
determines whether two values are equalclone(value)
duplicates the valuecopy(source, target)
copies the content ofsource
totarget
diff(base, target, outStream)
computes a patch frombase
totarget
patch(base, inpStream)
applies a patch tobase
to create a new value
Methods should obey the following semantics.
=== !
=== true
=== true
For situations where you don't have a base,
schemaschema
Schemas can be composed recursively by calling submethods. muschema
provides several common schemas for primitive types and some functions for combining them together into structs, tuples and other common data structures. If necessary user-defined applications can specify custom serialization and diff/patch methods for various common types.
for TypeScript
For TypeScript, the generic interface described above can be found in muschema/schema
. The module exports the interface as MuSchema<ValueType>
, which any schema types should implement.
2.2 primitives
muschema
comes with schema types for all primitive types in JavaScript out of the box.
2.2.1 void
An empty value type. Useful for specifying arguments to messages which do not need to be serialized.
const MuVoid = const EmptySchema = EmptySchemaidentity // always undefinedEmptySchemamuType // 'void' const nothingness = EmptySchema // undefinedEmptySchema // noopEmptySchema // always returns undefined
2.2.2 boolean
true
or false
const MuBoolean = const SwitchSchema = identity SwitchSchemaidentity // defaults to false if not specifiedSwitchSchemamuType // 'boolean' const switch = SwitchSchema // equals identitySwitchSchema
2.2.3 number
// for signed integers of 8/16/32-bitconst MuInt8 = const MuInt16 = const MuInt32 = // for unsigned integers of 8/16/32-bitconst MuUint8 = const MuUint16 = const MuUint32 = // for floating point of 32/64-bitconst MuFloat32 = const MuFloat64 = // here MuNumber stands for any of the number schema typesconst AnyNumberSchema = identity AnyNumberSchemaidentity // defaults to 0 if not specifiedAnyNumberSchemamuType // string of one of int8/int16/int32/uint8/uint16/uint32/float32/float64 // depending on the schema type const num = AnyNumberSchema // equals identityAnyNumberSchema // noopAnyNumberSchema // returns the value of `num`
- for numbers in general, use
MuFloat64
- but if you know the range of the numbers in advance, use a more specific data type instead
2.2.4 string
const MuString = const MuASCII = const MuFixedASCII = const MessageSchema = identityMessageSchemaidentity // defaults to '' if not specifiedMessageSchemamuType // 'string' const msg = MessageSchema // equals identityMessageSchema // noopMessageSchema // returns the value of `msg` const UsernameSchema = identityUsernameSchemaidentity // defaults to '' if not specifiedUsernameSchemamuType // 'ascii' const username = UsernameSchema // equals identityUsernameSchema // noopUsernameSchema // returns the value of `username` // for this schema type, you must either specify the identityconst phoneNumberSchema = '1234567890'phoneNumberSchemaidentity // '1234567890'phoneNumberSchemamuType // 'fixed-ascii'phoneNumberSchemalength // 10, the length of all strings in this schemaconst phone = phoneNumberSchema // '1234567890' // or the fixed lengthconst IDSchema = 8IDSchemaidentity // a string of 8 spacesIDSchemalength // 8 const id = IDSchema // a string of 8 spacesIDSchema // noopIDSchema // returns the value of `id`
- for strings in general, use
MuString
- if the strings consist of only ASCII characters, use
MuASCII
- if the strings consist of only ASCII characters and are of the same length, use
MuFixedASCII
instead
2.3 functors
Primitive data types in muschema
can be composed using functors. These take in multiple sub-schemas and construct new schemas.
2.3.1 struct
A struct is a collection of subtypes. Structs are constructed by passing in a dictionary of schemas. Struct schemas may be nested as follows:
const MuFloat64 = const MuStruct = const Vec2 = x: 0 y: 0const Particle = position: Vec2 velocity: Vec2 const p = Particleppositionx = 10ppositiony = 10 // Particle.free recursively calls Vec2.freeParticle
2.3.2 array
const MuStruct = const MuArray = const MuUint32 = const SlotSchema = item_id: amount: const InventorySchema = SlotSchema identity InventorySchemaidentity // defaults to [] if not specifiedInventorySchemamuType // 'array'InventorySchemamuData // SlotSchema const backpack = InventorySchema // always []InventorySchema // pools `backpack` and all its membersInventorySchema // returns a deep copy of `backpack`
2.3.3 sorted array
const MuStruct = const MuSortedArray = const MuUint8 = { if arank < brank return -1 else if arank > brank return 1 if asuit < bsuit return -1 else if asuit > bsuit return 1 else return 0 } const CardSchema = suit: rank: const DeckSchema = CardSchema compare identity DeckSchemaidentity // defaults to [] // if identity specified, will be a sorted copy of itDeckSchemamuType // 'sorted-set'DeckSchemamuData // CardSchemaDeckSchemacompare // reference to the compare function const deck = DeckSchema // always []DeckSchema // pools `deck` and all its membersDeckSchema // returns a deep copy of `deck`
2.3.4 union
A discriminated union of several subtypes. Each subtype must be given a label.
const MuFloat64 = const MuString = const MuUnion = const MuWriteStream MuReadStream = const FloatOrString = float: 'foo' string: 'bar' // create a new valueconst x = FloatOrStringxtype = 'float'xdata = 1 // compute a delta and write it to streamconst out = 32FloatOrString // apply a patchconst inp = outbufferuint32const y = FloatOrString
2.4 data structures
2.4.1 dictionary
A dictionary is a labelled collection of values.
const MuUint32 = const MuDictionary = const NumberDictionary = identityNumberDictionaryidentity // defaults to {} if not specifiedNumberDictionarymuType // 'dictionary'NumberDictionarymuData // a MuUint32 schema const dict = NumberDictionarydict'foo' = 3 NumberDictionary // pools `dict` and all its membersNumberDictionary // returns a deep copy of `dict`
2.4.2 vector
const MuVector = const MuFloat32 = const ColorSchema = 4ColorSchemaidentity // Float32Array [0, 0, 0, 0]ColorSchemamuType // 'vector'ColorSchemamuData // reference to the specified MuFloat32 schemaColorSchemadimension // 4 const rgba = ColorSchema // Float32Array [0, 0, 0, 0]ColorSchema // pools `rgba`ColorSchema // returns a copy of `rgba`
3 more examples
Check out mudb
for some examples of using muschema
.
TODO
3.1 features
- smarter delta encoding
- memory pool stats
3.2 schema types
- fixed point numbers
- enums
- tuples
- multidimensional arrays
3.3 TBD
- should models define constructors?
- should pool allocation be optional?
- some types don't need a pool
- pooled allocation can be cumbersome
- do we need JSON and RPC serialization for debugging?
credits
Copyright (c) 2017 Mikola Lysenko, Shenzhen Dianmao Technology Company Limited