Welcome to the documentation for the meta-cbot
-module of SisuJs. It features two components designed
to enhance web development by improving type handling and communication between server and browser. The components are:
- Cbot (Character Based Object Transport): A type validating protocol designed for web data transport, intended to replace JSON for server-browser communication.
- Meta -model: A framework for creating reflective types in Javascript/Typescript, enhancing type enforcement and data integrity.
For more details, refer to the API documentation or contact me via sisujs@sisujs.fi. I also appreciate any feedback to make this documentation and project better.
Both Cbot and Meta-model have emerged to counter limitations of JSON and Javascript-object-model in general. And they are intented to solve couple of real limitations.
JSON (JavaScript Object Notation), is widely used as a data interchange format but it exhibits several limitations due limited types and format:
- Lack of Enforced Validation: With JSON, the burden of data validation rests entirely on the receiver. The sender can transmit any data structure, leaving the receiver to manually verify if the data conforms to expected schema.
- Inefficiency in Data Representation: JSON is not optimized for data transport. It was originally designed as a simple data notation, which becomes inefficient in transport scenarios. For example, if property names are lengthy and repetitive, they consume excessive amount of space in messages.
- Limited Data Types: JSON inherently supports only basic data types, that is, strings, numbers, booleans, objects, and arrays. Types such as dates, big integers, or custom classes require additional encoding and handling.
- Lack of Enforced Types: As Javascript objects are basically just named functions masking to be maps, there is no language level type-validation available. To achieve this, a separate meta-model or schema is required.
To address the limitations of JSON, I introduce Cbot (Character Based Object Transport), a protocol with following design choices:
- Character based protocol: Unlike many other advanced protocols such as gRPC, Cbot is character-based, making it suitable for native communication over HTTP. It operates on a line-based protocol where each line is separated by newlines and can contain one or more operations. These operations collectively construct the desired object tree.
- Enforced data types: Cbot is able to ensure that data structures adhere to a defined schema at protocol level. This ensures that recieved message is exactly what it was intended to be.
- Machine first message format: Cbot is designed to be primarly machine readable with straightforward encoding. It also minimizes message sizes with optimizing repetive property names.
-
Support for extended data types: Cbot natively supports a broader range of data types, including
Date
,BigInt
,Set
, andMap
. This eliminates the need for manual encoding and handling of complex data types. In the future it is also able to incorporate Temporal API if it is actually released.
The Meta-model serves as a framework designed to manage and enforce types within Javascript and Typescript environments, where native objects lack comprehensive reflective information. This lack limits interactions to basic operations like iterating over property names and function calls, without inherent type support. Components most important features include:
-
Complementing Cbot: The Meta-model complements Cbot by providing the necessary infrastructure to enforce and map intended types for each object during serialization and deserialization.
-
Typed objects with external APIs: When dealing with databases or third-party APIs that require JSON for data transport or storage, the Meta-model can convert typed objects to plain JavaScript objects and vice versa. This includes:
- Adaptable encoding: During the encoding process, it is possible to alter objects to ensure compatibility with JSON, such as modifying data structures or adding additional metadata necessary for reconstructing the original typed objects upon decoding. Practical examples are polymorphic objects that may be hard to handle.
- Support for extended types: The Meta-model adds support for extended datatypes which are not originally supported by JSON, although some transformations may require the use of manual encoders due the limitations of JSON.
Yes, there exists also a Java-implementation made by this project. But expanding support to include other platforms such as Rust, Python and others would be appreciated. If you are interested in developing implementations for these or any other languages, please feel free to reach out for support.
Totally, yes. These concepts have been in use already for a long time as a private project. So making this public is the next step. Also I decided to start version numbering from 0.6. This gives some room for enhancement before declaring version as 1.0.
Instead of trying to explain all aspects of the protocol, I thought it is more meaningful to just show a couple of examples which will provide a usable overview of what these features are about. This approach also gives you an easy starting point if you choose to try Cbot in your own project. And it is really simple as demonstrated below.
Cbot can be used as a simple replacement for JSON.stringify()
and JSON.parse()
. In this example I will
be modeling a library with tree concepts: Author
, Book
and Catalog
.
I will also use interfaces to clarify the model, but objects are kept untyped.
Below is the used model:
interface Author {
name: string
birthdate: Date
}
interface Book {
title: string
isbn: string
author: Author
publicationYear: number
}
interface Catalog {
genre: string
books: Book[]
authors: Set<Author>
}
Notice that because Cbot supports natively dates and sets, I just decided do use them in the model as an example.
Next, let's define some actual values for the model:
const isaac:Author = {
name: "Isaac Asimov",
birthdate: new Date("1920-01-02")
}
const foundation:Book = {
title: "Foundation",
isbn: "978-0-394-51330-7",
publicationYear: 1951,
author: isaac
}
const iRobot:Book = {
title: "I, Robot",
isbn: "978-0-394-51331-4",
publicationYear: 1950,
author: isaac
}
const catalog:Catalog[] = [
{
genre: "Science Fiction",
books: [
foundation,
iRobot
],
authors: new Set([isaac])
}
]
In order to serialize something, one needs an instance of Cbot in some static variable. You should always create just one instance for each schema and use it globally.
In this case a totally schemaless version is created.
import { Cbot } from "@sisujs/common"
const cbot = Cbot.getInstance();
Now we can actually serialize an object to a string and vice versa:
const serialized = cbot.serialize(foundation);
const deserialized:Book = cbot.deserialize(serialized);
console.log(serialized);
And the console log will produce following output:
112345abb
E
A title
B JKFoundation
A !isbn
B !JK978-0-394-51330-7
A "publicationYear
B "Id1951
A #author
B #E
A $name
B $JKIsaac Asimov
A %birthdate
B %Ih1920-01-02T01:39:49.000+01:39
F
F
Because this protocol is meant to be machine readable character stream, it is not easy to decipher what is going on. For that it needs to be visualized in more human readable format. For this there are two categories:
-
Disassembly: This will display all opcodes and values in assembly-code like format. This format may be usefull for debugging or understanding what is going on.
-
YAML: YAML is probably the visualization you actually want to use and it displays the actual object tree with values. I chose YAML over JSON because it is more expressive and it also allows comments to be added for clarification.
Also within YAML, there are options for simple and full YAML. Simple displays only data information, but full YAML show more meta-information.
So let's first take a Dissasembly-view for the output:
console.log(cbot.visualize(serialized, VisualizationMode.DISASSEMBLY));
Which outputs:
MCSM 12345abb
OBJB (plain)
DEFN 0 title
ASGV 0 (title) STRN SSTR Foundation
DEFN 1 isbn
ASGV 1 (isbn) STRN SSTR 978-0-394-51330-7
DEFN 2 publicationYear
ASGV 2 (publicationYear) NATV FLOAT64 1951
DEFN 3 author
ASGV 3 (author) OBJB (plain)
DEFN 4 name
ASGV 4 (name) STRN SSTR Isaac Asimov
DEFN 5 birthdate
ASGV 5 (birthdate) NATV ZONED_DATETIME 1920-01-02T01:39:49.000+01:39
OBJE
OBJE
What you can see here is that every line starts with a specific operation code. Let's find out what each command means:
-
MCSM
(Model Checksum)- Each message typically begins with a checksum which receiver can use to determine if sender is using same schema. If different checksum is detected then message parsing is also rejected. Checksum contains a calculation based on version number and possible schema.
-
OBJE
(Object Begin)- This opcode begins a new object. In this case a plain object is created.
-
DEFN
(Define)- This opcode creates a relation between an
id
and akey
.Id
is a number encoded to 2-character long string. Andkey
is any string that needs to be referenced later.
- This opcode creates a relation between an
-
ASGV
(Assign value)- This opcode assigns a property value to an object. In this case it has an
id = 0
that represents property nametitle
. The rest of the line tells what is the actual value.
- This opcode assigns a property value to an object. In this case it has an
-
STRN
(String)- This opcode denotes a beginning of a string. This protocol has two types of strings, which is needed because strings do not have any length restrictions and they may also contain newlines which needs to be encoded.
-
SSTR
(Simple string)- This opcode denotes a simple string that fits into one line and does not contain newlines
-
NATV
(Native value)- Denotes a beginning of a value that protocol supports natively
-
FLOAT64
- Denotes a 64-bit float value
-
ISO_DATETIME
- Denotes a ISO 8601 Date and Time -value
-
OBJE
(Object End)- Ends an object
The same can also be visualized as YAML
console.log(cbot.visualize(serialized, VisualizationMode.SIMPLE_YAML));
Which outputs:
title: Foundation
isbn: 978-0-394-51330-7
publicationYear: 1951
author:
name: Isaac Asimov
birthdate: 1920-01-02T01:39:49.000+01:39
In the disassembly view, I previously mentioned the DEFN
opcodes. These opcodes come into play when the protocol
first encounters an unknown property name. At that point, it assigns a new 2-character long id
value to that name.
And if the same property name appears again within the same message, the id
is reused during serialization.
This mechanism requires that each new message re-establish these definitions. Therefore, it can be beneficial to predefine property names in advance, allowing for the omission of these definitions during the serialization process.
And this is the version of a classless schema. Now, let's define a new instance for Cbot:
const cbot2 = Cbot.getInstance({
staticKeys: [
'name',
'birthdate',
'title',
'isbn',
'author',
'publicationYear',
'books',
'genre',
'authors'
]
});
I have now configured a set of statically defined keys which no longer require external definitions. However, this configuration does not prevent the creation of new message-based definitions for property names that aren't included in the predefined keys.
Additionally, since this configuration represents a schema-full version, both the sender and receiver must use the same configuration to ensure correct operation.
We can now use cbot2
for serialization.
console.log(cbot2.visualize(
cbot2.toString(foundation),
VisualizationMode.DISASSEMBLY));
And, its disassembly view will be as follows:
MCSM 123b13f8
OBJB (plain)
ASGV 8 (title) STRN SSTR Foundation
ASGV 5 (isbn) STRN SSTR 978-0-394-51330-7
ASGV 7 (publicationYear) NATV FLOAT64 1951
ASGV 0 (author) OBJB (plain)
ASGV 6 (name) STRN SSTR Isaac Asimov
ASGV 2 (birthdate) NATV ZONED_DATETIME 1920-01-02T01:39:49.000+01:39
OBJE
OBJE
So within this disassembly view, you can see that the Model Checksum now has a value and all DEFN
opcodes are missing.
The checksum is not intended to be particularly secure or complex; it is used more as a quick sanity check.
Finally, just for fun, let's visualize the catalog
variable in full YAML, which displays more metadata.
console.log(cbot2.visualize(
cbot2.toString(catalog),
VisualizationMode.FULL_YAML));
Which outputs:
# Model Checksum: 123b13f8
# Array
- genre: !!str Science Fiction
books:
# Array
- title: !!str Foundation
isbn: !!str 978-0-394-51330-7
publicationYear: !!float 1951
author:
name: !!str Isaac Asimov
birthdate: !!timestamp 1920-01-02T01:39:49.000+01:39
- title: !!str I, Robot
isbn: !!str 978-0-394-51331-4
publicationYear: !!float 1950
author:
name: !!str Isaac Asimov
birthdate: !!timestamp 1920-01-02T01:39:49.000+01:39
authors:
# Set
- name: !!str Isaac Asimov
birthdate: !!timestamp 1920-01-02T01:39:49.000+01:39
This example demonstrated that Cbot can serve as a zero-configuration replacement for JSON while offering the benefits of using more native types. The decision to use it, however, is yours to make.
Although Cbot can function in this manner, it is not the primary reason I have been developing this protocol for several years. Rather, the key motivation is its ability to utilize a typed schema at the protocol level itself.
This example employs the same concepts as the untyped one but with actual types. It necessitates the creation of classes and a corresponding Meta-model. While the Meta-model has its own distinct use cases, they are not within the scope of this example.
The first step is to create a namespace. This namespace does not correspond in any way to Typescript namespaces,
but it is an important tool for categorizing the Meta-model into usable groups.
For instance, all Sisu-related classes are in the namespace sisujs
.
So let's import some stuff and create a namespace.
import { Meta, Namespace, Value } from '@sisujs/common';
const NS = Namespace.of('library').init();
The init()
function ensures that this is the first time namespace for the name is declared thus preventing
namespace pollution.
And the next thing to do is to create the model:
@NS.class('Author')
class Author {
@Value.string()
name: string
@(Value.of(SisuLocalDate))
birthdate: SisuLocalDate
}
@NS.class('Book')
class Book {
@Value.string()
title: string
@Value.string()
isbn: string
@(Value.of(Author))
author: Author
@Value.int32()
publicationYear: number
@Value.number()
rating: number
}
@NS.class('Catalog')
class Catalog {
@Value.string()
genre: string
@(Value.array().of(Book))
books: Book[]
@(Value.set().of(Author))
authors: Set<Author>
}
NS.seal([
Author,
Book,
Catalog
]);
What can be observed here is that classes and all their properties require a corresponding decorator that duplicates the intent for using the meta-model. It is also entirely possible to have (transient) properties without a decorator, and these will not be included in the model.
Also, I used a int32
as the value-type for publicationYear
. Although Javascript itself does not
support integers the protocol does. and it has its own rules how to deal with the case.
The last thing is to call seal()
function for all declared types, which prevents adding further declarations.
The next step is to create the values.
// Created manually
const isaac = new Author();
isaac.name = "Isaac Asimov";
isaac.birthdate = new SisuLocalDate('1920-01-02');
// Created by (partial) template
const foundation = Meta.of(Book, {
title: "Foundation",
isbn: "978-0-394-51330-7",
publicationYear: 1951,
author: isaac,
rating: 5.6
});
const iRobot = Meta.of(Book, {
title: "I, Robot",
isbn: "978-0-394-51331-4",
publicationYear: 1950,
author: isaac
});
const catalog = [
Meta.of(Catalog, {
genre: "Science Fiction",
books: [foundation, iRobot],
authors: new Set([isaac])
})
];
This demonstrates two ways of creating typed objects, whether by manually assigning values or using the template method.
The next thing to do is to get a proper Cbot-instance:
const cbot = Cbot.getInstance({
namespaces: [NS],
nativeCoders: {
localDateCoder: SisuLocalDateNativeCoder
}
});
Now, instead of adding a staticKeys
-parameter, I've added two other options to it.
First is the namespaces
which adds the namespaces I am interested in.
This illustrates why namespacing is important. You may have types for multiple purposes, and
you may not want to expose them all publicly. In such cases, a different namespace can be used
for the private types.
The other option is nativeCoders
which adds the ability to process those types that are not
natively supported by javascript. In this case in the model i have Author.birthdate
which has a type of SisuLocalDate
. So I also need to inform Cbot-instance that this type
should act as native localdate-type during serialization.
And now we are ready to begin the serialization process. There are no significant differences here compared to the untyped case, so I will simply demonstrate what disassembly and full YAML representation reveals.
First with disassembly:
console.log(cbot.visualize(
cbot.serialize(foundation),
VisualizationMode.DISASSEMBLY));
Which outputs:
MCSM 12466605
OBJB 7 (library.Book)
ASGV 0 (author) OBJB 6 (library.Author)
ASGV 2 (birthdate) NATV LOCAL_DATE 1920-01-02
ASGV 9 (name) STRN SSTR Isaac Asimov
OBJE
ASGV 5 (isbn) STRN SSTR 978-0-394-51330-7
ASGV 10 (publicationYear) NATV INT32 1951
ASGV 11 (rating) NATV FLOAT64 5.6
ASGV 12 (title) STRN SSTR Foundation
OBJE
The output is practically the same as before. The difference here is that now objects have also an id
, which corresponds to the
typename for the object.
And lastly Yaml:
console.log(cbot.visualize(
cbot.serialize(catalog),
VisualizationMode.FULL_YAML));
Which outputs:
# Model Checksum: 12466605
# Array
- # library.Catalog
authors:
# Set
- # library.Author
birthdate: !!timestamp 1920-01-02
name: !!str Isaac Asimov
books:
# Array
- # library.Book
author:
# library.Author
birthdate: !!timestamp 1920-01-02
name: !!str Isaac Asimov
isbn: !!str 978-0-394-51330-7
publicationYear: !!int 1951
rating: !!float 5.6
title: !!str Foundation
- # library.Book
author:
# library.Author
birthdate: !!timestamp 1920-01-02
name: !!str Isaac Asimov
isbn: !!str 978-0-394-51331-4
publicationYear: !!int 1950
rating: null
title: !!str I, Robot
genre: !!str Science Fiction
Again, the main difference here is that representation contains more comments to specify correct types.
And that's it! There is practically nothing more to show here. These examples are fully usable for you project as is and the API documentation contains more information. And do provide feedback to improve documentation further.
Although, there is one more thing for the old school guys. You can also create Meta-model totally manually without decorators like this:
var NS = Namespace.of('library');
function Author() {
this.name = null;
this.birthdate = null;
}
Value.string()(Author, 'name');
Value.date()(Author, 'birthdate');
NS.class('Author')(Author);