TeDB-Electron-Storage
A storage driver for electron using TeDB as the the datastore.
Installation
$ npm install --save tedb-electron-storage>$ yarn add tedb-electron-storage
Usage
TeDB-Electron-Storage is a storage driver for TeDB that interacts with the base file system of Linux/Mac(OS)/Windows Desktop file systems to save and retrieve data. Together TeDB and this package make a MongoDB like database to save, retrieve, and edit data safely and quickly. This is a persist only storage driver meaning that the data does not live on in memory. It simply retrieves and returns data read from files off the files system.
This storage driver also makes a backup of every file that is created so if for instance an error occurs during a write and the data file is lost there is a backup of its previous state. Every write is "crash proof" write, and should resist this possibility. This also goes for all indices that are persisted and saved.
It is highly recommended to persist indices often. If not then a good way to remove errors of not having correct indices is to use the sanitize methods on TeDB. These methods prevent indices to exist for files that do not exist and for files to not exist if they are not found in the indices. However these sanitize methods will do nothing if there is no index for the collection.
Since this package relies highly on NodeJS's FS module much of the fs methods were written into promises to make use of promise methodologies. Since many instances needed custom work the use of a 3rd party promise fs library was not used.
// ES6 options and available extensions;
Exports with the key term safe
will not reject and instead return false
for mistakes. This is useful in the package for cross platform errors such as reading files that do not exist. More on the files error handling could be implemented in the future.
Table of Contents
- Installation
- Usage
- Storage Driver Upfront Info
- Using TeDB and TeDB-Electron-Storage in Tandem
- Creating a DB and Collection
- Utils and Benefits of this Storage Driver
Storage Driver Upfront Info
Since the use of this package is through TeDB then the explanation will be examples of using TeDB to create a meaningful api for your collections. This explanation will also be written in TypeScript; if you are using regular Javascript, nothing will change except possibly needing to downgrade the ES version to ES5 and removing the type annotations which come after a :
.
It is also important to know the limitations of this storage driver. Currently there is no insert buffer for inserting large amounts of documents. Since each document has its own file this takes time for the OS to create the files. To insert 10k items on my 2015 mac took 13 seconds. However all other operations on 100k collections took under 50ms even if there was no index. With indices however you will get results for a find/update/remove within 1-2ms depending on how many fields are searched. For each key in a query a search is a composed. Then cross referenced and compacted down to the remaining results. So the less keys in the query the faster the search.
As for the saving location on your desktop, you can check out the AppDirectory directory and read the index file.
- For Mac your data will be saved at
user/Library/ApplicationSupport/collectionName
. - For Windows
user\AppData\Local\collectionName
. - And for linux
user/local/share/collectionName
.
There you can query your data within that directory. A db for your application might look like
> collectionName -> name of your db | > db -> db dir for possible db duplications or other version extensions in the future . | . > users -> collection you named and pushed data to . . | . . > username_index.db . . > NW03UVRHQUJBQUE9T3IyT0xTK2FlaTQ9T2FOVEdxcE5lRG89aWowa0NLS3pVdWM9.db . . > ... . . > `v0.0.1 -> version directory . . . | . . . > states -> another sub directory for version updates # note below about this . . . . | . . . . > NW03UVRHQUJBQUE9T3IyT0xTK2FlaTQ9T2FOVEdxcE5lRG89aWowa0NLS3pVdWM9 -> dir of backup of doc . . . . . | . . . . . > past -> file containing previous state of doc . . . . > username_index -> dir of backup of index . . . . . | . . . . . > past -> file containing previous state of index . . . . > otherBackups... . > otherCollections...
Now about why the version in the db directory is not the same as the package version. This is to possibly allow me to update the backup directories methodology without affecting your base data. If you update this package and see that the version number changed for that directory then there was a breaking change to how the data is laid out in the older version to the current. Items in the version directory are for the use of the database would recommend against tampering with the backup data. This does mean that your data is backup up and that your data is duplicated on disk.
I might work in a way in the future to opt out of this choice but it is an extra safety measure for lost data when desktops randomly crash and there is not time to finish the current write. Write to files overwrite completely. This means every update will overwrite the file and the past info lives on in the backup until the next update. This package does make use of the graceful-fs as dependency preventing many common errors with file accessing. So if you notice that a find is taking a very long time possibly you have no index and a very large query with many many keys. This will open up many files if your collection is very large and will bottleneck the IO. Graceful-fs will convert the async nature of this package to synchronous if to many files are being opened at once. Having indices will prevent this from happening.
If you are wondering why the choice of singe files for documents was made for this package check out the information given in the parent package TeDB writing a storage driver.
Using TeDB and TeDB-Electron-Storage in Tandem
It is important to understand that TeDB is simply the in memory handler of the indices and _ids which are the keys to files in this package; and that TeDB-Electron-Storage is used for file IO operations. Using both together in certain circumstances and at the right time is key to creating a robust api for your DB. During development it is useful to have the Storage driver's methods available for debugging your DB. However it is not recommended unless completely confident in your ability to use the Storage Driver methods except for the removeIndex
method.
Creating a DB and Collection
To view the the methods available to you visit the types/index
file and look through methods and their descriptions. They should explain their purpose. If not this go through of a good DB collection should be useful.
Here I will create class to describe a collection and what DB it should be apart of. Along with promise based events.
// imports;;//// type and default for event;;;;;;;;;;;;//// class interface //// Events for collection//// class
If You would like after using the this.Datastore.getIndices
method it returns the Map of indices you may choose to use this to loop over the indices
.then
Here for example I will show the method for a Users
collection.
; // DB is where the DB above is ; ;;;//// promise for event fired;;//// adding event to the collectionUser.setEventPRESYNC, promise1;User.setEventPOSTSYNC, promise2;// //;
Utils and Benefits of this Storage Driver
The utilities are very much tied to this package and don't see much being updated to them unless more error handling fixes come about. Some methods will be removed from this package and other packages under the tedb-org to create a new tedb-org/tedb-utils package in the future which will have all the utils and their tests all on one localized place.
The benefits of using this package and TeDB are that it is written in TypeScript and all the types are available. Also TypeScript can be used to create an interface
or type
that acts as a schema for your collection. Once you query data you simply state the return type as the type/interface
you declared. TeDB and this package will store your object exactly as you send it so there are no modifications to your object during save.
There is one thing I have to say about this package and that is don't blame the package right away for errors and issues. When creating your own database logically think about the way it is functioning before submitting an error.
This package and TeDB are both being used in production for Electron. As issues come up we will be available to discussion and fixes if need be. If this package does not suite your needs please create your own and make a pull request to TeDB to add it to our list of Storage Drivers. A React Native storage driver is in the future so keep an eye out.