qostorage

File system based object storage, can also be used as a persistent KV database. Divided into Node implementation and Rust implementation.

Overview

• This is a file system based single-threaded object storage system. Currently, there is no multi-threading support, no read-write locks, and no isolation locks.
• The data is stored as a shard file in the file system. You can define the size of the shard. The fragment content is divided into blocks. Again, you can customize the block size.
• Equivalent to the file system, the data is divided into multiple blocks. When writing, the block index position is recorded. Deleting data marks only the block as invalid and does not delete the block data. When the next data arrives, it writes the failed block, overwriting the old data. The current version does not implement file defragmentation, so there may be invalid data fragmentation, but it does not affect normal use, but it wastes storage space.
• The system does not implement file information, just ordinary KV storage. Note that if you need to store file information, rely on other implementations to store file information.

Version

• 1.0.5

Quick start

This project contains two versions of the instance.

cd example
# Node.JS
node file.js
# Rust
cargo run

Node.JS

First, you need to install the Node.JS environment, get it from the official website, if it is already installed, please skip.

Install npm package dependencies.

npm i qostorage

This is an example code.

const qostorage = require("qostorage")
const storage = new qostorage({
  pathname: "./storage",       // Storage directory.
  chunkSize: 1024,      // File fragment size.
  blockSize: 100       // Block size.
})

process.on("beforeExit", async function () {
  void await storage.drop()
  process.exit(0)
})

storage.on("ready", async function () {
  void await storage.insert("hello", "word")
  let data = await storage.get("hello")
  data && console.log(data.toString())    // "word"
})

Rust

First, you need to install the Rust and cargo environment, get it from the official website, if it is already installed, please skip. Next, you need to introduce dependencies in the ./native folder. Package document.

extern crate qstorage;

use qstorage::QStorage;
use std::io::Error;

fn main() -> Result<(), Error> {
    let dirname = String::from("./storage");
    let mut storage = QStorage::new(dirname, 10737418240, 1048576)?;

    storage.ready()?;

    let key = String::from("hello");
    let value = "word".as_bytes().to_vec();

    storage.insert(key.clone(), value)?;
    
    if let Some(data) = storage.get(key)? {
        println!("{}", String::from_utf8_lossy(data.as_slice()));
    }

    storage.closed()?;
    Ok(())
}

API

Rust

cd native
cargo doc --open

Node.JS

new (options)

Create an instance.

• Class.
• options {object}.
• [options.pathname] {string} Storage directory.
• [options.chunkSize] {number} File fragment size.
• [options.blockSize] {number} Block size.
• return {class}

.on(event, handle)

Binding event loop.

• Function.
• event {string} Event name.
• handle {function} Callback function.
• return void

.insert(key, value)

Write key value.

• Promise.
• key {string} Key name (Special characters and symbols are not allowed).
• value {string || buffer || int array} value data.
• return Promise

.get(key)

Get key data.

• Promise.
• key {string} Key name.
• return {buffer} value data.

.remove(key)

Delete key value.

• Function.
• key {string} Key name.
• return {boolean} Whether to delete the completion.

.push(key, stream)

Write data stream.

• Promise.
• key {string} Key name.
• stream {Stream} Readable Streams.
• return Promise

.pull(key, stream)

Read data stream.

• Promise.
• key {string} Key name.
• stream {Stream} Writable Streams.
• return Promise

.drop()

Clean up before shutdown. This is a required operation. You must do the final cleanup before each shutdown, save the state, you can also call this function each time you need to save the state. This operation will write the memory data to the file system. For example, listen to the exit event of the process, and then call this function.

• Promise.
• return void.

License

MIT Copyright (c) 2019 Mr.Panda.