crud-benchmark

Overview

Crud Benchmark is a tool to benchmark and compare different microservices/backend technologies by applying bulk crud operations and calculating the fastest one, the tool only measures the response time and doesn't have any measurement of memory or any hardware usage. The generates the results as both command line and HTML file that can you view in your browser. To run this tool you only need NodeJS to be installed on your machine

When to use it

you can use this tool if you need to comapre any backend related technologies based on response time, like comparing database engines, backend frameworks, backend servers or even different database drivers or code styles, all you need to do is defining the right endpoints in your backend apps and run the benchmark tool on them

Example Usecase

lets assume you have a requirement of writing a backend microservice that applies CRUD operations on a mysql database and you are not sure which techonology is faster, Nodejs or Java Springboot, you need apply these step:

• Create a Nodejs app a Springboot App where of them has these endpoints, you can give your endpoints any names while the functionality and httpmethod should be same as whats written here

Endpoint	HTTP Method	Functionality
/product/random	POST	Endpoint has no request body or params, once requested it auto generates a product and insert it into database then return the full product in the response body
/product	GET	Lists all products from the database
/product	PUT	Update product in the database by have a full product as request body
/product/{productId}	DELETE	deletes a product by id

and here is a smaple swagger definition for these endpoints

{
  "swagger": "2.0",
  "info": {
    "description": "Sample Endpoints for CRUD-BENCHMARK",
    "version": "1.0.0",
    "title": "CRUD-BENCHMARK Endpoints",
    "contact": {
      "email": "[e-mail]"
    }
  },
  "schemes": [
    "http"
  ],
  "paths": {
    "/product/random": {
      "post": {
        "tags": [
          "Product"
        ],
        "summary": "Add a new random generated product to the database and return in in response body",
        "operationId": "addProduct",
        "consumes": [
          "application/json"
        ],
        "produces": [
          "application/json"
        ],
        "parameters": [],
        "responses": {
          "200": {
            "description": "returns  full product object",
            "schema": {
              "$ref": "#/definitions/Product"
            }
          }
        }
      }
    },
    "/product": {
      "put": {
        "tags": [
          "Product"
        ],
        "summary": "Update an existing product",
        "description": "",
        "operationId": "updateProduct",
        "consumes": [
          "application/json"
        ],
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "in": "body",
            "name": "body",
            "description": "Product object that needs to be yodated",
            "required": true,
            "schema": {
              "$ref": "#/definitions/Product"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "returns full updated product",
            "schema": {
              "$ref": "#/definitions/Product"
            }
          }
        }
      },
      "get": {
        "tags": [
          "Product"
        ],
        "summary": "List all products from database",
        "operationId": "findProudcts",
        "produces": [
          "application/json"
        ],
        "responses": {
          "200": {
            "description": "return array of all products",
            "schema": {
              "type": "array",
              "items": {
                "$ref": "#/definitions/Product"
              }
            }
          }
        }
      }
    },
    "/product/{productId}": {
      "delete": {
        "tags": [
          "Product"
        ],
        "summary": "Deletes a pet",
        "description": "",
        "operationId": "deleteProduct",
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "name": "productId",
            "in": "path",
            "description": "Product id to delete",
            "required": true,
            "type": "integer"
          }
        ],
        "responses": {
          "200": {
            "description": "Product delete success returns product id"
          }
        }
      }
    }
  },
  "definitions": {
    "Product": {
      "type": "object",
      "properties": {
        "id": {
          "type": "integer"
        },
        "name": {
          "type": "string"
        }
      }
    }
  }
}

• After building these endpoints in both Node and Sprongboot we are ready start our benchmark tool but at first we need to give it the rigt unputs, create on your machine a json file env.json with the following content

{
  "numberOfCRUDRequests" : 1000,
  "methods": {
    "CREATE": "/product/random",
    "READ": "/product",
    "UPDATE": "/product",
    "DELETE": "/product"
  },
  "crudServers" :[

    {
      "name": "Node",
      "baseURL": "http://127.0.0.1:8080"
    },
       {
      "name": "Springboot",
      "baseURL": "http://127.0.0.1:8081"
    }
  ]
}

in this file we defined the required where we tell the tool how many requests to apply which is 1000, the endpoint for each method, and the apps we need to test which is the crudServers array

• install the tool

npm i -g crud-benchmark

now you can run the benchmark crud tool by the command

crud-benchmark.js -f env.json

the result is a report in your command like window like this where all units all milliseconds

{
  Node: {
    CREATE: { min: 23, max: 624, average: 102.2 },
    READ: { min: 28, max: 274, average: 69.5 },
    UPDATE: { min: 24, max: 142, average: 54.1 },
    DELETE: { min: 25, max: 88, average: 47.4 }
  },
    Springboot: {
    CREATE: { min: 22, max: 623, average: 101 },
    READ: { min: 27, max: 273, average: 67.5 },
    UPDATE: { min: 23, max: 141, average: 52.3 },
    DELETE: { min: 24, max: 87, average: 46.1 }
  }
}

also it generates HTML version of the report crudreport.html that is created in the same path of where you run the command line, the html report has a table similar to this

Method	Best Server	Node	Springboot
CREATE	Springboot	MIN: 23 MAX: 624 AVG: 102.2	MIN: 22 MAX: 623 AVG: 101
READ	Springboot	MIN: 28 MAX: 274 AVG: 69.5	MIN: 27 MAX: 273 AVG: 67.5
UPDATE	Springboot	MIN: 24 MAX: 142 AVG: 54.1	MIN: 23 MAX: 141 AVG: 52.3
DELETE	Springboot	MIN: 25 MAX: 88 AVG: 47.4	MIN: 24 MAX: 87 AVG: 46.1

How it works

For each app in the crudServers the tool is doing the following:

• The tool starts with the CREATE method and call it 1000 times which will create new products in your database and it will collect the result of each request in an array , so we have an array of all created products.

• The tool calls the READ for 1000 times and each time it's fetching 1000 records that were generated by the CREATE

• The tool calls the UPDATE for 1000 times with they payload from the saved users array

• The tool calls the DELETE for 1000 times with ids from the users array

• The tool takes the current timestamp in milliseconds before and after each request call and calucaltes the difference, then it calcaulates the min,max and average for each request, after that it generates both the cli and html report

• if you follow the right endpoints schemas then the tool will create the 1000 records in the begining of the benchmark during the CREATE process then by the end it's the deleteing the same 1000 records during the DELETE process, so it guarantess a clean database by the end of the benchmark.

  Do is require full CRUD ?

  the above example shows using the tool with full CRUD opration, but what if you only need to benchmark Read?, its possible to do by only defining READ in your json file and removing the other methods, same also applies on the CREATE as there logics are standalone while the UPDATE and DELETE depends on the results from the CREATE, so without the CREATE you can't benchmark update and delete