Nachos Pillage Milwaukee

    dynamodb-spec-generator

    0.0.7 • Public • Published

    dynamodb-spec-generator Badge Version 0.0.1

    A command line tool to generate documentation for a DynamoDB table and its access patterns, to assign in the design phase of a well-factored DynamoDB table

    Table of Contents

    Purpose

    Designing a DynamoDB table is very different than a SQL database because one must decide upfront what access patterns the table needs to support and pre-denormalize the data.

    Additionally, a well-factored DynamoDB table, with overloading GSIs, sparse indexes, local secondary indexes, composite keys (etc.) can be quite hard to document and understand.

    dynamodb-spec-generator was created to help alleviate this issue, to make it easier to design a DynamoDB according to the access patterns. All you have to do is provide it with a valid DynamoDB spec file and it will output a markdown document.

    Installation

    dynamodb-spec-generator is packaged as an npm package and requires node >= 8.0.0:

    # install the command globally 
    $ npm install dynamodb-spec-generator -g

    Alternatively, if you are working inside a nodejs project, you can add it as a dependency:

    # Install it as a development dependency 
    $ npm install dynamodb-spec-generator --save-dev

    If you install it as a dependency, make sure to run it with npx like so:

    $ npx dynamodb-spec-generator

    Usage

    dynamodb-spec-generator takes a single argument, the path to the spec file, and will output the markdown to stdout:

    $ npx dynamodb-spec-generator spec.json

    If you want to instead save it to a file, pass the -o flag:

    $ npx dynamodb-spec-generator spec.json -o README.md

    You can also have it watch for changes to the spec file and regenerate the markdown whenever you make a change by passing the --watch flag:

    $ npx dynamodb-spec-generator spec.json -o README.md --watch

    Examples

    Relational Modeling

    This example attempts to recreate the DynamoDB docs for converting a normalized SQL relational database into a single DynamoDB table with two Global Secondary Indexes.

    The spec file relationalModeling.json
    {
      "service": "HR Api Backend",
      "version": "1.0.0",
      "description": "A recreation of the Relational Modeling example in the DynamoDB Docs",
      "author": "Eric Allam",
      "tableDefinition": {
        "TableName": "HR-Table",
        "KeySchema": [
          {
            "AttributeName": "PK",
            "KeyType": "HASH"
          },
          {
            "AttributeName": "SK",
            "KeyType": "RANGE"
          }
        ],
        "AttributeDefinitions": [
          {
            "AttributeName": "PK",
            "AttributeType": "S"
          },
          {
            "AttributeName": "SK",
            "AttributeType": "S"
          },
          {
            "AttributeName": "Data",
            "AttributeType": "S"
          }
        ],
        "ProvisionedThroughput": {
          "ReadCapacityUnits": 5,
          "WriteCapacityUnits": 5
        },
        "BillingMode": "PAY_PER_REQUEST",
        "GlobalSecondaryIndexes": [
          {
            "IndexName": "gsi1",
            "KeySchema": [
              {
                "AttributeName": "SK",
                "KeyType": "HASH"
              },
              {
                "AttributeName": "Data",
                "KeyType": "RANGE"
              }
            ],
            "Projection": {
              "ProjectionType": "ALL"
            },
            "ProvisionedThroughput": {
              "ReadCapacityUnits": 5,
              "WriteCapacityUnits": 5
            }
          },
          {
            "IndexName": "gsi2",
            "KeySchema": [
              {
                "AttributeName": "GSI-Bucket",
                "KeyType": "HASH"
              },
              {
                "AttributeName": "Data",
                "KeyType": "RANGE"
              }
            ],
            "Projection": {
              "ProjectionType": "ALL"
            },
            "ProvisionedThroughput": {
              "ReadCapacityUnits": 5,
              "WriteCapacityUnits": 5
            }
          }
        ]
      },
      "accessPatterns": [
        {
          "title": "Look up Employee Details by Employee ID",
          "type": "get",
          "index": "main",
          "params": {
            "partition": "HR-EMPLOYEE1",
            "sort": { "value": "EMPLOYEE1", "operator": "=" }
          }
        },
        {
          "title": "Query Employee Details by Employee Name",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": { "value": "EMPLOYEE1" },
            "sort": { "value": "John", "operator": "begins_with" }
          }
        },
        {
          "title": "Get an employee's current job details only",
          "index": "main",
          "type": "query",
          "params": {
            "partition": { "value": "HR-EMPLOYEE1" },
            "sort": { "value": "v0", "operator": "begins_with" }
          }
        },
        {
          "title": "Get Open Orders for a customer for a date range",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": { "value": "CUSTOMER1" },
            "sort": { "value": "OPEN#2019-01", "operator": "begins_with" }
          }
        },
        {
          "title": "Show all Orders in OPEN status for a date range across all customers",
          "description": "Query in parallel for the range [0..N] to get all shards",
          "type": "query",
          "index": "gsi2",
          "params": {
            "partition": "Bucket-6",
            "sort": {
              "operator": "between",
              "minValue": "OPEN#2019-01",
              "maxValue": "OPEN#2019-02"
            }
          }
        },
        {
          "title": "All Employees Hired Recently",
          "index": "gsi1",
          "type": "query",
          "params": {
            "partition": "HR-CONFIDENTIAL",
            "sort": { "value": "2019-01-01", "operator": ">" }
          }
        },
        {
          "title": "Find all employees in a certain warehouse",
          "index": "gsi1",
          "type": "query",
          "params": {
            "partition": "WAREHOUSE1"
          }
        },
        {
          "title": "Get all OrderItems for a Product including warehouse location inventories",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": "PRODUCT1",
            "filters": [
              {
                "attribute": "Warehouse1",
                "operator": "attribute_exists"
              }
            ]
          }
        },
        {
          "title": "Get customers by Account Rep",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": "EMPLOYEE1",
            "filters": [
              {
                "attribute": "Address",
                "operator": "attribute_exists"
              }
            ]
          }
        },
        {
          "title": "Get orders by Account Rep and date",
          "description": "Scatter/Gather to query all statuses (OPEN, PENDING, FULFILLED)",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": "EMPLOYEE1",
            "sort": { "value": "OPEN#2019-01-12", "operator": "=" }
          }
        },
        {
          "title": "Get all employees with specific Job Title",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": "v0"
          }
        },
        {
          "title": "Get inventory by Product and Warehouse",
          "type": "get",
          "index": "main",
          "params": {
            "partition": "OE-PRODUCT1",
            "sort": { "value": "PRODUCT1", "operator": "=" }
          }
        },
        {
          "title": "Get total product inventory",
          "type": "get",
          "index": "main",
          "params": {
            "partition": "OE-PRODUCT1",
            "sort": {
              "value": "PRODUCT1",
              "operator": "="
            }
          }
        },
        {
          "title": "Get Account Reps ranked by Order Total and Sales Period",
          "type": "query",
          "index": "gsi1",
          "params": {
            "partition": "2018-Q4",
            "order": "DESC"
          }
        }
      ],
      "records": [
        {
          "PK": "HR-EMPLOYEE1",
          "SK": "EMPLOYEE1",
          "Data": "John Smith",
          "StartDate": "01-12-2019"
        },
        { "PK": "HR-EMPLOYEE1", "SK": "v0", "Data": "Principle Account Manager" },
        {
          "PK": "HR-EMPLOYEE1",
          "SK": "HR-CONFIDENTIAL",
          "Data": "2019-02-12",
          "Employee": "John Smith",
          "Salary": 50000
        },
        {
          "PK": "HR-EMPLOYEE1",
          "SK": "WAREHOUSE1",
          "Data": "2019-02-15",
          "Employee Name": "John Smith"
        },
        {
          "PK": "HR-EMPLOYEE1",
          "SK": "2018-Q4",
          "Data": "5000",
          "Employee Name": "John Smith"
        },
        {
          "PK": "HR-EMPLOYEE2",
          "SK": "2018-Q4",
          "Data": "10000",
          "Employee Name": "John Smith"
        },
        {
          "PK": "OE-ORDER1",
          "SK": "CUSTOMER1",
          "Data": "OPEN#2019-01-18",
          "GSI-Bucket": "Bucket-6"
        },
        {
          "PK": "OE-ORDER1",
          "SK": "PRODUCT1",
          "Data": "OPEN#2019-01-18",
          "GSI-Bucket": "Bucket-4",
          "UnitPrice": "$89.99"
        },
        {
          "PK": "OE-ORDER1",
          "SK": "EMPLOYEE1",
          "Data": "OPEN#2019-01-12",
          "OrderTotal": 2500
        },
        {
          "PK": "OE-PRODUCT1",
          "SK": "PRODUCT1",
          "Data": "Quickcrete Cement - 50lb bag",
          "Warehouse1": 46,
          "Warehouse2": 12
        },
        {
          "PK": "OE-CUSTOMER1",
          "SK": "CUSTOMER1",
          "Data": "Ace Building Supplies",
          "Address": "1600 Penn"
        },
        {
          "PK": "OE-CUSTOMER1",
          "SK": "EMPLOYEE1",
          "Data": "Ace Building Supplies",
          "Address": "1600 Penn"
        }
      ]
    }
    Will generate this markdown:

    HR Api Backend DynamoDB Spec

    A recreation of the Relational Modeling example in the DynamoDB Docs

    Table of Contents

    Table Spec

    Params to create the table using the CLI or the AWS SDK:
    {
      "TableName": "HR-Table",
      "KeySchema": [
        {
          "AttributeName": "PK",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "SK",
          "KeyType": "RANGE"
        }
      ],
      "AttributeDefinitions": [
        {
          "AttributeName": "PK",
          "AttributeType": "S"
        },
        {
          "AttributeName": "SK",
          "AttributeType": "S"
        },
        {
          "AttributeName": "Data",
          "AttributeType": "S"
        }
      ],
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 5,
        "WriteCapacityUnits": 5
      },
      "BillingMode": "PAY_PER_REQUEST",
      "GlobalSecondaryIndexes": [
        {
          "IndexName": "gsi1",
          "KeySchema": [
            {
              "AttributeName": "SK",
              "KeyType": "HASH"
            },
            {
              "AttributeName": "Data",
              "KeyType": "RANGE"
            }
          ],
          "Projection": {
            "ProjectionType": "ALL"
          },
          "ProvisionedThroughput": {
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
          }
        },
        {
          "IndexName": "gsi2",
          "KeySchema": [
            {
              "AttributeName": "GSI-Bucket",
              "KeyType": "HASH"
            },
            {
              "AttributeName": "Data",
              "KeyType": "RANGE"
            }
          ],
          "Projection": {
            "ProjectionType": "ALL"
          },
          "ProvisionedThroughput": {
            "ReadCapacityUnits": 5,
            "WriteCapacityUnits": 5
          }
        }
      ]
    }

    createTable Using the CLI:

    $ aws dynamodb create-table --table-name HR-Table --cli-input-json create-table.json

    Using the AWS SDK:

    const DynamoDB = require("aws-sdk/clients/dynamodb");
     
    const service = new DynamoDB({ region: process.env.AWS_REGION });
     
    service.createTable(tableJson, (err, data) => {
      console.log(data);
    });

    Access Patterns

    Look up Employee Details by Employee ID

    Perform a DocumentClient.get against the Main index:

    {
      "TableName": "HR-Table",
      "Key": {
        "PK": "HR-EMPLOYEE1",
        "SK": "EMPLOYEE1"
      }
    }

    Matching Records

    PK (HASH) SK (RANGE)
    HR-EMPLOYEE1 EMPLOYEE1 Data: John Smith StartDate: 01-12-2019

    Query Employee Details by Employee Name

    Perform a DocumentClient.query against the Gsi1 index with a begins_with(#Data, :Data) condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK and begins_with(#Data, :Data)",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Data": "Data"
      },
      "ExpressionAttributeValues": {
        ":SK": "EMPLOYEE1",
        ":Data": "John"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    EMPLOYEE1 John Smith PK: HR-EMPLOYEE1 StartDate: 01-12-2019

    Get an employee's current job details only

    Perform a DocumentClient.query against the Main index with a begins_with(#SK, :SK) condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#PK = :PK and begins_with(#SK, :SK)",
      "ExpressionAttributeNames": {
        "#PK": "PK",
        "#SK": "SK"
      },
      "ExpressionAttributeValues": {
        ":PK": "HR-EMPLOYEE1",
        ":SK": "v0"
      }
    }

    Matching Records

    PK (HASH) SK (RANGE)
    HR-EMPLOYEE1 v0 Data: Principle Account Manager

    Get Open Orders for a customer for a date range

    Perform a DocumentClient.query against the Gsi1 index with a begins_with(#Data, :Data) condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK and begins_with(#Data, :Data)",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Data": "Data"
      },
      "ExpressionAttributeValues": {
        ":SK": "CUSTOMER1",
        ":Data": "OPEN#2019-01"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    CUSTOMER1 OPEN#2019-01-18 PK: OE-ORDER1 GSI-Bucket: Bucket-6

    Show all Orders in OPEN status for a date range across all customers

    Query in parallel for the range [0..N] to get all shards

    Perform a DocumentClient.query against the Gsi2 index with a #Data BETWEEN :DataMin AND :DataMax condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#GSI-Bucket = :GSI-Bucket and #Data BETWEEN :DataMin AND :DataMax",
      "ExpressionAttributeNames": {
        "#GSI-Bucket": "GSI-Bucket",
        "#Data": "Data"
      },
      "ExpressionAttributeValues": {
        ":GSI-Bucket": "Bucket-6",
        ":DataMin": "OPEN#2019-01",
        ":DataMax": "OPEN#2019-02"
      },
      "IndexName": "gsi2"
    }

    Matching Records

    GSI-Bucket (HASH) Data (RANGE)
    Bucket-6 OPEN#2019-01-18 PK: OE-ORDER1 SK: CUSTOMER1

    All Employees Hired Recently

    Perform a DocumentClient.query against the Gsi1 index with a #Data > :Data condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK and #Data > :Data",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Data": "Data"
      },
      "ExpressionAttributeValues": {
        ":SK": "HR-CONFIDENTIAL",
        ":Data": "2019-01-01"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    HR-CONFIDENTIAL 2019-02-12 PK: HR-EMPLOYEE1 Employee: John Smith Salary: 50000

    Find all employees in a certain warehouse

    Perform a DocumentClient.query against the Gsi1 index:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK",
      "ExpressionAttributeNames": {
        "#SK": "SK"
      },
      "ExpressionAttributeValues": {
        ":SK": "WAREHOUSE1"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    WAREHOUSE1 2019-02-15 PK: HR-EMPLOYEE1 Employee Name: John Smith

    Get all OrderItems for a Product including warehouse location inventories

    Perform a DocumentClient.query against the Gsi1 index with filter params:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Warehouse1": "Warehouse1"
      },
      "ExpressionAttributeValues": {
        ":SK": "PRODUCT1"
      },
      "IndexName": "gsi1",
      "FilterExpression": "attribute_exists(#Warehouse1)"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    PRODUCT1 Quickcrete Cement - 50lb bag PK: OE-PRODUCT1 Warehouse1: 46 Warehouse2: 12

    Get customers by Account Rep

    Perform a DocumentClient.query against the Gsi1 index with filter params:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Address": "Address"
      },
      "ExpressionAttributeValues": {
        ":SK": "EMPLOYEE1"
      },
      "IndexName": "gsi1",
      "FilterExpression": "attribute_exists(#Address)"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    EMPLOYEE1 Ace Building Supplies PK: OE-CUSTOMER1 Address: 1600 Penn

    Get orders by Account Rep and date

    Scatter/Gather to query all statuses (OPEN, PENDING, FULFILLED)

    Perform a DocumentClient.query against the Gsi1 index with a #Data = :Data condition on the sort key:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK and #Data = :Data",
      "ExpressionAttributeNames": {
        "#SK": "SK",
        "#Data": "Data"
      },
      "ExpressionAttributeValues": {
        ":SK": "EMPLOYEE1",
        ":Data": "OPEN#2019-01-12"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    EMPLOYEE1 OPEN#2019-01-12 PK: OE-ORDER1 OrderTotal: 2500

    Get all employees with specific Job Title

    Perform a DocumentClient.query against the Gsi1 index:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK",
      "ExpressionAttributeNames": {
        "#SK": "SK"
      },
      "ExpressionAttributeValues": {
        ":SK": "v0"
      },
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    v0 Principle Account Manager PK: HR-EMPLOYEE1

    Get inventory by Product and Warehouse

    Perform a DocumentClient.get against the Main index:

    {
      "TableName": "HR-Table",
      "Key": {
        "PK": "OE-PRODUCT1",
        "SK": "PRODUCT1"
      }
    }

    Matching Records

    PK (HASH) SK (RANGE)
    OE-PRODUCT1 PRODUCT1 Data: Quickcrete Cement - 50lb bag Warehouse1: 46 Warehouse2: 12

    Get total product inventory

    Perform a DocumentClient.get against the Main index:

    {
      "TableName": "HR-Table",
      "Key": {
        "PK": "OE-PRODUCT1",
        "SK": "PRODUCT1"
      }
    }

    Matching Records

    PK (HASH) SK (RANGE)
    OE-PRODUCT1 PRODUCT1 Data: Quickcrete Cement - 50lb bag Warehouse1: 46 Warehouse2: 12

    Get Account Reps ranked by Order Total and Sales Period

    Perform a DocumentClient.query against the Gsi1 index:

    {
      "TableName": "HR-Table",
      "KeyConditionExpression": "#SK = :SK",
      "ExpressionAttributeNames": {
        "#SK": "SK"
      },
      "ExpressionAttributeValues": {
        ":SK": "2018-Q4"
      },
      "ScanIndexFoward": false,
      "IndexName": "gsi1"
    }

    Matching Records

    SK (HASH) Data (RANGE)
    2018-Q4 10000 PK: HR-EMPLOYEE2 Employee Name: John Smith
    2018-Q4 5000 PK: HR-EMPLOYEE1 Employee Name: John Smith

    Indexes

    Main

    PK (HASH) SK (RANGE)
    HR-EMPLOYEE1 EMPLOYEE1 Data: John Smith StartDate: 01-12-2019
    HR-EMPLOYEE1 v0 Data: Principle Account Manager
    HR-EMPLOYEE1 HR-CONFIDENTIAL Data: 2019-02-12 Employee: John Smith Salary: 50000
    HR-EMPLOYEE1 WAREHOUSE1 Data: 2019-02-15 Employee Name: John Smith
    HR-EMPLOYEE1 2018-Q4 Data: 5000 Employee Name: John Smith
    HR-EMPLOYEE2 2018-Q4 Data: 10000 Employee Name: John Smith
    OE-ORDER1 CUSTOMER1 Data: OPEN#2019-01-18 GSI-Bucket: Bucket-6
    OE-ORDER1 PRODUCT1 Data: OPEN#2019-01-18 GSI-Bucket: Bucket-4 UnitPrice: $89.99
    OE-ORDER1 EMPLOYEE1 Data: OPEN#2019-01-12 OrderTotal: 2500
    OE-PRODUCT1 PRODUCT1 Data: Quickcrete Cement - 50lb bag Warehouse1: 46 Warehouse2: 12
    OE-CUSTOMER1 CUSTOMER1 Data: Ace Building Supplies Address: 1600 Penn
    OE-CUSTOMER1 EMPLOYEE1 Data: Ace Building Supplies Address: 1600 Penn

    Gsi1

    SK (HASH) Data (RANGE)
    EMPLOYEE1 John Smith PK: HR-EMPLOYEE1 StartDate: 01-12-2019
    v0 Principle Account Manager PK: HR-EMPLOYEE1
    HR-CONFIDENTIAL 2019-02-12 PK: HR-EMPLOYEE1 Employee: John Smith Salary: 50000
    WAREHOUSE1 2019-02-15 PK: HR-EMPLOYEE1 Employee Name: John Smith
    2018-Q4 5000 PK: HR-EMPLOYEE1 Employee Name: John Smith
    2018-Q4 10000 PK: HR-EMPLOYEE2 Employee Name: John Smith
    CUSTOMER1 OPEN#2019-01-18 PK: OE-ORDER1 GSI-Bucket: Bucket-6
    PRODUCT1 OPEN#2019-01-18 PK: OE-ORDER1 GSI-Bucket: Bucket-4 UnitPrice: $89.99
    EMPLOYEE1 OPEN#2019-01-12 PK: OE-ORDER1 OrderTotal: 2500
    PRODUCT1 Quickcrete Cement - 50lb bag PK: OE-PRODUCT1 Warehouse1: 46 Warehouse2: 12
    CUSTOMER1 Ace Building Supplies PK: OE-CUSTOMER1 Address: 1600 Penn
    EMPLOYEE1 Ace Building Supplies PK: OE-CUSTOMER1 Address: 1600 Penn

    Gsi2

    GSI-Bucket (HASH) Data (RANGE)
    Bucket-6 OPEN#2019-01-18 PK: OE-ORDER1 SK: CUSTOMER1
    Bucket-4 OPEN#2019-01-18 PK: OE-ORDER1 SK: PRODUCT1 UnitPrice: $89.99

    Author

    Spec authored by Eric Allam and generated by dynamodb-spec-generator

    DynamoDB Spec

    TODO

    • Ability to use a live table
    • Code generation
    • Link to the Indexes in the Access Pattern descriptions
    • Query counts
    • Query projection
    • Query attributes to get
    • Consistent reads
    • Shards
    • Scatter/Gather access patterns
    • Pagination
    • Local Secondary Indexes
    • TTL attribute
    • Updates/Puts
    • Embed the spec json in the generated markdown as a comment
    • Ability to group access patterns by section
    • Add more information about each index instead of just showing records

    Keywords

    none

    Install

    npm i dynamodb-spec-generator

    DownloadsWeekly Downloads

    0

    Version

    0.0.7

    License

    MIT

    Unpacked Size

    89.2 kB

    Total Files

    15

    Last publish

    Collaborators

    • ericallam