ds-project-duplicate-script

Digital Site - Project Duplication Script

Description

This software aims to make Digital Site's projects duplication possible. Duplication of a project includes :

• database's project dump and import
• project's tuple from globalliste table from rgoc_queue database
• project's folder copy into an AZ cloud storage

Requirements

• node
• git
• Mysql connection parameters for source environement
• Mysql connection parameters for target environement
• For DS 19.3 environements ( source and target ) : user with
  • read rights to
    • folder BASE_PATH
  • azcopy and mysqldump binaries
• Azure Storage credentials
  • storage url
  • SAS token

Setup

Create a dot env file and add the environment variables needed for your use

Docker build

Launch

CLI

2 solutions

npm i -g
ds-project-duplicate-script --help

OR

npm run cli -- --help

HTTP Server

npm start

Configuration

File

configuration.json file at the root of the app folder can be found and modify to apply your configuration

{
    "source": {
        "database": {
            "host": "localhost",
            "port": "3306",
            "user": "user",
            "password": "password",
            "defaultDb": "database",
            "ssl": "false"
        },
        "file": {
            "azure": {
                "url": "",
                "token": ""
            },
            "basePath": "/path/to/data"
        }
        
    },
    "target": {
        "database": { 
            "host": "localhost",
            "port": "3306",
            "user": "user",
            "password": "password",
            "defaultDb": "database",
            "ssl": "false"
        }, 
        "file": {
            "azure": {
                "url": "http://azure.storage.url",
                "token": "?sas=token_from_azure_storage"
            },
            "basePath": ""
        }
    }
}

CLI

excute the following command

ds-project-duplicate-script configure

Environments variables

• DEBUG=true or false
• Azure
  • Source
    • AZURE_SOURCE_STORAGE_URL=https://storage.endpoint/path_to_target_folder
    • AZURE_SOURCE_TOKEN= SAS Token from AZ Cloud storage
  • Target
    • AZURE_TARGET_STORAGE_URL=https://storage.endpoint/path_to_target_folder
    • AZURE_TARGET_TOKEN= SAS Token from AZ Cloud storage
• Mysql
  • Source
    • MYSQL_SOURCE_HOST=localhost
    • MYSQL_SOURCE_USER=root
    • MYSQL_SOURCE_PASSWORD=toto
    • MYSQL_SOURCE_PORT=3306
    • MYSQL_SOURCE_DEFAULT_DB=rgoc_queue
    • MYSQL_SOURCE_SSL=true or false
  • Target
    • MYSQL_TARGET_HOST=localhost
    • MYSQL_TARGET_USER=root
    • MYSQL_TARGET_PASSWORD=toto
    • MYSQL_TARGET_PORT=3306
    • MYSQL_TARGET_DEFAULT_DB=rgoc_queue
    • MYSQL_TARGET_SSL=true or false
• SOURCE_BASE_PATH, TARGER_BASe_PATH: absolute path to rgds's folders
• HTTP Server
  • PORT=3001 ( port on which HTTP server responds )
  • NO_UI=true ( disable HTML template rendering and return 500 HTTP status code for GET '/' and POST '/' with url encoded data, run as usual for POST '/' with json body )
    • !!! Set 'NO_UI' variable to true is higly recommanded on production environments !!!

!!! In case of docker compose use !!! MYSQL_ROOT_PASSWORD=toto

HTML Templates

Pug templating use Bulma for CSS.

Files are store into 'html' folder.

• index

  contains a form with 1 input field for project rgds name; and a submit button

• result contains list of decorated pre for log lines

HELM charts

Charts are contained into 'charts' folder.

A deployment is declared.

Behavior

General

Application can be launch as CLI command or HTTP server Each step contains a verify and a run function. By default, it will execute all steps verifications and executions.

Input

• Project : name of project rgds into Digital Site

Steps

• Step Duplicate File

  • verify
    • Open connection to service through SSH
    • ls on configured base path on source environement and project rgds folder
    • Dispose of SSH connection
  • run
  • Execution of AZCopy binaries on rgds project's folder to cloud storage
  • [Optionnal target environement 19.3] append rgds launch php to rgds into 7pm_launchrapide.sh file => step has problem

• Step Duplicate Database

  • verify
    • Open connection to mysql source database of project
    • Close mysql connection
    • Open connection to mysql source rgoc_queue database
    • Verify existence of project's tuple into globalliste table
    • Close mysql connection
  • run
    • Dump source project's mysql database into local sql file using mysqldump
    • Open connection to mysql target server
    • Drop database if exists and create project's database
    • Close mysql connection from mysql target server
    • Import dump of project's database in mysql target project's database
    • Close import mysql connection from mysql target project's database
    • Open connection to mysql source rgoc_queue database
    • Select project's tuple from globalliste table
    • Close connection from mysql source rgoc_queue database
    • Open connection to mysql target rgoc_queue database
    • Insert project's tuple into globalliste table
    • Close connection from mysql target rgoc_queue database

CLI

Usage

 ds-project-duplicate-script [options] [command]

Options

  -h, --help                          display help for command

Commands

  run <project> : verify and run all steps for the given project rgds
  configure : offer dialog interface to set values of configuration file
  install : verifies azcopy and mysqldump binaries exists and install azcopy binaries if not
  help [command]  : display command help 

HTTP Server

Routes

• '/'
  • GET : HTML form page => ( can be disabled )
  • POST : launch run for project creation
    • query parameters => ( can be disabled )
      • input
        • project : string
      • output : HTML result page
    • JSON body
      • input
        • project : string
      • output : Array of strings containing each log lines of the executed run

Server Port can be configured setting PORT environment variable. On Route '/', GET requests and POST with query parameters requests can be disabled setting NO_UI environment variable to true.