github-task-manager

    5.7.2 • Public • Published

    github-task-manager

    npm version Build Status Code Climate Test Coverage Greenkeeper badge Commitizen friendly semantic-release Docker Build Status

    receive github hook, notify agent, receive task results, notify github (Unofficial)

    Aims

    Create an asynchronous CI agnostic mechanism for running custom test stage gates for github pull requests.

    • allow team leads to configure task sequences without leaving github
    • allow developers to see output from tasks without leaving github
    • trigger multiple jobs in parallel and indicate pending status on pr checks
    • then add results for each back to pull request check/comments as they complete
    • make extensible for other github event handling
    • stateless and deployable to Kubernetes at scale

    Design

    • Deploy two functions to lambda via serverless framework ('gtmGithubHook', 'gtmGithubResults')
    • github PR open hook pointed at 'gtmGithubHook' lambda which adds event to a 'PendingQueue' SQS queue
    • agent(s) watch SQS for new test execution jobs
    • agent notifies 'ResultsQueue' SQS queue that a task has started (lambda updates github PR state)
    • agent triggers CI PR build, deploy and tests (plugin for each CI type)
    • agent formats and adds test results to 'ResultsQueue' SQS queue
    • agent notifies SNS topic which triggers 'gtmGithubResults' lambda
    • 'gtmGithubResults' lambda posts results to github pull request.

    flow

    Install

    Environment variable description
    GTM_AWS_KMS_KEY_ID aws kms key id
    GTM_CRYPT_GITHUB_TOKEN encrypted access token for accessing github
    GTM_CRYPT_GITHUB_WEBHOOK_SECRET encrypted shared secret from github webook config
    GTM_CRYPT_AWS_ACCESS_KEY_ID encrypted aws key id - for agent only
    GTM_CRYPT_AWS_SECRET_ACCESS_KEY encrypted aws secret - for agent only
    GTM_CRYPT_AGENT_AWS_SECRET_ACCESS_KEY secret key for agent
    GTM_CRYPT_AGENT_AWS_ACCESS_KEY_ID access key for agent
    GTM_CRYPT_JENKINS_TOKEN encrypted token
    GTM_CRYPT_TEAMCITY_PASSCODE encrypted teamcity executor passcode
    GTM_CRYPT_SONAR_LOGIN encrypted sonar access token
    GTM_CRYPT_SONAR_GITHUB_OAUTH encrypted github token for sonar to post comments and status
    GTM_CRYPT_DOCKER_REG_PASSWORD encrypted docker private registry password
    GTM_AWS_REGION awsregion to create resources in
    GTM_SQS_PENDING_QUEUE name of SQS queue for new event
    GTM_SQS_RESULTS_QUEUE name of SQS queue for results
    GTM_SNS_RESULTS_TOPIC name of SNS topic for result ping
    GTM_GITHUB_HOST api hostname can be updated for github enterprise
    GTM_GITHUB_DEBUG debug mode for api calls
    GTM_GITHUB_TIMEOUT github api timeout
    GTM_GITHUB_PATH_PREFIX path prefix for github enterprise
    GTM_GITHUB_PROXY github api client proxy
    GTM_TASK_CONFIG_FILENAME filename in repo to look for for task config - default is .githubTaskManager
    AWS_PROXY URL of proxy to use for network requests. Optional
    GTM_AGENT_PORT defaults to 9091
    GTM_JENKINS_USER login for jenkins executor
    GTM_JENKINS_URL url executor uses to talk to jenkins
    GTM_JENKINS_CSRF is csrf enabled? true or false
    GTM_TEAMCITY_USER teamcity executor user
    GTM_TEAMCITY_URL teamcity api url
    GTM_DOCKER_IMAGE_WHITELIST comma separated list of regex of allows docker images eg. alpine:*,bash:latest
    GTM_DOCKER_IMAGE_WHITELIST_FILE use an optional docker whitelist file .dockerImageWhitelistExample
    GTM_DOCKER_COMMANDS_ALLOWED default is false, set to true to enable docker executor
    GTM_DOCKER_ALLOW_PULL allow agent to pull from registry
    GTM_DOCKER_DEFAULT_WORKER_IMAGE for running ci tasks, default is zotoio/gtm-worker:latest
    GTM_DOCKER_REG_USERNAME username for docker private registry
    GTM_DOCKER_REG_SERVER hostname for docker private registry
    IAM_ENABLED agent host uses IAM ?
    LAUNCHDARKLY_API_TOKEN token for launchdarkly sass executor
    GTM_LOGSTASH_HOST optional logstash host for elasticsearch analysis
    GTM_LOGSTASH_PORT optional logstash port
    GTM_SONAR_HOST_URL sonar host url to connect to
    GTM_SONAR_PROJECTNAME_PREFIX prefix if reporting to sonarqube
    GTM_SONAR_ANALYSIS_MODE mode for sonar runner, default preview for PRs
    GTM_SONAR_SOURCES default source dir is src
    GTM_SONAR_JAVA_BINARIES default is target
    GTM_SONAR_MODULES comma separated modules
    GTM_SONAR_GITHUB_ENDPOINT optional enterprise github api url
    GTM_TASK_CONFIG_DEFAULT_URL url to default sample config used when repo is missing .githubTaskManager.json
    GTM_TASK_CONFIG_DEFAULT_MESSAGE_PATH path to markdown comment file added to PRs when repo is missing .githubTaskManager.json
    GTM_DYNAMO_TABLE_EVENTS DynamoDB table to store event summaries
    GTM_DYNAMO_TABLE_AGENTS DynamoDB table to store agent summaries
    GTM_AWS_VPC_ID vpc id - only required for ddb endpoints
    GTM_BASE_URL Base url used to render links to agent ui - eg elb cname
    GTM_WELCOME_MESSAGE_ENABLED If not 'false', send a warning message on unconfigured repository pull requests
    GTM_S3_DEPENDENCY_BUCKET aws s3 storage of build dependencies
    GTM_AWS_S3_PROXY https_proxy for aws s3
    GTM_REPO_BLACKLIST comma separated list of regex to blackist repo names from triggering events
    GTM_SLS_EXECUTOR_AWS_STAGE stage override from default calculation of dev/test
    GTM_SLS_EXECUTOR_AWS_REGION aws region for lambdas default ap-southeast-2
    GTM_SLS_EXECUTOR_AWS_EXECUTION_ROLE docker serverless lambda execution role
    GTM_SLS_EXECUTOR_DEPLOY_MODE deploy multiple lambdas 'parallel' (default) or 'sequential'
    GTM_SLS_EXECUTOR_SNS_ERROR_TOPIC_ARN sns topic to notify on serverless error
    GTM_SLS_EXECUTOR_APIGW_ENDPOINT_TYPE EDGE or REGIONAL or PRIVATE
    GTM_SLS_EXECUTOR_APIGW_DOMAIN_SUFFIX reverse proxy domain name that will have apiId added as subdomain. eg. lambda.mysuffix.com will result in [apiId].lambda.mysuffix.com
    GTM_SLS_EXECUTOR_HTTP_PROXY proxy passed to serverless executor
    GTM_SLS_EXECUTOR_NO_PROXY no_proxy passed to serverless executor
    GTM_SLS_EXECUTOR_VPC_ID vpc id for private apigw endpoints
    GTM_SLS_EXECUTOR_VPC_SECURITY_GROUP_ID vpc security group id
    GTM_SLS_EXECUTOR_VPC_SUBNET_A vpc az subnet
    GTM_SLS_EXECUTOR_VPC_SUBNET_B vpc az subnet
    GTM_SLS_EXECUTOR_VPC_SUBNET_C vpc az subnet
    GTM_SLS_EXECUTOR_AWS_KMS_KEY_ID kms key id for sls env var encryption
    GTM_SLS_EXECUTOR_CONFIG_TYPE 'ssm' (aws parameter store) or 'spring' or 'dotenv'
    GTM_SLS_EXECUTOR_SPRING_CONFIG_ENDPOINT endpoint url of spring config server eg. http://spring:8888 if GTM_SLS_EXECUTOR_CONFIG_TYPE is 'spring'
    GTM_WORKER_SCRIPTS_CLONE for docker executors using https://github.com/zotoio/gtm-worker based image - url of git repo to overlay on workspace eg. https://github.com/zotoio/gtm-worker-scripts.git
    GTM_WORKER_SCRIPTS_PATH directory within scripts clone repo to overlay

    important: values of env vars prefixed with GTM_CRYPT_* must be created via npm run sls-encrypt [name] [value]

    Configure and deploy

    • run: npm run sls-deploy - note that this will create aws re$ources..
    • capture the hook url output in console and add to github repo pull request conf
    • run: npm run sls-logs-hook or npm run sls-logs-results to tail the logs
    • create a .githubTaskManager.json in your repo per https://github.com/zotoio/github-task-manager/wiki/Creating-a-Task-Configuration
    • start an agent locally using npm run build && npm start agent (or use docker/k8s)
    • create a pull request and confirm the hook is being hit and agent processes event

    Docker and Kubernetes agents

    You can run the latest image from docker hub: https://hub.docker.com/r/zotoio/github-task-manager

    npm run docker-hub-run
    

    ..or run using the local checkout and tail logs:

    npm run docker-local-bounce
    

    ..or if you have a k8s cluster and kubectl configured:

    npm run k8s-apply
    npm run k8s-delete
    

    note that these k8s npm script inject vars from .env into the manifest

    k8s

    a starting point k8s manifest is in ./k8s/k8s-gtm-agent.yml

    Agent Configuration

    Running the Agent

    The agent uses environment variables to configure itself as well as any executors running within it. The below environment variables are required for basic operation of the GTM Agent.

    TODO: Provide Variables

    Agent Homepage

    The GTM Agent provides an information page summarising the ongoing operation of the agent. The page is available on port 9091 by default.

    Plugins

    Task executors for Jenkins, Teamcity, Travis, Http, Docker are in progress. Custom task executors can be added by adding this project as a dependency, and registering new Executors and EventHandlers. Please see https://github.com/zotoio/gtm-agent for an example that you can fork and modify as required while still using this project as the core.

    • Executors contain the logic to run tasks against a given system type, and format the results.
    • EventHandlers are used to map Github events to specific functionality such as pull requests.

    Contributing

    Fork this repository and work on your enhancements, then send a pull request. If you build custom plugins that may be useful to others in your forked gtm-agent, please let us know and may be able to assist wwith backporting to this project.

    Use commitizen for conventional commit messages via git cz instead of git commit.
    To setup if not already installed:

    npm install -g commitizen
    npm install -g cz-conventional-changelog
    echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc
    

    ...or you can just use npm run commit which will use local commitizen install.

    Keywords

    Install

    npm i github-task-manager

    DownloadsWeekly Downloads

    14

    Version

    5.7.2

    License

    MIT

    Unpacked Size

    14 MB

    Total Files

    178

    Last publish

    Collaborators

    • wyvern8