Lotan

####Lotan -- the multi-headed Docker container runner

Wouldn't it be nice to be able to automatically deploy Docker containers?

• without any setup or teardown
• without dedicated hosts, or a virtual servers
• without any commands to run
• without any AWS or other cloud configuration changes
• without any changes to DNS configuration
• without having to change elastic load balancers
• without any extra security rules

For example -- imagine that every GitHub pull request for your project could be viewed in a browser by simply clicking on its name.

Lotan allows you to automatically run many containers of a Docker repository concurrently on a single port on a single host. Each container appears under a subdomain named after the tag for its image.

Setup is easy -- the source code in this GIT repo is pre-configured to run tagged images from the markriggins/todowrangler repository, which are just color variations of jbdely's angular example application

Once you've finished the setup, you can click on these links to automatically pull, run, and access various colorful versions todowrangler -- each running in its own container.

• red.todowrangler.lotan.com todo-wrangler
• green.todowrangler.lotan.com todo-wrangler
• purple.todowrangler.lotan.com todo-wrangler

Pull Request Previewer

Lotan was created to allow developers, managers and quality-assurance teams to easily view containers running for GitHub pull requests.

Suppose we have an web-application named todowangler that expects to run on port 80. Jenkins has been configured to build the todowrangler project, and successful builds are pushed to a Docker registry and tagged with the pull request number.

The following images have been built and pushed to DockerHub

Images	Pull Request Number	Description
markriggins/todowrangler:pr-3001	3001	Change the banner to Green
markriggins/todowrangler:pr-3002	3002	Change the banner to Purple

Once everything has been setup, the todowrangler application can be automatically viewed at any available tag by simply using the tag as a sub-domain prefix to the URL like this:

Tag	URL
pr-3001	http://pr-3001.todowrangler.lotan.info
pr-3002	http://pr-3002.todowrangler.lotan.info

Notice that once Lotan has been configured, new images can be run with no additional setup or intervention, no AWS Task Definitions, no Docker compose commands, yaml editing etc, no linux shell stuff. It just happens.

Other use cases

Lotan can be used to automatically run any tagged image from a Docker repository that is compatible with HTTP and nginx proxy_pass.

How it works

1. Tags -- Each container runs on its own separate sub-domain, that is named after the images's tag
2. Proxy -- All requests are handled by a special nginx proxy that routes requests to running containers based on the sub-domain names.
3. Containers -- If no container exists for the tag, then the tagged image will be pulled from the Docker registry and a new container will be run to handle requests on that subdomain
4. Ports -- Each container runs on a randomly-assigned port, so you can run as many simultaneous versions of the image as desired with no conflicting ports
5. Images -- Lotan keeps a record of containers and images, and a queue of actions to perform (such as starting a new container)
6. lotan-queue-processor -- A lotan-queue-processor process does all of the image pulling and container running by scanning its queue for actions. To start a new container, the lotan-queue-processor pulls the desired image at the specified tag from the registry and runs a new container on a random port. If the container successfully starts, then the random port is recorded for future use by the above proxy; otherwise, the errors are saved for future display via the admin interface.
7. lotan-reaper -- The lotan-reaper process periodically removes stale containers that have not been recently accessed and restarts containers that have somehow stopped but should be running. Stale images are periodically refreshed by pulling them from the registry.

Setting It all Up

Configuring Lotan

1. Install Lotan on a host and setup DNS to route the the top level domain and a wild-card for subdomains:

    tagwrangler.lotan.info
    *.tagwrangler.lotan.info
   

   The wild-card DNS name that will match any sub-domain. See Wild-Card DNS Names for details

   Note: you can use any DNS name you wish

   For demonstration purposes, you can add a short list of known-in-advance image names to your /etc/hosts file, using the IP address of your Docker engine, but the beauty of using wild-card DNS is that you do not have to know the tags in advance

    192.168.99.100 todowrangler.lotan.info
    192.168.99.100 red.todowrangler.lotan.info
    192.168.99.100 green.todowrangler.lotan.info
    192.168.99.100 purple.todowrangler.lotan.info
    192.168.99.100 bad.todowrangler.lotan.info
   

2. Configure the host with routing rules and privileges to access any resources needed by the images you wish to run.

3. Run Lotan, using the following command

       docker run --rm -it  \
            -p 80:80 \
            -v /var/run/docker.sock:/var/run/docker.sock \
            -v $(which docker):/bin/docker \
            --name lotan \
            markriggins/lotan
   

   By default, Lotan will run images from the markriggins/todowrangler repository, which contains images for the tags: red, green, blue, purple and pr-3002. But you can alter Lotans behavior by setting the following environment variables.

4. Environment Variables

   • TERM -- the terminal type. Default: xterm
   • REPOSITORY -- the docker repository of images. Note that Lotan will only run images from a single repository. Default: markriggins/todowrangler.
   • TL_DOMAIN -- the top-level DNS domain which must route to the host where Lotan is running. Default: todowrangler.lotan.info
   • INACTIVITY_STOP_TIMEOUT_MINUTES -- Stop containers that are inactive for more than $INACTIVITY_STOP_TIMEOUT_MINUTES minutes. Default: 10
   • INACTIVITY_RMI_TIMEOUT_MINUTES -- Remove images that are inactive for more than $INACTIVITY_RMI_TIMEOUT_MINUTES minutes. Default: 60
   • IMAGE_REFRESH_INTERVAL_MINUTES -- Refresh images from the registry every $IMAGE_REFRESH_INTERVAL_MINUTES minutes. Default: 5
   • ENV_ARGS -- a list of env args and values to pass to containers
   • LOTAN_CONTAINER_PORT -- Lotan proxies to a single port. If a container exposes multiple ports, then you must choose one and set export its value thru this environment variable.
   • TRANSLITERATE_SEMVER_UNDERSCORES_TO_DOTS -- Set to 0 to stop flattening tags like 1.9.5 to 1_9_5. [ The default behavior makes the tags fit into a single subdomain level, which works better for SSL wild-card certificates]
   • TAG_FILTER -- a Lua regex to filter tag names listed in the drop-down list. NOTE: Lua has its own rules for regexes; for example, % is the escape character!!
   • ADMIN_WELCOME -- HTML to present on the ADMIN page (home page) as a welcome message.
   • ADMIN_INFO -- HTML to present on the ADMIN beneath the Containers table.

Passing Environment Variable to the Containers

Environment variables that begin with LOTAN_CONTAINER_ENV_ will be passed thru to the container without the prefix. In other words, if you define LOTAN_CONTAINER_ENV_PLANET="earth" for Lotan, then Lotan will pass the environment variable PLANET="earth" to each container.

AWS ECR Setup

The AWS ECS Registry requires a special login step before images can be pulled from the registry. Lotan will handle this for you, but you must define some extra environment variables:

-e AWS_DEFAULT_REGION=us-east-1 \
-e AWS_SECRET_ACCESS_KEY=my-secret-aws-key \
-e AWS_ACCESS_KEY_ID=my-access-key-id \

If the user-account that starts Lotan has an AWS credentials file (such as ~/.aws/credentials), then you can mount the credentials file as a volume and keep the AWS_SECRET_ACCESS_KEY off the command-line where is may be viewed by a ps -ef

docker run -d -v $HOME/.aws/credentials:/root/.aws/credentials \
    -e AWS_DEFAULT_REGION=us-east-1 ...

In general, secrets should not be passed through the environment in AWS as part of a TaskDefinition because aws ecs describe-task-definition will include the environment variables, and thus leak the secrets to any user account with permission to describe task definitions

Testing it

1. Auto-Run Do all the setup and then try accessing subdomains for known-good tags. If the tag is good, on the first attempt you will be given a spinner page that will wait for the image to pull and the container to start running. If you wait long enough, then the spinner will be replaced with the output from the new container.

   Containers usually start quickly -- but you don't have to wait -- just try the url again in a few minutes.

2. Admin A very simple admin panel is available at the top level domain, which will list the running containers an allow you to start, stop or remove them, as well as add new tags.

   • Click on the tag to access the container
   • Click Stop to stop the container
   • Click X to remove it.
   • Enter a tag and click Add to start a new container

   The admin interface runs on your TL_DOMAIN http://green.todowrangler.lotan.info. See Setting It all Up for details on customizing your TL_DOMAIN.

3. Common errors and their meanings

   Errors	What it means
   no container tag	Lotab could not determine the tag by the DNS name that you used to access the site. Your dns subdomain name must be a valid Docker tag without dashes, and you must have wild-card DNS working
   no such container tag	Lotan tried to pull the image with the given tag but no such image exists
   Error messages	When Lotan tried to start the image, it failed with the given error messages (Displayed in your browser when you try to access the subdomain)

Security Concerns

Lotan is a simple router -- your application is responsible for its own security.

Lotan minimizes security risks by:

• limiting its user inteface to simple tags and actions
• restricting itself to running images from a single repository
• simply proxying requests to your application

But Lotan is alpha-quality -- DO NOT use Lotan for sensitive or personally identifiable information

Lotan intentionally limits itself to work with a single repository. Although it might seem to be a good idea to be able to run just any image, you must remember that docker containers from arbitrary sources pose significant security risks. So for security reasons, run a separate Lotan host for each repository that you wish to support.

The error messages that Lotan displays to browsers are limited to those that occur while trying to start the image, typically Docker error messages, but there is some risk of leaking information if your application writes sensitive information to stdout or stderr during startup.

Performance and Stability

Lotan is alpha quality for use at your own risk. Since it only runs a single container per image, Lotan should only be used for testing and previewing, never for production.

Wild-Card DNS Names

If you are using AWS Route 53, then you can add A records for *.your-domain, but life is more difficult for OS X and even Linux

1. dnsproxy -- One solution is to use a dns proxy that does understand wildcards -- such as dnsproxy.py

2. Dnsmask -- On OS X, you can brew install Dnsmasq

3. Linux -- On linux systems, if you are running a local DNS server then checkout http://rivenlinux.info/wildcard-dns/

4. /etc/hosts -- If you know the possible subdomain names in advance, you can simply add them to your /etc/hosts file

Future Features

1. On-demand builds. If no image exists on the Docker registry with the tag, but the tag exists in the GIT repository, then: a. clone the GIT repository b. checkout the tag c. build the image d. run it locally. e. support an option to also push the image to the registry

   But I think this is better left to Jenkins or Travis

2. Multiple Repositories per Registry. We could determine the registry from the domain name like we do now for the tag.

   • < tag > . < registry > . lotan.info
   • registry white list -- we would have to lock down the registry to a white list
   • The Admin UI would have to support multiple repositories.