Continous Integration with Drone

Drone CI is an innovative self-hosted continous integration solution which is build on Docker and fits integration testing for heterogeneous microservice environments perfectly. In this article, I will give a short introduction into Drone, and will explain how to set it up securely.

TLDR; Drone is a self-hosted continous integration solution build on Docker, which perfectly fits for integration testing in heterogeneous microservice environments. In the first part of this article, I will give a brief outline of the innovative features of Drone, and afterwards I will explain how to set it up in a secure way. Drone itself is quite new, but the development is very active and people like it, see repository. Personally, we use it at work and, except for some quirks, we are very satisfied with it.

Introduction

Continous integration is a must have in every serious development cycle. Unit tests and integration tests are important to ensure the refactoring or new code is not breaking the functionality of software components. A large code base might consist of several heterogeneous microservices (overloaded word, I know) with external third-party dependencies like Kafka, Redis and so on.

Instead of writing scripts for installation, configuration and migration of schema for each service on each test run, Drone allows you to specify Docker images, which run side-by-side with your test environment (which is also defined by a Docker image). By putting each external dependency in a container, we can pull them from the registry when we need them. There are several more advantages to Drone which I will briefly describe, before I explain how to set it up.

First Impressions

The innovative idea of Drone is, that it runs in a container, uses plugins in containers, calls test and build environments in containers, and not suprisingly it can build Docker images ready to deploy. Also it is self-hosted, so you are able to scale up the VM instance if builds take too much time, but you also have to upgrade it yourself. For now, Drone targets the integration with Github out of the box with OAuth authentication for Drone users and webhooks. I have not tried other hosted git providers, but it might be difficult to set them up seamlessly, so I would not recommend using Drone without Github.

Similiar to Travis and CircleCI you have a drone.yml which lets you specify the environment as well as the steps to execute to test and build the project. This is an example drone.yml of a Scala project which relies on Kafka and Redis and is tested, compiled, assembled to a fat-jar with SBT. In the next section, I will describe different parts of this file.

compose:
  redis-cache:
    image: redis
    ports:
      - "6379:6379"

  database:
    image: spotify/kafka
    ports:
      - "9092:9092"
    environment:
      - ADVERTISED_HOST=127.0.0.1
      - ADVERTISED_PORT=9092

build:
  image: myrepo/scala-sbt
  auth_config:
    username: "$$DOCKER_USER" 
    password: "$$DOCKER_PASS"
  environment:
    - DEBUG=false
  commands:
    - sbt test
    - sbt assembly
    - sh upload_build.sh

Drone.yml

The top example shows how you can use a custom base image as your build environment. This allows you to adapt it to perfectly suit your needs. The base image is pulled from a private registry. Since it is a private registry we need to specify the registry crendentials in the auth_config parameter (also for pushing Docker images). The variables are read from the optional encrypted .drone.sec file which will be decrypted in runtime. Sure, you can use this for arbritary parameters.

Furthermore, we set up two other containers, a Redis container and a Kafka container which can be used right away in the build environment. The containers bind on localhost. And the best, the Docker images are cached in the local registry in Drone, which supports loading the test environment very fast. In case you wonder, of course you can also do caching of directories across builds, see here.

Install Drone

For the rest of this article, I will explain how to set up stable version 0.4 of Drone with Github, which can, from experience, be used in production. It will deviate from the original set up guide in the reverse proxy configuration via Nginx in front of Drone to enable HTTPS. For SSL/TLS certificate generation we will use the free-of-charge Let's encrypt project. The requirements are a fresh virtual instance of e.g. Ubuntu or CoreOS from your favourite cloud provider (DigitalOcean, Azure or AWS to name a few), or a Linux box that can run at least Docker 0.10. For the Github OAuth authentication it is also important to have a DNS name for that machine, a subdomain is also sufficient.

Configuration

For the first part, you can also cross check with the official installation guide, but note that I will change some parameters to run it with HTTPS. First, lets get our feet wet and pull Drone on the fresh box. For this you need to have Docker installed.

$ sudo docker pull drone/drone:0.4

Now you have to add an application on Github, so Drone can administrate your projects, adding deploy keys, adding hooks and such. For this go ahead an create a file called /etc/drone/dronerc and put the following lines inside:

REMOTE_DRIVER=github
REMOTE_CONFIG=https://github.com?client_id=....&client_secret=....

Fill the client_id and client_secret with the ones you get from the Github interface when adding the OAuth application. Once you did that you are ready to start a unsecure Drone container by:

$ sudo docker run \
  --volume /var/lib/drone:/var/lib/drone \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --env-file /etc/drone/dronerc \
  --restart=always \
  --publish=8000:8000 \
  --detach=true \
  --name=drone \
  drone/drone:0.4

Intentionally, we don't bind Drone on port 80. We will proxy the traffic later on to port 8000. We will use Nginx in front of Drone to encrypt the HTTP traffic. Hence, for the next step we need to request a certificate.

Encrypt the HTTP Traffic

Since Docker, it became very easy to request a certificate from Let's Encrypt. But first, let's create some folders we can mount to store the required file.

$ sudo mkdir /etc/letsencrypt
$ sudo mkdir /var/lib/letsencrypt

When the folders are created and you made sure that the ports 443 and 80 are free to bind, let's fire up the certbot Docker container which automatically spawns a webserver which is used to verify that you are the owner of that domain.

$ sudo docker run -it --rm -p 443:443 -p 80:80 --name certbot \
  -v "/etc/letsencrypt:/etc/letsencrypt" \
  -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
  quay.io/letsencrypt/letsencrypt:latest auth

Fill in the answers in this nostalgic ncurses interface and then you should retrieve a certificate and keys. For the sake of simplicity, we will use the Nginx Docker image instead of installing Nginx on the host. Nevertheless we need to supply Nginx with configuration files. So we need to create and later on mount the following folders. Lets also copy the certificate and keys to the ssl folder, replace "hostname" by your domain name.

$ sudo mkdir /etc/nginx
$ sudo mkdir /etc/nginx/ssl
$ sudo cp /etc/letsencrypt/live/hostname/* /etc/nginx/ssl/

Nginx Configuration

To recap a little, we have Drone running on port 8000. And we will use Nginx to proxy the traffic from port 80 and port 443 encrypted to port 8000. When using Drone with a reverse proxy you have to add specific parameters to the proxy configuration, see the Drone readme here. Please note, that for the SSL/TLS settings I used Cipherlist, which is maintaining a list of strong ciphers for webservers. This is the final configuration nginx.conf file (after this quote block, there is another section):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include /etc/nginx/mime.types;
  default_type  application/octet-stream;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
  '$status $body_bytes_sent "$http_referer" '
  '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;
  sendfile        on;
  keepalive_timeout  65;
  include /etc/nginx/conf.d/*.conf;

  server {
    listen 80;
    server_name drone.hipcompany.com;
    return 301 https://drone.hipcompany.com$request_uri;
  }

  server {
    listen 443 ssl;
    server_name drone.hipcompany.com;
    gzip on;

    ssl_certificate /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
    ssl_session_cache shared: SSL:10m;
    ssl_session_tickets off; # Requires nginx >= 1.5.9
    ssl_stapling on; # Requires nginx >= 1.3.7
    ssl_stapling_verify on; # Requires nginx => 1.3.7
    resolver 8.8.8.8 209.244.0.3 valid = 300s;
    resolver_timeout 5s;
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Host $http_host;
      proxy_set_header Origin "";
      proxy_redirect off;
      proxy_http_version 1.1;
      proxy_buffering off;
      proxy_pass http://172.17.0.1:8000;
      chunked_transfer_encoding off;
    }
  }
}

Starting Nginx

Be sure to change the domain accordingly. The 172.17.0.1 is the IP of the Docker bridge. Check if this one is the same on your machine if you experiencing any problems. Now we are good to go and we can fire up the Nginx in front of Drone.

$ sudo docker run --name nginx -v /etc/nginx/ssl:/etc/nginx/\
ssl -v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf -p 443:443\
-p 80:80 -d nginx

Logs

If you want to double check if the services run correctly, you can use docker logs nginx or docker logs drone to print the recent output of the running Docker containers. I hope you could encourage you to take Drone for a test ride, if you have any suggestions or questions regardings this brief guide feel free to get in contact via mail.


One or two mails a month about the latest technology I'm hacking on.