Export Python Code Coverage from a Docker Container to Jenkins

The full tests workflow is presented in the diagram below. We presented the steps to build a Docker container and upload it to ECR in another post. Here we will focus on generating Python coverage results and getting the files to be presented into Jenkins.

Actively running the casperjs tests flow against a fully provisioned WebApp will require to

• Bring-up an ECS instance with the Docker container.
• Run the tests from the Jenkins Server against the WebApp HTTP end-point
• Shutdown the container which will have for effect to copy the coverage files on disk.
• Copy the coverage files back to the Jenkins server at an appropriate location and let Jenkins handle presentation of results.

Bring-up the Dockerized WebApp

The major features released in Jenkins v2 are Jenkinsfile and Pipelines. Unfortunately after much tinkering, we found out that these features rely on a new API for plug-ins and Cobertura does not support it yet (Jenkins version 2.32.2, #352, #50). We reverted back to Old-style Jenkins projects and good old fashion Python scripts to drive provisioning (calling on boto3), running tests and gathering results.

The steps to provision an ECS cluster and deploy a Docker container inside it are somewhat generic (see drundocker.py for details). What is most interesting is how to setup our Docker container and web application to make it all work.

Configuration for Python coverage

The whole point of building a Docker container is to have the same container used during testing and deployed in production. We are thus very careful to only install packages required for production use. For example, we purposely insure that django-debug-toolbar or django-extensions are not installed in the container. An exception to that rule is python-coverage.

On a side note, we also don't want to pip install (statement in the Dockerfile) a module that requires a native module and ends up pulling gcc. Ideally when building a Docker container, we would want all packages (Python or otherwise) installed through the system package manager.

We modify wsgi.py such that code coverage is recorded and saved to disk.

import os
import signal

def save_coverage(*args, **kwargs):
    sys.stderr.write("saving coverage\n")
    cov.stop()
    cov.save()

if os.getenv('DJANGO_COVERAGE'):
    import atexit, sys
    import coverage
    cov = coverage.coverage(data_file=os.path.join(os.getenv('DJANGO_COVERAGE'),
        ".coverage.%d" % os.getpid()))
    cov.start()
    atexit.register(save_coverage)
    try:
        signal.signal(signal.SIGTERM, save_coverage)
    except ValueError as e:
        # trapping signals does not work with manage.py
        # trying to do so fails with ValueError.
        # Signal only works in main thread.
        pass

Flexible config files

We need a way to start a Docker container with either tests or production config files. Extending on the environment variables approach described in Docker on Elastic Beanstalk, we add the following code in settings.py

from deployutils import load_config, update_settings

APP_NAME = os.path.basename(BASE_DIR)

update_settings(sys.modules[__name__],
    load_config(APP_NAME, 'credentials', 'site.conf',
        location=os.getenv("SETTINGS_LOCATION", None),
        passphrase=os.getenv("SETTINGS_CRYPT_KEY", None)))

This way we pass two environment variables to the Docker container which are the location of the config files and a key to decrypt them. In practice we use a credentials file that holds secrets such as API keys, e-mail passwords, etc. and a site.conf file that holds database names, service end points, etc.

Gathering the coverage files

When running the container for testing, we mount a directory on the host filesystem inside the container such that coverage files are written there. Later on it is only a matter of copying the files back from the host machine to the Jenkins server.

There was a bit of trial and error to get coverage files written out when the gunicorn daemon is shutdown but we finally got it to work as is reflected in the wsgi.py code above.

We still had to massage the coverage output files a bit because of Cobertura / Python Coverage issues but otherwise we are done. We have code coverage displayed in Jenkins for the Django WebApp running inside a Docker container.

More to read

If you are looking for related posts, Jenkins, Docker build and Amazon EC2 Container Registry and Docker on Elastic Beanstalk are good reads.

More technical posts are also available on the DjaoDjin blog. For fellow entrepreneurs, business lessons learned running a subscription hosting platform are also available.