Jenkins, SELinux and Python Coverage

by Sebastien Mirolo on Thu, 28 May 2015

Time for an upgrade of the Jenkins machine. This time we are setting up Jenkins with Jetty 9 on Fedora 21 to continuously run CasperJS tests against the Django web application.

Starting Jetty/Jenkins

Jetty is available as a package in Fedora 21. Jenkins is available as a war file that we install inside a Jetty containers directory.

Terminal

# Fedora 21
$ yum install jetty
$ cd /usr/share/jetty/webapps
$ wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war
$ setsebool -P httpd_execmem 1
$ systemctl enable jetty.service
$ systemctl start jetty
Starting Jetty: WARNING: Nothing to start, exiting ...
java.io.FileNotFoundException: /usr/share/java/jetty/etc/webdefault.xml

First road bump, we are missing a webdefault.xml. Since I couldn't find any straightforward example on the local system, I ended up copying a sample from the jetty GitHub repostory.

Terminal

$ git clone https://github.com/eclipse/jetty.project.git
$ mkdir -p /usr/share/java/jetty/etc
$ cp jetty-webapp/src/main/config/etc/webdefault.xml /usr/share/java/jetty/etc

Second road bump, as explained in the post Install Jenkins 1.4 with Jetty 9, we are missing a security context.

2015/11/07 UPDATE: We used to create the context file into /usr/share/jetty/contexts but it stopped working when we deployed on a a new machine. The infamous error would pop up again.

Terminal

FAILED org.eclipse.jetty.security.ConstraintSecurityHandler: \
java.lang.IllegalStateException: No LoginService \
    for org.eclipse.jetty.security.authentication.FormAuthenticator

The only way I found to reliably solve this is to create jenkins.xml in /usr/share/jetty/webapps.

Terminal

$ sudo mkdir -p /usr/share/jetty/webapps
$ vi /usr/share/jetty/webapps/jenkins.xml
<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath">/jenkins</Set>
  <Set name="war"><SystemProperty name="jetty.base" default="."/>/webapps/jenkins.war</Set>
  <Set name="extractWAR">true</Set>
  <Set name="copyWebDir">false</Set>
  <Set name="defaultsDescriptor"><SystemProperty name="jetty.base" default="."/>/etc/webdefault.xml</Set>
  <Get name="securityHandler">
    <Set name="loginService">
      <New class="org.eclipse.jetty.security.HashLoginService">
        <Set name="name">Jenkins Realm</Set>
        <Set name="config"><SystemProperty name="jetty.base" default="."/>/etc/realm.properties</Set>
      </New>
    </Set>
    <Set name="authenticator">
      <New class="org.eclipse.jetty.security.authentication.FormAuthenticator">
        <Set name="alwaysSaveUri">true</Set>
      </New>
    </Set>
    <Set name="checkWelcomeFiles">true</Set>
  </Get>
</Configure>

At this point in the logs, we find a baffling error about JSP support.

Terminal

INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet

A few trials are error lead to modify a .ini file and update the command line use to run the jetty server.

Terminal

$ diff -u  $JETTY_HOME/start.d/jsp.ini
-#-Dorg.apache.jasper.compiler.disablejsr199=true
+-Dorg.apache.jasper.compiler.disablejsr199=true

$ java -jar start.jar --add-to-startd=http,plus,jsp,deploy --debug

Almost there, the jenkins home defaults to /usr/share/jetty/.jenkins, we need to update it to say /var/jenkins.

Terminal

$ sudo mkdir -p /var/jenkins
$ sudo chown jetty:jetty /var/jenkins
$ diff -u prev /usr/lib/systemd/system/jetty.service
+Environment="JENKINS_HOME=/var/jenkins"

At this point, we stumble into SELinux permissions issues to first have Jetty bind to the listening port, then for Jenkins to run a job on the local machine.

Before moving on and configure SELinux to let us run jobs locally, we install a set of Jenkins plugins:

Configuring SELinux

Really baffling at first is that running the testsuite script from the command line is completing correctly as expected, yet we get an execute permission denied when clicking on Build Now in the Jenkins web interface.

It all as to do with the SELinux context in which the Jetty daemon is running. The bits of SELinux to fiddle with to actually execute our testsuite were tricky. Relying on audit2allow by itself wasn't enough as it messed up lots of other permissions, with changes painful to roll back. I ended up having to write a custom policy.

Terminal

$ ps agxZ | grep jetty
system_u:system_r:httpd_t:s0    ... bash /usr/share/jett/bin/jetty.sh

$ sudo semanage fcontext -a -t 'httpd_cache_t' '/var/jenkins(/.*)?'
$ sudo restorecon -R -v /var/jenkins

$ setsebool -P httpd_builtin_scripting 1

$ cat jenkins.te
module jenkins 1.0;

require {
        type httpd_t;
        type hadoop_namenode_port_t;
        type unreserved_port_t;
        class tcp_socket name_bind;
        type var_t;
        type httpd_cache_t;
        class file { open read write relabelfrom relabelto getattr execute };
}

#============= httpd_t ==============
allow httpd_t httpd_cache_t:file { relabelfrom relabelto execute };
allow httpd_t var_t:file { open read getattr };
allow httpd_t hadoop_namenode_port_t:tcp_socket name_bind;
allow httpd_t unreserved_port_t:tcp_socket name_bind;

$ yum install checkpolicy policycoreutils-python
$ checkmodule -M -m -o jenkins.mod ./jenkins.te
$ semodule_package -m jenkins.mod -o jenkins.pp
$ semodule -i jenkins.pp

Running Python code coverage with Jenkins

Integration of pylint into jenkins was relatively straightforward. Integration of eslint and csslint were not much more complicated. Most of the trouble came from setting up python-coverage and the Cobertura Coverage plugin.

The problem is two folds. First, no code coverage will be displayed if the build has failed. To avoid static linting to fail a build, we increase the linter "switch to fail" minimum for pylint, eslint and csslint.

The second problem is that the Cobertura Coverage plugin cannot find the source files even why python-coverage writes absolute pathnames into the coverage XML output. The problem stems from commit 57aeba3a:

Terminal

private File getSourceFile() {
  if (hasPermission()) {
    return new File(owner.getProject().getRootDir(), "cobertura/" + relativeSourcePath);
  }
  return null;
}

The workaround is thus to remove the $(WORKSPACE) prefix from the python-coverage generated output!

Terminal

$ coverage combine
$ coverage xml -o coverage-pre.xml
$ sed -e 's,$(WORKSPACE)/,,g' \
    -e 's,<!-- Generated by coverage.py: http://nedbatchelder.com/code/coverage -->,<sources><source>$(siteTop)</source></sources>,g' \
    coverage-pre.xml > coverage.xml

Jenkins, SELinux and Python Coverage

Starting Jetty/Jenkins

Configuring SELinux

Running Python code coverage with Jenkins

More to read