Welcome to the DjaoDjin Blog!

A place to share experiences in building Software-as-a-Service.

Nginx, Jetty and Jenkins (updated)

by Sebastien Mirolo on Sat, 13 May 2017

It has been a while since we setup Nginx, Jetty and Jenkins together the first time around. Fedora is on version 25, Jetty on version 9 and Jenkins on version 2.60.

We are going to install jenkins in a jetty web container. Jetty running behind an nginx reverse-proxy. First let's install the prerequisites and configure jetty.

$ dnf install nginx jetty wget

We configure nginx to pass requests to jetty.

$ diff -u prev /etc/nginx/conf.d/backstage.conf

+upstream proxy_jenkins {
+        server  127.0.0.1:8080;
+}

server {

+        location /jenkins {
+            try_files /jenkins$uri/index.html /jenkins$uri.html /jenkins$uri $uri/index.html $uri.html $uri @forward_to_jenkins;
+        }

+        location @forward_to_jenkins {
+            proxy_pass    http://proxy_jenkins;
+            include       /etc/nginx/proxy_params;
+            add_header    Pragma "no-cache";
+        }

$ sudo systemctl enable nginx.service
$ sudo systemctl start nginx.service

Then we start on the configuration of jetty.

# configuration files
$ ls /usr/share/jetty/webapps \
     /etc/jetty/jetty.xml \
     /usr/share/jetty/start.ini \
     /usr/lib/systemd/system/jetty.service

$ diff -u prev /usr/lib/systemd/system/jetty.service
[Service]
+User=jetty

$ sudo mkdir -p /var/run/jetty
$ sudo chown jetty:jetty /var/run/jetty
$ sudo restorecon -vF /var/run/jetty

We are going to disable the websockets module because it does not seem to work out of the box, add requests logging and debug.

$ cd /usr/share/jetty
$ java -jar ./start.jar --list-modules

$ diff -u start.ini
+ --module=requestlog
...
- --module=websocket
+ #--module=websocket

# fix the default path
$ /usr/share/jetty/bin/jetty.sh
-  JETTY_LOGS=/var/log/jetty/logs
+  JETTY_LOGS=/var/log/jetty

# fix permissions on log directory
$ chmod 700 /var/log/jetty

Making any changes to the requestlog configuration is playing with fire. (ex: Stackoverflow question: access log is no longer logging). For us the most bafling issue was a NullPointerException in main.java when the '/' character was not present as the first character of the request log filename.

$ diff -u prev etc/jetty-requestlog.xml
-          <Default><Property name="jetty.requestlog.dir" default="/logs"/>/yyyy_mm_dd.request.log</Default>
+          <Default><Property name="jetty.requestlog.dir" default="/logs"/>/request-yyyy_mm_dd.log</Default>
...
-     <Set name="filenameDateFormat"><Property name="jetty.requestlog.filenameDateFormat" deprecated="requestlog.filenameDateFormat" default="yyyy_MM_dd"/></Set>
+     <Set name="filenameDateFormat"><Property name="jetty.requestlog.filenameDateFormat" deprecated="requestlog.filenameDateFormat" default="yyyyMMdd"/></Set>

There does not seem any way to configure or prevent log rotation at all that we could find.

$ sudo -u jetty /usr/lib/jvm/java/bin/java -Djetty.logs=/var/log/jetty -Djetty.home=/usr/share/jetty -Djetty.base=/usr/share/jetty -jar /usr/share/jetty/start.jar

No exceptions. Jetty looks fine, let's move on to install and configure jenkins.

$ cd /usr/share/jetty
$ mkdir -p /var/lib/jetty/jenkins
$ chown jetty:jetty /var/lib/jetty/jenkins
$ sudo restorecon -vF jenkins
$ ln -s /var/lib/jetty/jenkins .jenkins

$ cd /usr/share/jetty/webapps
$ wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war

We add a context for Jenkins to avoid errors like jenkins Service Unavailable, or java.lang.IllegalStateException: No LoginService.

$ cat /usr/share/jetty/webapps/jenkins.xml
<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath">/jenkins</Set>
  <Set name="war"><SystemProperty name="jetty.base" default="."/>/webapps/jenkins.war</Set>
  <Set name="extractWAR">true</Set>
  <Set name="copyWebDir">false</Set>
  <Set name="defaultsDescriptor"><SystemProperty name="jetty.base" default="."/>/etc/webdefault.xml</Set>
  <Get name="securityHandler">
    <Set name="loginService">
      <New class="org.eclipse.jetty.security.HashLoginService">
        <Set name="name">jenkins Realm</Set>
        <Set name="config"><SystemProperty name="jetty.base" default="."/>/etc/realm.properties</Set>
      </New>
    </Set>
    <Set name="authenticator">
      <New class="org.eclipse.jetty.security.authentication.FormAuthenticator">
        <Set name="alwaysSaveUri">true</Set>
      </New>
    </Set>
    <Set name="checkWelcomeFiles">true</Set>
  </Get>
</Configure>

$ sudo systemctl enable jetty.service
$ sudo systemctl start jetty.service

You might encounter an "AWT is not properly configured on this server" error. You will need to add -Djava.awt.headless=true to the java command-line by modifying /usr/share/jetty/start.ini as follow:

$ diff -u prev /usr/share/jetty/start.ini
-# --exec
+ --exec
+ -Djava.awt.headless=true

You might also have to install special font packages as well


$ yum install dejavu-lgc-sans-fonts \
    dejavu-lgc-sans-mono-fonts \
    dejavu-lgc-serif-fonts \
    dejavu-sans-fonts \
    dejavu-sans-mono-fonts \
    dejavu-serif-fonts

At that point, the jetty/jenkins server was up and running though deadly slow. Furthermore, the jetty service crashed unexpectedly, often after 5 to 10 minutes of activity.

I tried to install Oracle JDK


$ rpm -ivh jdk-7u7-linux-x64.rpm --force
# .. shows unpacking errors but apparently everything works afterwards ...
$ alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000

Once the underlying virtual machine had enough ram, jetty/jenkins became responsive and stable. Time to install some plug-ins, first the Git Plugin to monitor our source repository, second the Email-ext plugin to report build failures, regressions, etc., finally the different plugins to configure a slave node (see previous post on the subject)