Nginx, Jetty and Jenkins (updated)
by Sebastien Mirolo on Sat, 13 May 2017
It has been a while since we setup Nginx, Jetty and Jenkins together
the first time around. Fedora is on version 25, Jetty on version 9
and Jenkins on version 2.60.
We are going to install jenkins in a jetty web container. Jetty running behind an nginx reverse-proxy. First let's install the prerequisites and configure jetty.
$ dnf install nginx jetty wget
We configure nginx to pass requests to jetty.
$ diff -u prev /etc/nginx/conf.d/backstage.conf
+upstream proxy_jenkins {
+ server 127.0.0.1:8080;
+}
server {
+ location /jenkins {
+ try_files /jenkins$uri/index.html /jenkins$uri.html /jenkins$uri $uri/index.html $uri.html $uri @forward_to_jenkins;
+ }
+ location @forward_to_jenkins {
+ proxy_pass http://proxy_jenkins;
+ include /etc/nginx/proxy_params;
+ add_header Pragma "no-cache";
+ }
$ sudo systemctl enable nginx.service
$ sudo systemctl start nginx.service
Then we start on the configuration of jetty.
# configuration files
$ ls /usr/share/jetty/webapps \
/etc/jetty/jetty.xml \
/usr/share/jetty/start.ini \
/usr/lib/systemd/system/jetty.service
$ diff -u prev /usr/lib/systemd/system/jetty.service
[Service]
+User=jetty
$ sudo mkdir -p /var/run/jetty
$ sudo chown jetty:jetty /var/run/jetty
$ sudo restorecon -vF /var/run/jetty
We are going to disable the websockets module because it does not seem to work out of the box, add requests logging and debug.
$ cd /usr/share/jetty $ java -jar ./start.jar --list-modules $ diff -u start.ini + --module=requestlog ... - --module=websocket + #--module=websocket # fix the default path $ /usr/share/jetty/bin/jetty.sh - JETTY_LOGS=/var/log/jetty/logs + JETTY_LOGS=/var/log/jetty # fix permissions on log directory $ chmod 700 /var/log/jetty
Making any changes to the requestlog configuration is playing with fire.
(ex: Stackoverflow question: access log is no longer logging).
For us the most bafling issue was a NullPointerException
in main.java when the '/' character was not present as the first character
of the request log filename.
$ diff -u prev etc/jetty-requestlog.xml - <Default><Property name="jetty.requestlog.dir" default="/logs"/>/yyyy_mm_dd.request.log</Default> + <Default><Property name="jetty.requestlog.dir" default="/logs"/>/request-yyyy_mm_dd.log</Default> ... - <Set name="filenameDateFormat"><Property name="jetty.requestlog.filenameDateFormat" deprecated="requestlog.filenameDateFormat" default="yyyy_MM_dd"/></Set> + <Set name="filenameDateFormat"><Property name="jetty.requestlog.filenameDateFormat" deprecated="requestlog.filenameDateFormat" default="yyyyMMdd"/></Set>
There does not seem any way to configure or prevent log rotation at all that we could find.
$ sudo -u jetty /usr/lib/jvm/java/bin/java -Djetty.logs=/var/log/jetty -Djetty.home=/usr/share/jetty -Djetty.base=/usr/share/jetty -jar /usr/share/jetty/start.jar
No exceptions. Jetty looks fine, let's move on to install and configure jenkins.
$ cd /usr/share/jetty $ mkdir -p /var/lib/jetty/jenkins $ chown jetty:jetty /var/lib/jetty/jenkins $ sudo restorecon -vF jenkins $ ln -s /var/lib/jetty/jenkins .jenkins $ cd /usr/share/jetty/webapps $ wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war
We add a context for Jenkins to avoid errors like
jenkins Service Unavailable,
or java.lang.IllegalStateException: No LoginService.
$ cat /usr/share/jetty/webapps/jenkins.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/jenkins</Set>
<Set name="war"><SystemProperty name="jetty.base" default="."/>/webapps/jenkins.war</Set>
<Set name="extractWAR">true</Set>
<Set name="copyWebDir">false</Set>
<Set name="defaultsDescriptor"><SystemProperty name="jetty.base" default="."/>/etc/webdefault.xml</Set>
<Get name="securityHandler">
<Set name="loginService">
<New class="org.eclipse.jetty.security.HashLoginService">
<Set name="name">jenkins Realm</Set>
<Set name="config"><SystemProperty name="jetty.base" default="."/>/etc/realm.properties</Set>
</New>
</Set>
<Set name="authenticator">
<New class="org.eclipse.jetty.security.authentication.FormAuthenticator">
<Set name="alwaysSaveUri">true</Set>
</New>
</Set>
<Set name="checkWelcomeFiles">true</Set>
</Get>
</Configure>
$ sudo systemctl enable jetty.service
$ sudo systemctl start jetty.service
You might encounter an "AWT is not properly configured on this server" error. You will need to add -Djava.awt.headless=true to the java command-line by modifying /usr/share/jetty/start.ini as follow:
$ diff -u prev /usr/share/jetty/start.ini -# --exec + --exec + -Djava.awt.headless=true
You might also have to install special font packages as well
$ yum install dejavu-lgc-sans-fonts \
dejavu-lgc-sans-mono-fonts \
dejavu-lgc-serif-fonts \
dejavu-sans-fonts \
dejavu-sans-mono-fonts \
dejavu-serif-fonts
At that point, the jetty/jenkins server was up and running though deadly slow. Furthermore, the jetty service crashed unexpectedly, often after 5 to 10 minutes of activity.
I tried to install Oracle JDK
$ rpm -ivh jdk-7u7-linux-x64.rpm --force # .. shows unpacking errors but apparently everything works afterwards ... $ alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000
Once the underlying virtual machine had enough ram, jetty/jenkins became responsive and stable. Time to install some plug-ins, first the Git Plugin to monitor our source repository, second the Email-ext plugin to report build failures, regressions, etc., finally the different plugins to configure a slave node (see previous post on the subject)
by Sebastien Mirolo on Sat, 13 May 2017
