Logging gunicorn messages through journald to syslog-ng
by Sebastien Mirolo on Wed, 22 Dec 2021We are interested here to log messages from a gunicorn application to journald, eventually forwarding the messages to syslog-ng.
gunicorn.conf
The first step is write the gunicorn messages to stdout and stderr.
$ cat /etc/gunicorn/webapp.conf ... accesslog="-" errorlog="-" ...
Systemd service
There are often problems with Type=forking when it comes to journald logging. We thus do not daemonize gunicorn and use a Type=simple.
Since we will forward the log to
$ cat /usr/lib/systemd/system/webapp.service ... [Service] Type=simple User=webapp PIDFile=/var/run/webapp/webapp.pid EnvironmentFile=-/etc/sysconfig/webapp WorkingDirectory=/app/reps/webapp ExecStart=/app/bin/gunicorn -c /etc/gunicorn/webapp.conf webapp.wsgi ExecReload=/bin/kill -s HUP $MAINPID ExecStop=/bin/kill -s QUIT $MAINPID SyslogIdentifier=app.webapp PrivateTmp=true ...
journald.conf
Forwarding journald messages to syslog is straigtforward.
$ diff -u prev /etc/systemd/journald.conf ... -#ForwardToSyslog=no +ForwardToSyslog=yes ...
syslog-ng.conf
We add a filter to write gunicorn forwarded message to a file, and a rule to exclude them from being written in the default logfile.
$ cat /etc/syslog-ng/conf.d/webapp.conf ... filter f_webapp { program("app.webapp"); }; destination d_webapp { file("/var/log/gunicorn/app.webapp.log"); }; log { source(s_sys); filter(f_webapp); destination(d_webapp); };
$ diff -u prev /etc/syslog-ng/syslog-ng.conf ... filter f_default { level(info..emerg) and + not (program("app.*") or facility(mail) or facility(authpriv) or facility(cron)); }; ...
logrotate.conf
Finally since log files are rotated, we add a rule for the log files defined
earlier (
$ diff -u prev /etc/logrotate.d/syslog /var/log/cron /var/log/maillog /var/log/messages /var/log/secure /var/log/spooler + /var/log/gunicorn/app.*.log { ...
Voila, we just need to restart all the services involved.
systemctl daemon-reload systemctl restart syslog-ng.service systemctl restart webapp.service
to debug, we look at the journal and the final log file.
sudo journalctl --since "5 min ago" -l _SYSTEMD_UNIT=webapp.service tail -f /var/log/gunicorn/app.webapp.log
More to read
You might also like to read:
- Django, Gunicorn and Syslog-ng
- Debugging logrotate scripts
- Logrotate, S3 storage, and AWStats
- Fast-tracking server errors to a log aggregator on S3
More technical posts are also available on the DjaoDjin blog, as well as business lessons we learned running a SaaS application hosting platform.