Logging Docker containers output through journald to syslog

Configuring the Docker daemon

First we look at the service configuration /usr/lib/systemd/system/docker.service

...
EnvironmentFile=-/etc/sysconfig/docker
...

We found the environment file where we can override options at /etc/sysconfig/docker

...
OPTIONS='--selinux-enabled --log-driver=journald --log-opt tag="{{.Name}}/{{.ID}}"'
...

The log-driver is already set to journald in this case.

We look at the log for the docker.service and running container.

$ journalctl --since "5 min ago" -l _SYSTEMD_UNIT=docker.service
-- Logs begin at Wed 2021-10-13 19:47:17 UTC, end at Thu 2021-12-23 13:10:35 UTC. --
Dec 23 13:10:10 hostname dockerd-current[1333]: [250B blob data]

$ journalctl --since "5 min ago" CONTAINER_NAME=livedemo
.compute.internal dockerd-current[27777]: [250B blob data]

Both show a size followed by blob data which would mean journalctl believes the data is in binary form. When we force all output and look at it through an hex dumper, we notice Carriage Return (0x0d, '\r', CR) characters.

$ journalctl --since "5 min ago" --all CONTAINER_NAME=livedemo | hexdump -C
...
00072980  2e 39 34 22 0d 0a 4e 6f  76 20 30 32 20 30 33 3a  |.94"..Nov 02 03:|
...

These are the symptoms of a pseudo-tty - i.e. we started the container with the -t command line option.

We will thus restart the container in detached mode without the -t command line option.

$ docker run -d -p 8000:80 --name livedemo  ghcr.io/djaodjin/djaoapp/livedemo:2021-10-05
$ journalctl --since "5 min ago" CONTAINER_NAME=livedemo
Dec 24 13:15:48 ip.compute.internal dockerd-current[27578]: 172.17.0.1 livedemo.djaoapp.com - [24/Dec/2021:13:15:48 +0000] "GET / HTTP/1.0" 200 17308 "-" "Mozilla/5.0"

Docker journald logging driver documentation states that SYSLOG_IDENTIFIER can be specified by setting --log-opt tag=.

$ journalctl --since "5 min ago" -o json-pretty CONTAINER_NAME=livedemo
{
...
    "SYSLOG_IDENTIFIER": "livedemo/aee81ff4d735",
    "CONTAINER_TAG" : "livedemo/aee81ff4d735",
...
}

This works well on Amazon Linux 2 (Docker version 20.10.7). CentOS7 uses Docker version 1.13.1. Even though CONTAINER_TAG is specified, SYSLOG_IDENTIFIER is never set.

The rest of the setup is similar to described in Logging gunicorn messages through journald to syslog-ng.

More to read

You might also like to read:

• Django, Gunicorn and Syslog-ng
• Jenkins, Docker build and Amazon EC2 Container Registry
• First day with Docker
• Logrotate, S3 storage, and AWStats

More technical posts are also available on the DjaoDjin blog, as well as business lessons we learned running a SaaS application hosting platform.