Shipping ssh login events through OpenTelemetry

Configuring journald receiver

Most Linux systems record security events (including user login) through auditd and manage logs locally on the machine through journald.

We will thus look through the journald receiver documentation, and modify the configuration file to look for specific user login events.

receivers:
  journald:
    directory: /var/log/journal
    priority: "7"
    matches:
      - _TRANSPORT: audit

exporters:
  debug:
    verbosity: normal
    use_internal_logger: false

service:

  pipelines:

    logs:
      receivers: [journald]
      exporters: [debug]
      

Let's make sure the otelcol-contrib user has access to the journal. Then that the collector service is running properly.

$ sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contrib
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.

$ sudo usermod -aG systemd-journal otelcol-contrib
$ sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contrib
Sep 13 18:21:26 hostname ...

$ sudo systemctl start otelcol-contrib.service

$ ps auxwww|grep otel
otelcol-contrib    /usr/bin/otelcol-contrib --config=/etc/otelcol-contrib/config.yaml
otelcol-contrib    journalctl --utc --output=json --follow --unit ssh --priority info --directory /run/log/journal
      

Let's make sure something is showing up in the collector.

$ sudo journalctl -S "5 min ago" -l _SYSTEMD_UNIT=otelcol-contrib.service  --no-pager
      

Filtering for messages we are interested in

First, let's investigate which events we can reliably depend on. Since all ssh connections will go through a bastion/sally machine, we will attempt to ssh to a machine through the bastion (ProxyJump) or ssh into the bastion directly.

We will settle on filtering USER_START, more specifically PAM:session_open, log events only.

The otelco-contrib filter processor only has syntax to drop out records. We are interested in keeping (filter-in) specific log statements. So we will have to do so in the receiver instead.

receivers:
  journald:
    directory: /var/log/journal
    priority: "7"
    matches:
      - _TRANSPORT: audit
    grep: "USER_START.*op=PAM:session_open"
...   

Exporting OpenTelemetry events through HTTP requests

At this point, we use the experience gathered earlier to configure the otlphttp exporter.

...
exporters:
  otlphttp:
    logs_endpoint: https://example.com/api/events
    headers:
      Authorization: Bearer *****
      Content-Type: application/json
    encoding: json
    compression: none
      

Conclusion

OpenTelemetry plays a similar role to syslog and journalctl aggregation features. The philosophy is to push every events through unless it is actively dropped.

More to read

You might also like to read failed experiment to ship OpenTelemetry events to GitLab, Fast-tracking server errors to a log aggregator on S3, or Logging Docker containers output through journald to syslog.

More technical posts are also available on the DjaoDjin blog, as well as business lessons we learned running a SaaS application hosting platform.

We also came across these insightful posts from across the Web while doing research:

• Adopting OpenTelemetry for our logging pipeline
• PCI Compliance for Linux - OpenTelemetry
• How to Collect, Process, and Ship Log Data with Rsyslog