Exporting HTTP errors through OpenTelemetry

by Sebastien Mirolo on Thu, 19 Sep 2024

Having gained some insights into OpenTelemetry through previous experiments, we planned to forward 50x nginx log records to an aggregator through OpenTelemetry.

The filelog receiver

A OpenTelemetry nginx module but it requires to compile some C++ code, then install a shared library on the machines we are running nginx on.

NGinx is happily writing a line for every HTTP requests into a log file on disk. So for now, we will use the filelog receiver to monitor nginx log files and spit out OpenTelemetry log events we are interested in.

/etc/otelcol-contrib/config.yaml
receivers:
  filelog:
    include: [ /var/log/nginx/*-access.log ]
    operators:
      - type: regex_parser
        regex: '^.*Root=(?P\S+)-(?P\S+)-(?P\S+) \[(?P.+)\] "[^"]+"\s+(?P\d+)\s+.*$'
        timestamp:
          parse_from: attributes.time_local
          layout: '%d/%b/%Y:%H:%M:%S %z'
        status_code:
          parse_from: attributes.status_code
        trace:
          trace_id:
            parse_from: attributes.trace_id
      - type: filter
        expr: 'attributes.status_code matches "^[1-4][0-9]{2}$"'

First, we are running the nginx server behind an AWS Load balancer. We have thus configured the nginx server to record the trace_id created by the load balancer in the HTTP_X_AMZN_TRACE_ID HTTP header.

/etc/nginx/nginx.conf
    log_format  main '$remote_addr $host $http_x_amzn_trace_id '
                     '[$time_local] "$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent" '
                     '"$http_x_forwarded_for" '
                     '$request_time $upstream_response_time $pipe';

The AWS trace format is different from the W3C Recommendation for Trace Context, but clear enough that we can write a regex to parse it.

The filter operator will drop all events matching the pattern so, as we are only interested in 5xx errors, we will create a regular expression that matches all other valid HTTP Status codes.

Making sure the collector can access the nginx log files

Since the otelcol-contrib service runs as the otelcol user, we need to make sure the permissions on the log files are set properly.

Terminal
$ chgrp -R otelcol /var/log/nginx

The logs are rotated, so we also make sure the new files will have the correct group.

/etc/logrotate.d/nginx
/var/log/nginx/*.log {
-    create 0640 nginx root
+    create 0640 nginx otelcol

Multi-line log records

We have other ways to ship error context, but if you are trying to export application exceptions to an aggregator, you will most likely deal with multi-line log records.

More to read

You might also like to read Shipping ssh login events through OpenTelemetry, failed experiment to ship OpenTelemetry events to GitLab or Fast-tracking server errors to a log aggregator on S3.

More technical posts are also available on the DjaoDjin blog, as well as business lessons we learned running a SaaS application hosting platform.

by Sebastien Mirolo on Thu, 19 Sep 2024


Receive news about DjaoDjin in your inbox.

Bring fully-featured SaaS products to production faster.

Follow us on