Exporting HTTP errors through OpenTelemetry
by Sebastien Mirolo on Thu, 19 Sep 2024Having gained some insights into OpenTelemetry through previous experiments, we planned to forward 50x nginx log records to an aggregator through OpenTelemetry.
The filelog receiver
A OpenTelemetry nginx module but it requires to compile some C++ code, then install a shared library on the machines we are running nginx on.
NGinx is happily writing a line for every HTTP requests into a log file on disk. So for now, we will use the filelog receiver to monitor nginx log files and spit out OpenTelemetry log events we are interested in.
receivers: filelog: include: [ /var/log/nginx/*-access.log ] operators: - type: regex_parser regex: '^.*Root=(?P\S+)-(?P \S+)-(?P \S+) \[(?P .+)\] "[^"]+"\s+(?P \d+)\s+.*$' timestamp: parse_from: attributes.time_local layout: '%d/%b/%Y:%H:%M:%S %z' status_code: parse_from: attributes.status_code trace: trace_id: parse_from: attributes.trace_id - type: filter expr: 'attributes.status_code matches "^[1-4][0-9]{2}$"'
First, we are running the nginx server behind an AWS Load balancer. We have thus
configured the nginx server to record the trace_id
created
by the load balancer in the HTTP_X_AMZN_TRACE_ID
HTTP header.
log_format main '$remote_addr $host $http_x_amzn_trace_id ' '[$time_local] "$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '"$http_x_forwarded_for" ' '$request_time $upstream_response_time $pipe';
The AWS trace format is different from the W3C Recommendation for Trace Context, but clear enough that we can write a regex to parse it.
The filter
operator will drop all events matching the pattern
so, as we are only interested in 5xx errors, we will create a regular expression
that matches all other valid HTTP Status codes.
Making sure the collector can access the nginx log files
Since the otelcol-contrib
service runs as the otelcol
user, we need to make sure the permissions on the log files are set properly.
$ chgrp -R otelcol /var/log/nginx
The logs are rotated, so we also make sure the new files will have the correct group.
/var/log/nginx/*.log { - create 0640 nginx root + create 0640 nginx otelcol
Multi-line log records
We have other ways to ship error context, but if you are trying to export application exceptions to an aggregator, you will most likely deal with multi-line log records.
More to read
You might also like to read Shipping ssh login events through OpenTelemetry, failed experiment to ship OpenTelemetry events to GitLab or Fast-tracking server errors to a log aggregator on S3.
More technical posts are also available on the DjaoDjin blog, as well as business lessons we learned running a SaaS application hosting platform.