nginx logs with recfiles

2025-11-16 01:34:35 +01:00
parent 71ef311c70
commit ebbc9ae1c8
1 changed files with 65 additions and 0 deletions
--- a/data/recfiles/nginx_logs.md
+++ b/data/recfiles/nginx_logs.md
@@ -0,0 +1,65 @@
+---
+title: "nginx logs with recfiles"
+tags: [ "data", "recfiles", "logs" ]
+requires: [ "Recfiles", "nginx" ]
+---
+
+The standard `nginx` log format has such a lack of consistency or meaning that you might squint your face into a whirlpool making sense of them:
+
+
+```nonsense
+18.97.14.85 - - [16/Nov/2025:00:52:12 +0100] "GET /posts/learning_without_experts/content.html HTTP/1.1" 200 1704 "-" "CCBot/2.0 (https://commoncrawl.org/faq/)"
+57.141.0.25 - - [16/Nov/2025:00:52:18 +0100] "GET /posts/hope_you_win/ HTTP/1.1" 200 61997 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
+201.17.157.249 - - [16/Nov/2025:00:52:19 +0100] "GET https://ttrpgs.com/post/wp/ HTTP/1.1" 200 45202 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+47.246.164.151 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css HTTP/1.1" 200 27109 "https://ttrpgs.com/post/wp/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+47.246.164.154 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/fonts/Metropolis-MediumItalic.woff2 HTTP/1.1" 200 28100 "https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+47.246.164.135 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/fonts/Metropolis-Regular.woff2 HTTP/1.1" 200 24152 "https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
+```
+
+Someone created this logging format on purpose, to make sure nobody could parse it with a hundred `column`, `cut`, or `awk` pipes.
+
+The problem lies in `/etc/nginx/nginx.conf`:
+
+
+```conf
+log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
+                  '$status $body_bytes_sent "$http_referer" '
+                  '"$http_user_agent" "$http_x_forwarded_for"';
+```
+
+Despite a request of three strings, this format returns one string.
+
+It can output to recfile format like this:
+
+
+```conf
+log_format  main  '\nIP: $remote_addr\n'
+                 'User: $remote_user\n'
+                 'Date: $time_local\n'
+                 'Request: $request\n'
+                 'Status: $status\n'
+                 'Bytes: $body_bytes_sent\n'
+                 'Referrer: $http_referer\n'
+                 'Agent: $http_user_agent\n'
+                 'XForward: $http_x_forwarded_for\n';
+
+access_log  /var/log/nginx/access.rec  main;
+```
+
+Note the newline (`\n`) symbol, required to start a new entry on a new line.
+
+1. `cp /etc/nginx.conf /etc/nginx.conf.bak`
+1. Change `/etc/nginx.conf` to match the format above.
+1. Check the file works with `nginx -t`.
+1. Restart the `nginx` service.
+1. Access that web page to make sure that at least one log exists.
+1. Check the file with `recfix /var/log/nginx/access.rec`.
+
+Once it works, you can add the usual recfile headers:
+
+
+```sh
+sed -i '1 i \ ' /var/log/nginx/access.rec
+sed -i '1 i %rec: Weblog' /var/log/nginx/access.rec
+sed -i '2 i %doc: nginx access logs' /var/log/nginx/access.rec
+```