66 lines
3.5 KiB
Markdown
66 lines
3.5 KiB
Markdown
---
|
|
title: "nginx logs with recfiles"
|
|
tags: [ "data", "recfiles", "logs" ]
|
|
requires: [ "Recfiles", "nginx" ]
|
|
---
|
|
|
|
The standard `nginx` log format has such a lack of consistency or meaning that you might squint your face into a whirlpool making sense of them:
|
|
|
|
|
|
```nonsense
|
|
18.97.14.85 - - [16/Nov/2025:00:52:12 +0100] "GET /posts/learning_without_experts/content.html HTTP/1.1" 200 1704 "-" "CCBot/2.0 (https://commoncrawl.org/faq/)"
|
|
57.141.0.25 - - [16/Nov/2025:00:52:18 +0100] "GET /posts/hope_you_win/ HTTP/1.1" 200 61997 "-" "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)"
|
|
201.17.157.249 - - [16/Nov/2025:00:52:19 +0100] "GET https://ttrpgs.com/post/wp/ HTTP/1.1" 200 45202 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
|
|
47.246.164.151 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css HTTP/1.1" 200 27109 "https://ttrpgs.com/post/wp/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
|
|
47.246.164.154 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/fonts/Metropolis-MediumItalic.woff2 HTTP/1.1" 200 28100 "https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
|
|
47.246.164.135 - - [16/Nov/2025:00:52:22 +0100] "GET https://ttrpgs.com/fonts/Metropolis-Regular.woff2 HTTP/1.1" 200 24152 "https://ttrpgs.com/css/styles.dc38388a8f0b890e788bd3a99b7495d14e7d5ac4359ed3b49abeb778497863b284ad4cc7e496ef58c84139295f9bafed82f5a41345eda86bd2d429cccb7c2596.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
|
|
```
|
|
|
|
Someone created this logging format on purpose, to make sure nobody could parse it with a hundred `column`, `cut`, or `awk` pipes.
|
|
|
|
The problem lies in `/etc/nginx/nginx.conf`:
|
|
|
|
|
|
```conf
|
|
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
|
'$status $body_bytes_sent "$http_referer" '
|
|
'"$http_user_agent" "$http_x_forwarded_for"';
|
|
```
|
|
|
|
Despite a request of three strings, this format returns one string.
|
|
|
|
It can output to recfile format like this:
|
|
|
|
|
|
```conf
|
|
log_format main '\nIP: $remote_addr\n'
|
|
'User: $remote_user\n'
|
|
'Date: $time_local\n'
|
|
'Request: $request\n'
|
|
'Status: $status\n'
|
|
'Bytes: $body_bytes_sent\n'
|
|
'Referrer: $http_referer\n'
|
|
'Agent: $http_user_agent\n'
|
|
'XForward: $http_x_forwarded_for\n';
|
|
|
|
access_log /var/log/nginx/access.rec main;
|
|
```
|
|
|
|
Note the newline (`\n`) symbol, required to start a new entry on a new line.
|
|
|
|
1. `cp /etc/nginx.conf /etc/nginx.conf.bak`
|
|
1. Change `/etc/nginx.conf` to match the format above.
|
|
1. Check the file works with `nginx -t`.
|
|
1. Restart the `nginx` service.
|
|
1. Access that web page to make sure that at least one log exists.
|
|
1. Check the file with `recfix /var/log/nginx/access.rec`.
|
|
|
|
Once it works, you can add the usual recfile headers:
|
|
|
|
|
|
```sh
|
|
sed -i '1 i \ ' /var/log/nginx/access.rec
|
|
sed -i '1 i %rec: Weblog' /var/log/nginx/access.rec
|
|
sed -i '2 i %doc: nginx access logs' /var/log/nginx/access.rec
|
|
```
|