Rsyslog Troubleshooting Guide

by

Initiating Debug Mode

To kick off debug logging from the get-go, prepend your rsyslog.conf file with these lines. This ensures debug logging activates immediately upon the rsyslog service launching:

$DebugFile /var/log/rsyslog.debug
$DebugLevel 2

After implementing the above and restarting rsyslog, it will start generating a detailed debug log.

Note: Debug logging can quickly consume disk space, leading to potential full disk issues if left running. It’s advisable to use this feature temporarily.

Sample Configuration: 50-default.conf

Path: /etc/rsyslog.conf/50-default.conf

auth,authpriv.*               /var/log/auth.log
*.*;auth,authpriv.none        -/var/log/syslog
daemon.*                      -/var/log/daemon.log
kern.*                        -/var/log/kern.log
lpr.*                         /var/log/lpr.log
mail.*                        /var/log/mail.log
user.*                        /var/log/user.log

mail.info                     -/var/log/mail.info
mail.warn                     -/var/log/mail.warn
mail.err                      /var/log/mail.err

*.=debug;\
  auth,authpriv.none;\
  news.none;mail.none         -/var/log/debug
*.=info;*.=notice;*.=warn;\
  auth,authpriv.none;\
  cron,daemon.none;\
  mail,news.none              -/var/log/messages

*.emerg                       :omusrmsg:*

This setup delineates how incoming messages are categorized and directed to appropriate log files or other destinations like emails or system alerts.

The configuration’s left column acts as a message filter, routing specific logs to designated files or actions. The right column lists the target destinations.

The omusrmsg module, paired with *, enables broadcasting messages to all logged-in users, useful for system-wide notifications such as impending reboots.

Understanding Debug Logs

Files within /etc/rsyslog.d/ (or any specified inclusion directory) are loaded alphabetically. This ordering is crucial for many Unix/Linux services, rsyslog included. The sequence is determined by ASCII character values, sorting numbers before letters, and uppercase before lowercase.

  • Number-Led Filenames First: Numeric filenames precede alphabetical ones, meaning 10-default.conf loads before auth.conf.
  • Uppercase Before Lowercase: Files starting with uppercase letters are read before those starting with lowercase, so A.conf precedes a.conf.
  • Comprehensive Sorting: The full filename dictates order, positioning 10-default.conf before 20-default.conf, and auth-logs.conf ahead of cron-logs.conf.

Locating “Action 5”

To pinpoint “action 5” within debug logs:

  1. Organize Files: List /etc/rsyslog.d/ contents in ascending order with ls -1 /etc/rsyslog.d/, reflecting rsyslog’s read sequence.
  2. Sequential Review: Begin with the alphabetically first file, progressing through each, tallying actions encountered.
  3. Maintain a Running Total: Continue your count across files in the order they appear.
  4. Identify the Target Action: Reaching the fifth count reveals the sought-after action.

An “action” in this context refers to any directive causing rsyslog to process log messages, whether by saving, forwarding, or executing additional scripts. Keeping track of these directives through each configuration file will lead you to the correct identification of “action 5,” based on rsyslog’s operational logic.