Detecting Linux Log Tampering: Gaps, Blanks and Stops

TL;DR. Root can edit any single log on a Linux host. What an intruder rarely manages is editing all of them consistently. Tampering shows up as disagreement between sources (a line in the journal but not in auth.log, a session in audit.log but not in wtmp, a lastlog time with no wtmp record) and as structural damage (sequence-number gaps, zeroed records, truncated files, stopped daemons). Rule out the benign causes, then report what each source says.

1. Journal sequence-number gaps

Every journal entry carries a sequence number from a counter shared by all journal files of the machine. Deleting a whole journal file, or rewriting a file without some entries, leaves a range of numbers that no collected entry holds.

Benign explanations to exclude first:

user-<UID>.journal files that were not collected (their entries use the same counter).
Archived files vacuumed by retention (the gap is at the start of the range).
A collection taken while journald was writing (the gap is at the very end).

A gap in the middle of the incident window, with user journals present, or one that coincides with an rm of a journal file in audit.log or in a sudo line, is a finding. The journal header also helps: head_entry_seqnum and tail_entry_seqnum say which range a file should contain. See journal forensics.

2. Auth lines in the journal, missing from auth.log

On most systemd hosts, rsyslog receives its lines from journald. The same sshd, sudo and PAM message therefore exists in both places. If the journal holds Accepted password for X from IP and auth.log, which covers that period, does not, the text file was edited (typically with sed -i or a log cleaner that knows only text files).

Check coverage first: the line must fall between the first and last line of the auth.log generations you have, and rsyslog must have been running (its start and stop are in the journal). The reverse, a line in auth.log but not in the journal, usually means the journal did not retain that period.

wtmp cleaners overwrite a record with zeros so the file size stays aligned. Signs:

records that are entirely zero (type 0, no user, 1970 date in utmpdump),
a file size that is not a multiple of the record size (384 bytes on x86_64, 400 on aarch64),
timestamps going backwards between consecutive records,
a zero-byte wtmp or btmp with an old modification time on a server in use.

Then cross-check: an interactive SSH session that auth.log, the journal and audit.log all show, but wtmp does not, while wtmp covers that time; or a lastlog entry (the last login of a UID) newer than any wtmp session of that user. Remember that non-interactive SSH (commands, scp, sftp) often has no wtmp record at all. Details in the wtmp guide.

4. Logging stopped or reconfigured

Look for:

rsyslog.service, systemd-journald.service or auditd.service stopping outside a reboot or package upgrade,
journalctl --vacuum-time, --vacuum-size, --rotate run interactively,
audit CONFIG_CHANGE records, auditctl -D (delete all rules) or auditctl -e 0 (disable),
edits to /etc/rsyslog.conf, /etc/rsyslog.d/, journald.conf (Storage=none, volatile) or /etc/audit/rules.d/,
a text log that stops mid-day while the journal keeps running: the problem is the syslog daemon.

Reboots and upgrades stop services too; read what surrounds each event.

5. Commands that clear logs and history

audit.log (with an execve rule), sudo lines and the journal can capture the clearing itself:

rm, shred, truncate or > file against /var/log/*, journal files or /var/log/wtmp,
scripts run with a log path as argument, such as a Python script given /var/log/wtmp and a user name,
ln -sf /dev/null ~/.bash_history, unset HISTFILE, export HISTSIZE=0, history -c (builtins and variable changes leave no execve record; the ln does),
journalctl --vacuum-*, utmpdump -r to rewrite wtmp from edited text.

logrotate deletes old files as root from cron on a schedule; an interactive command from a user session is different.

6. Time manipulation

A clock set backwards makes events look older. In the journal, realtime going down while sequence numbers go up exposes it; systemd-timesyncd or NTP messages and clock-change entries help. In wtmp, records going back in time are the equivalent.

Writing it up

For each indicator, state what the evidence shows and what else could explain it. "Journal sequence numbers are missing between 10:02 and 10:50; no user-1001.journal was found; a sudo line at 10:44:10 shows rm -f run on that file" is a finding. "Logs were deleted" is a conclusion the reader should be able to reach from it.

In the browser

Linux Log Parser checks these points automatically: journal sequence gaps (with a warning when no user journal was loaded), auth lines in the journal but missing from auth.log, blanked, partial and backwards wtmp records, logging daemons stopping, journal vacuums and audit rule changes, log-clearing commands, and interactive logins with no wtmp record. Each finding lists the events it rests on. The walkthrough shows three of them on one intrusion, and the auth.log cheat sheet lists more tampering tells per source.