Lab 9.4 — Linux Forensic Investigation

Lab Overview

The Scenario

Your SOC received an alert: the linux-web-01 server made an outbound connection to a known C2 IP address at 03:17 UTC. The connection was blocked by the firewall, but the server may already be compromised. Your task is to perform a full forensic investigation using Velociraptor to determine:

1. How did the attacker get in? (Initial access vector)
2. What did they do? (Post-exploitation activity)
3. How are they persisting? (Persistence mechanisms)
4. What tools did they use? (Attacker tooling)
5. What's the blast radius? (Data accessed or exfiltrated)

The Velociraptor client running on this endpoint has pre-planted artifacts that simulate a realistic compromise — you'll collect and analyze them just as you would on a real investigation.

Part 1: Authentication Log Analysis

Collect auth.log Artifacts

In the Velociraptor web UI, navigate to the client endpoint and launch a new collection. Search for and select the artifact:

Linux.Sys.AuthLogs

This artifact parses /var/log/auth.log and extracts structured authentication events including timestamps, usernames, source IPs, and authentication outcomes.

Analyze the results and answer:

• How many failed SSH login attempts are present?
• What usernames were targeted (root, admin, deploy, etc.)?
• What source IP addresses appear in the failed attempts?
• At what time did the first SUCCESSFUL login from a suspicious IP occur?
• What user account was compromised?

Identify Lateral Movement

After initial access, attackers often try to move to other accounts. Look for:

su - root
sudo -i
ssh user@internal-host

Document every account transition in your investigation notes.

Part 2: Command History Reconstruction

Collect bash_history

Launch a collection for:

Linux.Forensics.BashHistory

This artifact collects .bash_history files from all user home directories, providing a record of commands the attacker executed.

Analyze the history and reconstruct the attack timeline:

1. Reconnaissance commands — What did the attacker run first? (whoami, id, uname -a, ifconfig, cat /etc/passwd)
2. Tool download commands — Did they use wget, curl, or scp to pull in tools?
3. Privilege escalation commands — sudo, su, kernel exploit execution
4. Data collection commands — find, cat, tar, zip on sensitive files
5. Cleanup commands — history -c, rm -rf, shred, log tampering

Click to enlarge

Part 3: Persistence Mechanism Discovery

Cron Jobs

Collect cron artifacts:

Linux.Sys.Crontab

Examine all cron entries across /etc/crontab, /etc/cron.d/, and per-user crontabs in /var/spool/cron/crontabs/.

Look for suspicious indicators:

• Jobs that run at unusual intervals (every minute, every 5 minutes)
• Jobs that execute scripts from /tmp, /dev/shm, or hidden directories
• Jobs that download and execute remote content (curl | bash)
• Jobs created or modified after the initial compromise timestamp

Systemd Services

Collect systemd artifacts:

Linux.Sys.Services

Examine all service unit files, especially any created in /etc/systemd/system/ after the compromise timestamp.

Red flags in systemd services:

• Service names mimicking legitimate services (e.g., system-update.service, cron-helper.service)
• ExecStart pointing to scripts in /tmp or hidden directories
• Restart=always ensuring the backdoor survives crashes
• Services enabled but not part of the base OS package

Startup Scripts

Check for modifications to shell initialization files:

Linux.Forensics.BashRC

Attackers often add reverse shells or tool downloads to .bashrc, .profile, or /etc/profile.d/ scripts so they execute on every login.

Part 4: Filesystem Artifact Analysis

/tmp and Staging Directories

Collect filesystem artifacts:

Linux.Search.FileFinder

Configure the artifact to search /tmp, /dev/shm, /var/tmp, and any hidden directories under /home/ for recently modified files.

Document every artifact found:

FILESYSTEM ARTIFACTS
═══════════════════════════════
Path: /tmp/.hidden/
Files Found:
  - reverse.sh (chmod +x, created [timestamp])
  - nmap_scan.txt (output from internal recon)
  - data_dump.tar.gz (compressed sensitive files)
  - .ssh_key (stolen private key)

Path: /dev/shm/
Files Found:
  - payload.elf (compiled binary, likely exploit or RAT)

File Timeline Analysis

For each suspicious file, collect MAC timestamps (Modified, Accessed, Created):

Linux.Forensics.Timeline

Sort all findings chronologically. The timeline should show the progression: access → recon → tool staging → escalation → persistence → data collection.

Click to enlarge

Part 5: Build Your Investigation Report

Compile all findings into a structured forensic investigation report:

LINUX FORENSIC INVESTIGATION REPORT
═══════════════════════════════════════
Endpoint: linux-web-01
Investigation Date: [today's date]
Investigator: [your name]
Tool Used: Velociraptor

1. INITIAL ACCESS
   Method: [SSH brute force / credential stuffing / exploit]
   Source IP: [attacker IP]
   Timestamp: [first successful auth]
   Account Compromised: [username]

2. POST-EXPLOITATION ACTIVITY
   Commands Executed: [list from bash_history]
   Tools Downloaded: [list with source URLs]
   Privilege Escalation: [method and timestamp]

3. PERSISTENCE MECHANISMS
   Cron Jobs: [suspicious entries with paths]
   Systemd Services: [backdoor service names and configs]
   Shell Scripts: [modified rc files]

4. STAGING & TOOLING
   Staging Directory: [path]
   Tools Found: [list with hashes]
   Data Collected: [what was targeted]

5. TIMELINE SUMMARY
   [HH:MM] - Initial brute force begins
   [HH:MM] - Successful authentication
   [HH:MM] - Reconnaissance commands
   [HH:MM] - Tool download
   [HH:MM] - Privilege escalation
   [HH:MM] - Persistence installed
   [HH:MM] - Data collection
   [HH:MM] - C2 connection attempted (blocked)

6. RECOMMENDATIONS
   - [Immediate containment actions]
   - [Evidence preservation steps]
   - [Remediation priorities]

Deliverable Checklist

Before completing the lab, ensure you have:

• Auth.log Analysis — Failed/successful login timeline with IPs, users, and timestamps
• Command Reconstruction — Full attacker command sequence from bash_history
• Persistence Inventory — All cron jobs, systemd services, and modified startup scripts identified
• Filesystem Artifacts — Files in /tmp, /dev/shm, and hidden directories cataloged with MAC timestamps
• Investigation Report — Complete structured report with all 6 sections filled

What's Next

In Lab 9.5 — Memory Forensics, you'll go beyond disk artifacts to analyze what's happening in live memory. Process injection, hidden network connections, and runtime-only malware leave no traces on disk — only in memory.