The Complete Linux Server Security Guide: SSH Keys, Fail2Ban & Beyond

A real-world Linux server hardening guide written from an actual server compromise and recovery on a Contabo VPS. SSH key authentication, Fail2Ban, iptables firewall, compromise detection, forensic cleanup, and a master checklist — every command tested on Ubuntu 24.04.

The Complete Linux Server Security Guide: SSH Keys, Fail2Ban & Beyond

Last updated: May 2026 · By JB (Muke Johnbaptist) — written after a real Contabo VPS compromise and forensic cleanup.

A real-world hardening guide — written from an actual server compromise and recovery.

Real World Incident Report
How the Attack Happened
The Takedown Strategy
Part 1 — Security Checklist: Is Your Server Safe Right Now?
Part 2 — Protection Setup: How to Harden Your Server
Part 3 — Is Your Local Machine Compromised?
Part 4 — Ongoing Monitoring Habits
Part 5 — Emergency Response Playbook
Part 6 — Recovery: What to Do If You Lose Your SSH Key
Master Security Checklist

Real World Incident Report

This guide was born from a real server compromise. Here is exactly what happened.

The Setup

A Cloud VPS running Ubuntu 24 was provisioned with default password-based SSH authentication. The server hosted a Node.js application, several Docker containers, a PostgreSQL database, and a Redis cache — a typical production stack.

The Abuse Report

Two weeks after provisioning, the hosting provider (Contabo) sent an abuse complaint. The server's IP (213.136.89.197) had been detected launching FTP brute-force attacks against over 160 other servers, attempting to guess passwords using usernames like admin, feinisnack, and octeniderm — running almost continuously for over 10 hours.

The server was suspended.

What the Forensic Investigation Found

After regaining access, a full investigation revealed a professional multi-stage hacking operation had been running on the server:

1. Perl IRC Botnet A Perl script disguised as /sbin/syslogd had been running since May 1st — 13 days. It was connected to an IRC command-and-control server at 192.253.248.9 on port 6667. The attacker was sending commands through an IRC channel, instructing the server to attack targets. The script had deleted itself after launching to avoid detection, leaving no file on disk — only a running process.

2. FTP Brute-Force Toolkit Hidden directories /tmp/.dar and /tmp/.eroz contained:

Compiled Go executables named bomb and brute
Hundreds of millions of target IP addresses across multiple text files
Lists of successfully compromised FTP credentials
Password lists used for the attacks

This was the toolkit that caused the Contabo abuse complaint.

3. WHM/cPanel Cracking Operation Hidden in /tmp/.ad/fastwhm:

A Go-based tool targeting WHM hosting control panels
A 54MB list of hosting domains to attack
GOOD.TXT — a file of successfully hacked cPanel accounts
An upload executable for deploying shells to compromised websites

4. Root Backdoor Account An account named mtab had been created with full root privileges (UID 0), disguised to look like a system file reference. This gave the attacker a persistent backdoor even if the SSH password was changed.

The Scale of Damage

The server had been used to:

Attack 160+ servers via FTP brute force
Attempt to compromise millions of IP addresses
Crack WHM/cPanel hosting panels across hundreds of domains
Store gigabytes of stolen credentials

All of this was running silently in the background while the legitimate application continued working normally.

How the Attack Happened

The attack chain followed a classic sequence:

Internet Scanner
      │
      ▼
Found port 22 open
      │
      ▼
Brute-forced root password (default/weak)
      │
      ▼
Gained root access
      │
      ▼
Downloaded attack toolkit to /tmp (hidden directories)
      │
      ▼
Launched Perl IRC bot (disguised as syslogd)
      │
      ▼
Created backdoor user 'mtab' with UID 0
      │
      ▼
Received commands via IRC → launched FTP attacks
      │
      ▼
Collected stolen credentials → cracked cPanel panels
      │
      ▼
Server suspended by hosting provider for abuse

The entire compromise was enabled by one thing: password-based SSH authentication on port 22 with no brute-force protection.

The Takedown Strategy

Here is the exact step-by-step strategy used to identify, neutralize, and clean the compromised server.

Phase 1 — Reconnaissance (Know Before You Act)

Before touching anything, assess the full situation:

# Who is currently logged in?
w
who -a
 
# What processes are consuming the most CPU?
ps aux --sort=-%cpu | head -20
 
# What outgoing connections does the server have?
ss -tnp
 
# What failed login attempts have there been?
lastb | head -30

Key find: ps aux revealed /sbin/syslogd using 99.7% CPU — a process that shouldn't exist.

Phase 2 — Identify the Threat

# The binary doesn't exist on disk — deleted after launch
ls -la /sbin/syslogd
# Result: No such file or directory
 
# But we can find the real binary through /proc
ls -la /proc/2074135/exe
# Result: /proc/2074135/exe -> /usr/bin/perl
 
# Read the full command
cat /proc/2074135/cmdline | tr '\0' ' '
 
# Check its network connections
ss -tnp | grep 2074135
# Result: ESTAB 213.136.89.197:59086 → 192.253.248.9:6667

Port 6667 = IRC. This confirmed a Perl IRC botnet phoning home to an attacker's server.

Phase 3 — Cut the Connection First

Before killing the process, block the attacker's server permanently:

iptables -A OUTPUT -d 192.253.248.9 -j DROP

Phase 4 — Kill the Malware

kill -9 2074135
ps aux | grep syslogd  # Verify it's gone

Phase 5 — Hunt the Toolkit

# Check all temp directories for hidden folders
ls -la /tmp/
ls -la /var/tmp/
 
# Investigate every hidden directory
ls -laR /tmp/.ad
ls -laR /tmp/.dar
ls -laR /tmp/.eroz
ls -laR /tmp/.popo
 
# Read any scripts
find /tmp/.ad /tmp/.dar /tmp/.eroz /tmp/.popo -type f | xargs cat 2>/dev/null

Found: Four hidden directories containing the complete attack toolkit.

Phase 6 — Destroy the Toolkit

rm -rf /tmp/.ad /tmp/.dar /tmp/.eroz /tmp/.popo

Phase 7 — Find and Remove the Backdoor

# Check for unexpected user accounts
cat /etc/passwd | grep -v nologin | grep -v false
 
# Found: mtab:x:0:1000::/home/mtab:/bin/sh
# UID 0 = full root access
 
# Remove it directly from passwd and shadow
sed -i '/^mtab:/d' /etc/passwd
sed -i '/^mtab:/d' /etc/shadow
userdel -r mtab 2>/dev/null
rm -rf /home/mtab

Phase 8 — Harden and Lock Down

Install all protections (covered in detail in Part 2):

# SSH keys, disable passwords, Fail2Ban, firewall

Phase 9 — Verify and Reboot

# Verify no suspicious processes remain
ps aux --sort=-%cpu | head -10
 
# Verify no outgoing connections to attacker
ss -tnp | grep -v "127.0.0.1\|YOUR_IP"
 
# Reboot to apply kernel updates
reboot

Phase 10 — Post-Reboot Verification

id mtab               # Should return: no such user
fail2ban-client status sshd
iptables -L -n | head -15
ss -tnp

Result: Clean server. Fail2Ban immediately began banning new attackers upon reboot.

Part 1 — Security Checklist: Is Your Server Safe Right Now?

Run through every item below. Each command should return the expected green result. Any red result needs immediate attention.

🔍 Check 1 — Who Is Currently Logged In?

w
who -a

✅ Green: Only your own IP addresses appear in active sessions. 🔴 Red: Unknown IPs or unknown usernames — someone else may be on your server.

🔍 Check 2 — Are There Suspicious Processes Running?

ps aux --sort=-%cpu | head -20

✅ Green: Only known processes — your app, docker, nginx, postgres, etc. 🔴 Red: Any of the following are serious red flags:

Unknown process consuming high CPU
Process names that sound like system tools but feel wrong (syslogd, kworker variants)
perl, python, bash running as root with no clear purpose

Investigate any suspicious process:

ls -la /proc/PROCESS_ID/exe
cat /proc/PROCESS_ID/cmdline | tr '\0' ' '
ss -tnp | grep PROCESS_ID

🔍 Check 3 — Are There Outgoing Connections to Unknown Servers?

ss -tnp

✅ Green: Only connections to known services (your database, APIs, etc.) 🔴 Red: Any ESTABLISHED connection to an unknown IP, especially on ports:

6667 = IRC botnet command and control
4444, 1234, 31337 = classic backdoor ports
Any connection your app should not be making

lastb | head -30

✅ Green: Few or no entries — or Fail2Ban is already banning the attackers. 🔴 Red: Hundreds of failed attempts from multiple IPs — active brute force attack in progress.

🔍 Check 5 — Are There Hidden Files in /tmp?

ls -la /tmp/
ls -la /var/tmp/

✅ Green: Only system directories (systemd-private-*, snap-private-tmp, cloud-init). 🔴 Red: Any hidden directory (starting with .) that isn't a system directory — especially ones with names like .ad, .dar, .popo, .x, .cache containing executables.

Check contents of anything suspicious:

ls -laR /tmp/.suspicious_dir
file /tmp/.suspicious_dir/*

🔍 Check 6 — Are There Unexpected User Accounts?

cat /etc/passwd | grep -v nologin | grep -v false

✅ Green:

root:x:0:0:root:/root:/bin/bash
sync:x:4:65534:sync:/bin:/bin/sync

Plus any accounts you personally created.

🔴 Red: Any unknown account, especially one with:

UID 0 (root privileges)
A shell like /bin/bash or /bin/sh
A name that looks like a system file (mtab, syslog, daemon2)

🔍 Check 7 — Are SSH Authorized Keys Clean?

cat /root/.ssh/authorized_keys

✅ Green: Only your own public key(s) — ones you recognise. 🔴 Red: Any key you did not add. Remove it immediately:

nano /root/.ssh/authorized_keys
# Delete the unknown key line, save and exit

🔍 Check 8 — Are Cron Jobs Clean?

crontab -l
cat /etc/crontab
ls -la /etc/cron.d/
ls -la /etc/cron.hourly/
ls -la /etc/cron.daily/
cat /etc/cron.hourly/*
cat /etc/cron.daily/*

✅ Green: Only standard system jobs (logrotate, apt-compat, man-db, sysstat). 🔴 Red: Any cron job running a script from /tmp, /var/tmp, or running perl, python, wget, curl downloading from unknown URLs.

🔍 Check 9 — Is SSH Password Authentication Disabled?

sshd -T | grep passwordauthentication

✅ Green: passwordauthentication no 🔴 Red: passwordauthentication yes — see Part 2 to fix this.

🔍 Check 10 — Is Fail2Ban Running?

fail2ban-client status sshd

✅ Green: Shows active monitoring with Currently failed and Banned IP list. 🔴 Red: Command not found or service not running — see Part 2 to install it.

🔍 Check 11 — Is the Firewall Active?

iptables -L -n | head -20

✅ Green: Shows INPUT rules with specific ACCEPT rules and a final DROP rule. 🔴 Red: Empty output or Chain INPUT (policy ACCEPT) with no DROP rule — all ports are open to the world.

🔍 Check 12 — Are System Binaries Intact?

dpkg --verify

✅ Green: No output (all packages verified). 🔴 Red: Lines starting with ??5?????? indicate a modified system binary — this suggests a rootkit.

🔍 Check 13 — Were Any Files Recently Modified?

find / -type f -mtime -7 2>/dev/null \
  | grep -v /proc \
  | grep -v /sys \
  | grep -v /run \
  | grep -v /dev \
  | grep -v /usr/lib/python3 \
  | grep -v ".pyc"

✅ Green: Only log files, your application files, and system update files. 🔴 Red: Modified system binaries in /bin, /sbin, /usr/bin, or unknown files in /tmp.

Part 2 — Protection Setup: How to Harden Your Server

Protection 1 — SSH Key Authentication with Named Keys (Critical)

Why: Eliminates password brute-force attacks entirely. Using a named key per server instead of the default id_ed25519 means you can tell at a glance which key belongs to which machine, rotate or revoke a single server's access without touching the others, and avoid the classic mistake of reusing one master key everywhere.

Step 1 — Generate a named key on your local machine

Pick a name that identifies the server (e.g. prod-api, myapp-vps, client-acme). The convention is <purpose>_ed25519.

ssh-keygen -t ed25519 \
  -f ~/.ssh/myapp-vps_ed25519 \
  -C "myapp-vps@$(hostname) - $(date +%Y-%m-%d)"

What each flag does:

-t ed25519 — the algorithm. Ed25519 is faster, shorter, and at least as secure as RSA 4096. Always pick this in 2026 unless you have a specific reason not to.
-f ~/.ssh/myapp-vps_ed25519 — the filename. This is what creates the named key — the file is now myapp-vps_ed25519 (private) and myapp-vps_ed25519.pub (public) instead of the generic id_ed25519.
-C "myapp-vps@$(hostname) - $(date +%Y-%m-%d)" — the comment baked into the public key. When you cat authorized_keys later, the comment tells you which key it is and when it was made. Format it however you like — purpose, machine, date.

You'll be prompted for a passphrase. Always set one — it encrypts the private key on disk so a stolen laptop doesn't equal a stolen key.

Step 2 — Copy the public key to the server

ssh-copy-id -i ~/.ssh/myapp-vps_ed25519.pub root@your_server_ip

-i ~/.ssh/myapp-vps_ed25519.pub makes sure you copy the named public key (otherwise ssh-copy-id copies every key in ~/.ssh/).

Step 3 — Add an entry in `~/.ssh/config` for one-word access

This is the productivity payoff. Once configured, you ssh into the server by name — no flags, no IP, no -i needed.

nano ~/.ssh/config

Append:

Host myapp-vps
    HostName        123.45.67.89
    User            root
    Port            22
    IdentityFile    ~/.ssh/myapp-vps_ed25519
    IdentitiesOnly  yes

Host myapp-vps — the short alias you'll type.
HostName — the server IP (or domain).
IdentityFile — points at your named private key.
IdentitiesOnly yes — tells SSH "only try this key, don't offer every key in ~/.ssh/". This avoids "Too many authentication failures" errors when you have several keys and Fail2Ban is watching.

If you later change the SSH port to 2299 (Protection 6 below), just bump the Port line.

Step 4 — Test the named key in a NEW terminal window

Do not close your current SSH session. Open a brand new terminal and try the alias:

ssh myapp-vps

If you see the server prompt, the named key works. If anything fails, fix it from the still-open original session before moving on.

Step 5 — Only after confirming the key works, disable passwords

sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^#*ChallengeResponseAuthentication.*/ChallengeResponseAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^#*PubkeyAuthentication.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
systemctl restart ssh

Verify:

sshd -T | grep passwordauthentication
# Expected: passwordauthentication no

⚠️ Before you disable passwords, read Part 6 — Recovery: what to do if you lose your SSH key below. Once PasswordAuthentication no is set and your key is gone, SSH is no longer an option — you have to recover through the VPS console.

Step 6 — How to copy your existing SSH key so you can back it up

Once everything works, you should immediately back up the private key to a safe place (password manager, encrypted USB, secure cloud vault). Here's how to view and copy any key you have on disk — whether it's the default ~/.ssh/id_ed25519 or a named one like ~/.ssh/grit_managed_deploy.

🔐 Private key handling rules. The private key (the file without .pub) is the secret. Treat it like a password:

Never paste a private key into chat, Slack, Discord, email, ChatGPT, GitHub Issues, or a public gist.

Never commit it to a git repo.

The matching .pub file is safe to share — that's the one that goes onto servers.

1. List what keys you actually have:

ls -la ~/.ssh/

You'll see pairs like id_ed25519 + id_ed25519.pub, or grit_managed_deploy + grit_managed_deploy.pub. The file without .pub is the private key; the one with .pub is the public key.

2. View a key in your terminal (so you can read it):

# Private key — KEEP SECRET
cat ~/.ssh/grit_managed_deploy
 
# Public key — safe to share
cat ~/.ssh/grit_managed_deploy.pub

A private Ed25519 key looks like this (truncated):

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
...
-----END OPENSSH PRIVATE KEY-----

3. Copy a key straight to your clipboard:

The cleanest way to back up a key is to copy it to the system clipboard, then paste it into your password manager (Bitwarden, 1Password, KeePassXC — all support secure notes / attachments).

macOS:

# Private key → clipboard
pbcopy < ~/.ssh/grit_managed_deploy
 
# Public key → clipboard
pbcopy < ~/.ssh/grit_managed_deploy.pub

Linux (most distros, requires xclip or wl-clipboard):

# X11
xclip -selection clipboard < ~/.ssh/grit_managed_deploy
 
# Wayland
wl-copy < ~/.ssh/grit_managed_deploy

If neither is installed: sudo apt install xclip (Debian/Ubuntu) or sudo dnf install wl-clipboard (Fedora).

Windows (PowerShell):

# Replace the filename with whatever key you want
Get-Content $HOME\.ssh\grit_managed_deploy | Set-Clipboard

Windows (Git Bash / WSL):

cat ~/.ssh/grit_managed_deploy | clip.exe

4. Make a portable encrypted backup:

If you'd rather copy the file itself somewhere (USB drive, Nextcloud, Dropbox), encrypt it first so a lost drive doesn't equal a lost key. Two good options:

Option A — age (modern, simple, recommended):

# Install once: brew install age   |   sudo apt install age
age -p ~/.ssh/grit_managed_deploy > grit_managed_deploy.age
# You'll be prompted for a passphrase — pick a strong one and store it in your password manager.
 
# To restore later:
age -d grit_managed_deploy.age > ~/.ssh/grit_managed_deploy
chmod 600 ~/.ssh/grit_managed_deploy

Option B — gpg:

gpg --symmetric --cipher-algo AES256 ~/.ssh/grit_managed_deploy
# Produces grit_managed_deploy.gpg — safe to upload anywhere.
 
# To restore later:
gpg --decrypt grit_managed_deploy.gpg > ~/.ssh/grit_managed_deploy
chmod 600 ~/.ssh/grit_managed_deploy

5. (Recommended) Save the matching public key as a separate note

You'll need the .pub content to add the key to a new server later. Just cat it and save the text to your password manager alongside the encrypted private key. Public keys are not secret.

6. Tell yourself what you backed up

In the password-manager note alongside the encrypted backup, write:

Which key this is (e.g. "grit_managed_deploy")
Which servers/services it's authorised on
When it was created
The passphrase to decrypt it (if different from the SSH passphrase)

When future-you needs to recover, this note is the difference between "5 minutes" and "two hours of guessing".

Protection 2 — Fail2Ban (Critical)

Why: Automatically bans IPs that repeatedly fail login attempts.

apt update && apt install fail2ban -y

Create strict config:

nano /etc/fail2ban/jail.local

[DEFAULT]
bantime  = 86400
findtime = 600
maxretry = 3
 
[sshd]
enabled  = true
port     = ssh
logpath  = %(sshd_log)s
backend  = %(sshd_backend)s
maxretry = 3
bantime  = 86400

systemctl enable fail2ban
systemctl restart fail2ban
fail2ban-client status sshd

Useful commands:

# See banned IPs
fail2ban-client status sshd
 
# Unban yourself if locked out
fail2ban-client set sshd unbanip YOUR_IP
 
# Watch live bans
tail -f /var/log/fail2ban.log

Protection 3 — Firewall with iptables (Critical)

Why: Blocks all traffic except what your server specifically needs.

# Allow established connections first (prevents lockout)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 
# Allow SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
 
# Allow HTTP and HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
 
# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
 
# Drop everything else
iptables -A INPUT -j DROP
 
# Make rules permanent
apt install iptables-persistent -y
netfilter-persistent save

Verify:

iptables -L -n --line-numbers

Protection 4 — Keep System Updated (Important)

Why: Unpatched vulnerabilities are a primary attack vector.

apt update && apt upgrade -y

Enable automatic security updates:

apt install unattended-upgrades -y
dpkg-reconfigure --priority=low unattended-upgrades

Always reboot when a kernel update is available:

# Check if reboot is needed
cat /var/run/reboot-required 2>/dev/null

Protection 5 — Block Known Attacker IPs (Important)

If you identify an attacker's IP or C&C server:

# Block all outgoing connections to attacker
iptables -A OUTPUT -d ATTACKER_IP -j DROP
 
# Block all incoming connections from attacker
iptables -A INPUT -s ATTACKER_IP -j DROP
 
# Save
netfilter-persistent save

Protection 6 — Change Default SSH Port (Recommended)

Why: Eliminates the vast majority of automated scanning noise.

nano /etc/ssh/sshd_config

Change:

Port 22

To:

Port 2299

Allow the new port and restart:

iptables -A INPUT -p tcp --dport 2299 -j ACCEPT
systemctl restart ssh

Connect going forward:

ssh -p 2299 root@your_server_ip

Protection 7 — Create a Non-Root Sudo User (Recommended)

Why: Running everything as root means any compromise = full access.

adduser yourusername
usermod -aG sudo yourusername
 
# Copy SSH key to new user
mkdir -p /home/yourusername/.ssh
cp /root/.ssh/authorized_keys /home/yourusername/.ssh/
chown -R yourusername:yourusername /home/yourusername/.ssh
chmod 700 /home/yourusername/.ssh
chmod 600 /home/yourusername/.ssh/authorized_keys

Test login as new user in a new terminal, then disable root login:

sed -i 's/^#*PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
systemctl restart ssh

Protection 8 — Regular Audit Script (Recommended)

Save this script on your server and run it weekly:

nano /root/security-audit.sh

#!/bin/bash
echo "======================================"
echo " SERVER SECURITY AUDIT - $(date)"
echo "======================================"
 
echo ""
echo "--- LOGGED IN USERS ---"
w
 
echo ""
echo "--- TOP CPU PROCESSES ---"
ps aux --sort=-%cpu | head -10
 
echo ""
echo "--- ACTIVE OUTGOING CONNECTIONS ---"
ss -tnp | grep ESTAB
 
echo ""
echo "--- FAIL2BAN STATUS ---"
fail2ban-client status sshd
 
echo ""
echo "--- USER ACCOUNTS WITH SHELLS ---"
grep -E "/bin/bash|/bin/sh|/bin/zsh" /etc/passwd
 
echo ""
echo "--- SSH AUTHORIZED KEYS ---"
cat /root/.ssh/authorized_keys
 
echo ""
echo "--- HIDDEN FILES IN /tmp ---"
ls -la /tmp/ | grep "^\."
 
echo ""
echo "--- RECENT FAILED LOGINS (last 10) ---"
lastb | head -10
 
echo ""
echo "--- FIREWALL STATUS ---"
iptables -L INPUT -n | head -10
 
echo ""
echo "--- SUSPICIOUS PROCESSES ---"
ps aux | grep -E "hydra|medusa|ncrack" | grep -v grep
 
echo ""
echo "======================================"
echo " AUDIT COMPLETE"
echo "======================================"

chmod +x /root/security-audit.sh

Run anytime:

bash /root/security-audit.sh

Part 3 — Is Your Local Machine Compromised?

Your server is only as safe as the machine you connect from. If your local machine is compromised, attackers can steal your SSH private key.

Check 1 — Are There Unknown Processes Running?

On Windows:

# Open Task Manager and check CPU/Network usage
# Or in PowerShell:
Get-Process | Sort-Object CPU -Descending | Select-Object -First 20

On macOS/Linux:

ps aux --sort=-%cpu | head -20

🔴 Red flags: Unknown processes using high CPU or network, especially at night.

Check 2 — Are There Unknown Network Connections?

On Windows (PowerShell as Administrator):

netstat -ano | findstr ESTABLISHED

On macOS/Linux:

ss -tnp
# or
netstat -tnp

Look up any unknown IP at https://ipinfo.io/IP_ADDRESS

🔴 Red flag: Connections to unknown IPs, especially on ports 6667 (IRC), 4444, or 1337.

Check 3 — Is Your SSH Private Key Safe?

# Check permissions on your private key (should be 600)
ls -la ~/.ssh/id_ed25519
# Expected: -rw------- (600)
 
# If wrong, fix immediately:
chmod 600 ~/.ssh/id_ed25519

On Windows: Right-click the key file → Properties → Security → ensure only your user has access.

🔴 Red flag: Key file permissions are too open, or the file was recently modified without your knowledge.

Check 4 — Check Your SSH Known Hosts

cat ~/.ssh/known_hosts

🔴 Red flag: Entries you don't recognise — could indicate someone has connected from your machine to unknown servers.

Check 5 — Check Startup Programs

On Windows:

Task Manager → Startup tab

Or:

Get-CimInstance Win32_StartupCommand | Select-Object Name, Command, Location

On macOS:

ls -la ~/Library/LaunchAgents/
ls -la /Library/LaunchAgents/
ls -la /Library/LaunchDaemons/

On Linux:

systemctl list-units --type=service --state=running
ls -la ~/.config/autostart/

🔴 Red flag: Unknown startup entries, especially ones running scripts or executables from temp folders.

Check 6 — Scan for Malware

On Windows: Run Windows Defender full scan, or install Malwarebytes (free).

On macOS:

# Install and run ClamAV
brew install clamav
freshclam
clamscan -r --bell -i /Users/yourusername

On Linux:

apt install clamav -y
freshclam
clamscan -r /home /tmp /var/tmp

Check 7 — Verify Your SSH Key Hasn't Been Stolen

If you suspect your private key was compromised:

Generate a new key pair immediately:

ssh-keygen -t ed25519 -C "new_key_$(date +%Y%m%d)"

Add the new public key to all your servers:

ssh-copy-id -i ~/.ssh/new_key.pub root@your_server_ip

Remove the old key from all servers:

nano ~/.ssh/authorized_keys  # on each server
# Delete the old key line

Delete the compromised old key pair from your local machine.

Part 4 — Ongoing Monitoring Habits

Security is not a one-time setup — it requires ongoing habits.

Daily (takes 2 minutes)

# Quick health check
fail2ban-client status sshd
ss -tnp | grep -v "127.0.0.1"

Weekly

# Run full audit
bash /root/security-audit.sh
 
# Check for system updates
apt update && apt list --upgradeable

Monthly

# Full system update
apt update && apt upgrade -y
 
# Review all user accounts
cat /etc/passwd | grep -v nologin | grep -v false
 
# Review SSH authorized keys
cat /root/.ssh/authorized_keys
 
# Review firewall rules
iptables -L -n --line-numbers
 
# Check system binary integrity
dpkg --verify
 
# Review cron jobs
crontab -l
cat /etc/crontab
ls -la /etc/cron.*

After Any Incident or Suspicion

Run the full audit script immediately
Check all user accounts
Check all authorized SSH keys
Check all outgoing network connections
Review processes by CPU usage
Check /tmp and /var/tmp for hidden directories
Review recent file modifications

Part 5 — Emergency Response Playbook

If you suspect your server is compromised right now, follow this exact sequence:

Step 1 — Don't Panic, Don't Reboot Yet

Rebooting destroys evidence. Investigate first.

Step 2 — Capture the Evidence

# Save process list
ps aux --sort=-%cpu > /root/incident_processes.txt
 
# Save network connections
ss -tnp > /root/incident_connections.txt
 
# Save user accounts
cat /etc/passwd > /root/incident_passwd.txt
 
# Save running process details
for pid in $(ps aux | awk '{print $2}' | tail -n +2); do
    echo "=== PID $pid ===" >> /root/incident_proc_details.txt
    ls -la /proc/$pid/exe 2>/dev/null >> /root/incident_proc_details.txt
done

Step 3 — Identify the Threat

# Find high CPU processes
ps aux --sort=-%cpu | head -5
 
# For each suspicious PID, find the real binary
ls -la /proc/SUSPICIOUS_PID/exe
cat /proc/SUSPICIOUS_PID/cmdline | tr '\0' ' '
ss -tnp | grep SUSPICIOUS_PID

Step 4 — Cut the Attacker Off

# Block attacker's C&C server immediately
iptables -A OUTPUT -d ATTACKER_IP -j DROP

Step 5 — Kill the Malware

kill -9 MALWARE_PID

Step 6 — Search and Destroy

# Hidden toolkit directories
ls -la /tmp/ /var/tmp/
find /tmp /var/tmp -type d -name ".*"
 
# Remove everything suspicious
rm -rf /tmp/.suspicious_dir
 
# Find and remove backdoor accounts
cat /etc/passwd | grep -v nologin | grep -v false
sed -i '/^suspicious_user:/d' /etc/passwd
sed -i '/^suspicious_user:/d' /etc/shadow

Step 7 — Harden Immediately

# Install Fail2Ban if not present
apt install fail2ban -y
systemctl start fail2ban
 
# Disable password SSH if not done
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh

Step 8 — Reboot and Verify

reboot

After reboot:

id suspicious_user      # Should say: no such user
ps aux --sort=-%cpu | head -10
ss -tnp
fail2ban-client status sshd

Part 6 — Recovery: What to Do If You Lose Your SSH Key

This is the scenario that scares people away from disabling password authentication: your laptop dies, your key is gone, and the server only accepts SSH keys. SSH is locked. Now what?

Good news — you are never actually locked out. SSH is just one access path. The VPS itself is still running, and every cloud provider gives you at least one out-of-band way back in: a browser console, a rescue boot, a metadata API, or support. The trick is knowing which lever to pull.

This section gives you the playbook, ordered by speed and reliability.

Recovery option 1 — VPS web console (5 minutes, works everywhere)

Every major VPS provider has a browser-based console that connects directly to your server's virtual screen and keyboard — no SSH involved. You log in with your normal root password (the one you set in the provider's panel when the server was created — not the SSH password, the provider account password).

Provider quick links:

Contabo → Customer Control Panel → Your VPS → VNC Console (sometimes labelled "Console" or "Remote Console").
Hetzner → Cloud Console → your server → Console (top right).
DigitalOcean → Droplet → Access → Launch Recovery Console or Console.
Linode → Linode → Launch LISH Console.
AWS Lightsail / EC2 → "Connect using SSH" → falls back to browser-based EC2 Instance Connect or Session Manager.

Once you have a console shell, the recovery is short:

# 1. Make sure you're root (or sudo to root)
sudo -i
 
# 2. Add your NEW public key to authorized_keys
mkdir -p /root/.ssh
chmod 700 /root/.ssh
echo "ssh-ed25519 AAAA…YOUR_NEW_PUBLIC_KEY… you@laptop" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
 
# 3. (Optional) Temporarily re-enable password auth as a backup
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication yes/' /etc/ssh/sshd_config
systemctl restart ssh

Now test the new key over SSH from your laptop, and once it works, flip PasswordAuthentication back to no and restart ssh again.

The web console expects you to know the root password that was set when the server was provisioned. If you never set one (or never recorded it), use Recovery Option 2.

Recovery option 2 — Reset the root password from the provider panel

If you don't remember the root password, every major provider lets you reset it from outside the VM, then power-cycle the server so the new password takes effect:

Contabo → VPS → Reset Password (sends the new root password to your account email).
Hetzner → Server → Reset root password (returns the new password in the panel).
DigitalOcean → Droplet → Access → Reset root password (emails it to you).
Linode → Linode → Settings → Reset Root Password.

After the reset, reboot the VM from the panel. Wait two minutes, then open the web console and log in with the new password. From there, follow the same three commands as Recovery Option 1 (add new key, optional password-auth toggle, restart ssh).

Recovery option 3 — Rescue / single-user boot (for tougher cases)

If the password reset doesn't work (rare, but happens when the VM's filesystem is in a weird state), boot the VPS into rescue mode. This boots a small live Linux image with your disk mounted as a regular partition — you become root automatically and can edit any file.

The rough flow on most providers:

In the provider panel, click Rescue / Boot from rescue ISO / Single user mode.
Reboot the VPS. You're dropped into the rescue shell.

Mount your real disk:

# Usually /dev/sda1 or /dev/vda1 — `lsblk` shows you the layout
mount /dev/sda1 /mnt

Edit SSH config to re-enable passwords (so you can get back in normally afterwards):

sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication yes/' /mnt/etc/ssh/sshd_config

Or append your new public key directly:

mkdir -p /mnt/root/.ssh
echo "ssh-ed25519 AAAA…NEW_KEY…" >> /mnt/root/.ssh/authorized_keys
chmod 700 /mnt/root/.ssh
chmod 600 /mnt/root/.ssh/authorized_keys

Reboot back to normal mode from the panel.

Recovery option 4 — Cloud-init / metadata SSH key injection

Newer VPS images ship with cloud-init, which re-reads the SSH keys configured in the provider panel on every boot. On these images you can:

In the provider panel, add a new SSH key to your account / project.
Attach it to the server (DigitalOcean, Linode, Hetzner all support this).
Reboot the VPS.

cloud-init will read the metadata service on boot and write the new key into /root/.ssh/authorized_keys for you. This works without ever opening a console.

Not all images support this (older Ubuntu Minimal or custom-built images may not have cloud-init). If you don't see your panel-managed SSH keys appearing on the server, fall back to Option 1.

Recovery option 5 — Contact provider support

The last resort if nothing above works (rare). All paid VPS providers have support tickets that can get you back in. Mention you've lost SSH access and provide proof of account ownership.

Prevention is cheaper than recovery

Before you ever need this section:

Always keep at least two named SSH keys registered with each server (e.g. myapp-vps_ed25519 on your laptop and a backup_ed25519 stored on a second machine or encrypted USB).
Back up your ~/.ssh/ directory to a password-protected archive — Bitwarden's secure attachments, an encrypted ZIP on cloud storage, or age into your password manager.
Record the provider root password in your password manager the day you create the VPS. The web console is your fail-safe — don't lose the key to the fail-safe.
Document recovery steps for each server in a private repo or vault — provider, control-panel URL, web-console path. When you need it, you'll be panicking; the doc removes the thinking.
Test the web console once on a non-critical server. Five minutes of practice now beats an hour of fumbling at 2am.

If you do all of the above, losing a laptop is an annoyance, not a crisis.

Master Security Checklist

🔍 Server Health Checks

🛡️ Protection Setup

💻 Local Machine Checks

No unknown processes running at high CPU
No unknown outgoing network connections
SSH private key permissions are 600
No unknown startup programs
Antivirus/malware scan completed
SSH known_hosts only contains servers I've connected to

📅 Ongoing Habits

Daily — check Fail2Ban status and outgoing connections
Weekly — run full audit script
Monthly — full system update and account review
Immediately after any incident — full forensic audit

Quick Reference — Commands to Memorise

# Am I being attacked right now?
lastb | head -20 && fail2ban-client status sshd
 
# Is anything suspicious running?
ps aux --sort=-%cpu | head -10
 
# Is anything connecting out to unknown servers?
ss -tnp
 
# Who is on my server?
w
 
# Is my firewall working?
iptables -L -n | head -10
 
# Run full audit
bash /root/security-audit.sh

This guide was written following a real server compromise and recovery. Every command was tested on Ubuntu 24.04. The incident resulted in a professional-grade forensic cleanup and a complete server hardening — all documented here so you never have to go through the same experience.

Stay paranoid. Stay patched. Stay safe.