Why Linux Servers Get Compromised: Detection, Prevention, and Real-World Remediation

Mohamed Magdy

Mohamed Magdy

— 7 min read
Share
Why Linux Servers Get Compromised: Detection, Prevention, and Real-World Remediation

Linux powers the internet. Cloud platforms, databases, APIs, web services, containers, and internal infrastructure all run Linux. That ubiquity makes it an enormous target. But the reason so many Linux servers get compromised isn't because Linux is inherently broken. It's because once a system is deployed, it starts drifting. Updates lag. Temporary exceptions become permanent. Passwords stop rotating. Monitoring falls away. Over time, these operational gaps stack until attackers find a foothold.

The difference between a secure Linux infrastructure and a compromised one is rarely about fancy zero-days or advanced exploits. It's about whether someone is consistently checking the systems you deployed six months ago.

The attacker's actual workflow

Attackers don't start with advanced techniques. They start with reconnaissance — automated scans looking for:

  1. Exposed services (SSH, databases, APIs reachable from the internet)
  2. Old software (systems last patched months ago)
  3. Weak authentication (password-only SSH, default credentials, reused passwords)
  4. Forgotten systems (development boxes, testing infrastructure, cloud instances that outlived their purpose)

When one of these is found, exploitation is usually straightforward. The attacker doesn't need a zero-day. They need a service that's exposed, a password that wasn't rotated, or a patch that was skipped.

Detection first: you can't defend what you can't see

The security industry spends billions on tools that prevent attacks. Most Linux compromises happen because nobody is watching for the initial access.

Audit exposed services

Every service listening on the network is a potential attack surface.

# What's actually listening on this system?
sudo ss -tulpn

# Typical output:
# tcp  LISTEN  0  128  0.0.0.0:22    0.0.0.0:*  users:(("sshd",pid=1234,...))
# tcp  LISTEN  0  128  127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=5678,...))

# The key detail: 0.0.0.0 means "listen on all interfaces" = reachable from internet

For each service listening on 0.0.0.0, ask: Should this be internet-accessible?

  • SSH on :22 — maybe (but protect it)
  • PostgreSQL on :5432 — almost certainly not
  • Docker API on :2375 — definitely not
  • Kubernetes dashboard on :6443 — definitely not

If a service doesn't need to be internet-facing, firewall it:

# Block everything except SSH from specific IPs
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow from 203.0.113.0/24 to any port 22  # Admin access only
sudo ufw allow 80,443  # Web traffic
sudo ufw enable

Or use cloud provider security groups for AWS/Azure/GCP to restrict at the network level before it reaches the OS.

Audit authentication mechanisms

SSH is the most common initial access point. Most attacks start here because it's usually exposed and often weak.

# Is password authentication enabled?
sudo grep -E "^PasswordAuthentication|^PubkeyAuthentication|^PermitRootLogin" /etc/ssh/sshd_config

# Secure output should show:
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no

# If password auth is still on, you have a problem

Current state (check monthly):

# Pull SSH logs and see who's actually connecting
grep "Accepted" /var/log/auth.log | tail -20

# See failed login attempts (attackers probing)
grep "Failed password" /var/log/auth.log | wc -l

# If this number is high (>100/day), you're being scanned

Remediation:

# Disable password authentication
sudo sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config

# Disable root login
sudo sed -i 's/^PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

# Change SSH to non-standard port (obscurity, not security, but reduces noise)
sudo sed -i 's/^#Port 22/Port 2222/' /etc/ssh/sshd_config

# Test the config before reloading
sudo sshd -t

# Reload SSH
sudo systemctl reload ssh

Audit patch status

Old systems are easy targets. Most distributions release patches within days of a vulnerability disclosure. The gap between disclosure and patching is when you're vulnerable.

# Ubuntu/Debian: what's not patched?
sudo apt update && sudo apt list --upgradable

# RHEL/AlmaLinux/CloudLinux: what's not patched?
sudo dnf check-update

# How old are these unpatched packages?
# If it shows "kernel" or "openssl" or database packages, prioritize those

The hard truth: You can't patch everything instantly. Some updates break applications. Some require maintenance windows. The real question is: do you know which systems are unpatched?

Create a monthly audit:

#!/bin/bash
# save as /usr/local/bin/audit-patches.sh

echo "=== Unpatched Systems Report ==="
for host in $(cat /etc/hosts | grep -v "^#" | awk '{print $2}'); do
  ssh $host 'apt list --upgradable 2>/dev/null | wc -l' | \
    awk -v h=$host '{print h": " $1 " upgradable packages"}'
done

Run this monthly. If a system hasn't been patched in 90+ days, that's your risk.

Audit what's actually running

Malware often hides inside normal processes. Drift means the running system looks nothing like the deployed system.

# What processes are running right now?
ps auxf

# What's actually listening? (rerun this regularly)
sudo netstat -tulpn 2>/dev/null || sudo ss -tulpn

# What's starting at boot?
sudo systemctl list-unit-files --state=enabled

# What cron jobs exist?
sudo crontab -l
sudo for user in $(cut -f1 -d: /etc/passwd); do echo "=== $user ==="; sudo -u $user crontab -l 2>/dev/null; done

# What services are active?
sudo systemctl list-units --type=service --state=running

This is where drift shows up. If you're seeing services, users, or cron jobs you don't recognize, investigate immediately.

Audit privilege escalation paths

The most dangerous drift is in sudo permissions. Over time, operational convenience leads to "let this user run anything as sudo."

# What sudo permissions exist on this system?
sudo -l -U root

# What can each user actually do?
for user in $(grep -E "^[a-z]" /etc/passwd | cut -d: -f1); do
  echo "=== $user ==="
  sudo -l -U $user 2>/dev/null | grep -v "^sudo"
done

# Dangerous patterns to find and fix:
# user ALL=(ALL) ALL          <- full sudo access without password
# user ALL=(ALL) NOPASSWD:ALL <- same, but no password
# user ALL=(ALL) NOPASSWD:/usr/bin/docker <- Docker as any user is dangerous

Remediation: Use the principle of least privilege. A cPanel account should only be able to restart Apache, not run arbitrary commands.

# Edit sudoers safely
sudo visudo -c  # Check syntax
sudo visudo     # Edit

# Add specific permissions, not blanket access:
appuser ALL=(root) NOPASSWD: /usr/bin/systemctl restart myapp
webadmin ALL=(root) NOPASSWD: /usr/bin/certbot renew

Detection: find existing compromises

If a system was already compromised, what does that look like?

SSH key forensics

Attackers almost always add their own SSH keys for persistent access.

# Examine all authorized_keys files
find /home -name "authorized_keys" -exec echo "File: {}" \; -exec cat {} \;
find /root -name "authorized_keys" -exec echo "File: {}" \; -exec cat {} \;

# Unexpected entries = compromise indicator
# Look for keys you didn't add or keys from unknown sources

Process forensics

Malware often disguises itself as legitimate processes.

# List all running processes with their executable paths
ps auxf | head -20

# Check what those executables actually are
file /proc/<pid>/exe

# If it says "ELF" but the process name looks like "sshd", check its path
readlink /proc/<pid>/exe

# Cryptominers often masquerade as:
# - "system" or "kernel"
# - Legitimate service names
# - Look for processes running from /tmp or unusual locations

Memory and CPU forensics

Cryptominers and other persistent malware consume resources.

# What's using the most CPU?
top -b -n1 | head -15

# If you see sustained high CPU on an unknown process, investigate:
ps -p <pid> -o comm=,etime=,args=

# Check where it came from:
ls -la /proc/<pid>/cwd
cat /proc/<pid>/environ | tr '\0' '\n'

Scheduled task forensics

Attackers use cron, systemd timers, and at jobs for persistence.

# Cron jobs for all users
for user in $(cut -f1 -d: /etc/passwd); do
  echo "=== Cron jobs for $user ==="
  crontab -u $user -l 2>/dev/null
done

# Systemd timers
systemctl list-timers --all

# At jobs (one-time scheduled tasks)
atq

# Look for:
# - Jobs you didn't create
# - Jobs with obfuscated command names
# - Jobs that run frequently but serve no business purpose

File integrity forensics

Attackers modify files. Use file integrity monitoring to detect changes.

# Create a baseline of critical files
sha256sum /etc/ssh/sshd_config /etc/sudoers /etc/passwd > /root/baseline.txt

# Later, verify nothing changed:
sha256sum -c /root/baseline.txt

# If hashes don't match, a file was modified after deployment

Automate this with aide or tripwire for production systems.

Prevention: build systems that resist drift

1. Automation over manual steps

Every manual configuration is a place where drift will happen.

# Use configuration management (Ansible, Puppet, Chef)
# Version control your infrastructure
# Deploy from git

# Example Ansible playbook for hardening:
---
- hosts: linux_servers
  tasks:
    - name: Disable password authentication
      lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^PasswordAuthentication'
        line: 'PasswordAuthentication no'
      notify: restart ssh

    - name: Update all packages
      apt:
        update_cache: yes
        upgrade: dist
      when: ansible_os_family == "Debian"

    - name: Remove unnecessary services
      systemd:
        name: "{{ item }}"
        enabled: no
        state: stopped
      loop:
        - avahi-daemon
        - cups

  handlers:
    - name: restart ssh
      systemd:
        name: ssh
        state: restarted

2. Continuous monitoring

Set up alerts for the things that actually indicate compromise.

# Using auditd to track file modifications:
sudo auditctl -w /etc/shadow -p wa -k shadow_changes
sudo auditctl -w /etc/sudoers -p wa -k sudoers_changes

# Monitor logs for suspicious activity:
# - Failed SSH attempts > 10/hour
# - New users being created
# - Privilege escalation attempts
# - Unexpected network connections

# Use tools like osquery or auditbeat to send this to a SIEM

3. Immutable infrastructure

The harder a system is to change, the harder it is to compromise.

# Make critical files immutable
sudo chattr +i /etc/shadow
sudo chattr +i /etc/passwd
sudo chattr +i /etc/sudoers

# Make home directories read-only for sensitive users
chmod 555 /home/webapp

# Use containers instead of VMs when possible
# (if a container is compromised, re-deploy a fresh one)

4. Segmentation

Limit what each service can access.

# Don't run everything as root
# Create dedicated users for each service
useradd -r -s /bin/false nginx
useradd -r -s /bin/false postgres

# Restrict filesystem access
chown root:root /etc/shadow
chmod 640 /etc/shadow

# Use SELinux or AppArmor for process confinement
# (in production, these prevent most privilege escalation attempts)

The operating reality of Linux security

Perfect security is impossible. What you're actually aiming for is:

  1. Visibility — you know what's running on your systems
  2. Consistency — configurations don't drift after deployment
  3. Detection — if something does go wrong, you see it quickly
  4. Automation — security controls don't depend on manual effort

Most Linux compromises don't happen because of Linux. They happen because a system was deployed, worked fine for 6 months, and nobody checked on it again. The attacker doesn't need sophistication. They need time.

The systems that don't get compromised are the ones that get checked regularly. Whether that's a automated scan looking for exposed services, a monthly patch audit, or just running ps auxf and looking for things you don't recognize, consistency matters more than complexity.

Linux itself provides all the tools needed to run a secure infrastructure. The difficult part is maintaining that security over time, when priorities shift and operational reality gets messy. That's not a Linux problem. That's a human problem.

And those are the hardest ones to solve.

Read more