Server unreachable: understand why it stopped responding and restore it
An unreachable server no longer responds over SSH, HTTP or sometimes even ping. Every service it hosts (websites, APIs, databases, email) becomes unavailable at once. Causes range from a simple full disk to a network incident at the provider, through memory exhaustion, an overly strict firewall or an attack.
The reflex of a hard reboot sometimes fixes the symptom but can make things worse: database corruption, loss of the logs that would have explained the origin, a server that goes down again a few minutes later. You first need to determine whether the problem is network, system or application related.
This page walks you through the checks to run from the outside, from the provider's rescue console and from the server itself, the most frequent causes, and the moments when a professional's intervention avoids data loss.
Typical symptoms
- ssh: connect to host ... port 22: Connection timed out, or Connection refused.
- Ping does not respond, or responds while every service is silent.
- All hosted sites and applications went down at the same time.
- The provider's VNC or KVM console shows "No space left on device", "Out of memory" or a kernel panic.
- The server responds intermittently, with very long delays between responses.
- The provider's dashboard shows CPU or network usage at 100% or an incident alert.
Possible causes
Full disk
Growing logs, accumulated backups, temporary files or an expanding database. Once the disk is full, services can no longer write and lock up, including SSH.
Memory exhaustion and the OOM killer
A process (MySQL, PHP-FPM, Node, Java) consumes all the memory; the kernel kills processes at random, sometimes SSH or the database, and the server becomes unusable.
Blocking firewall or fail2ban
A modified iptables, ufw rule or cloud security group, or your own IP address banned by fail2ban after failed login attempts.
Network or hardware incident at the provider
Rack, hypervisor or router failure, or maintenance operation: the server is healthy but unreachable.
CPU saturation or attack
Infinite loop, cron jobs piling up, crypto-mining script installed by an attacker or a denial-of-service attack saturating the bandwidth.
System update or reboot gone wrong
Unbootable kernel, file system check (fsck) at boot, network service not restarted or invalid SSH configuration after an update.
IP address or DNS change
IP address reassigned after an instance was recreated, floating IP not attached, or DNS record pointing to a machine that no longer exists.
Checks to perform
- 1
Test from the outside
ping ip-address, then nc -zv ip-address 22 and nc -zv ip-address 443 to see whether the ports respond. traceroute or mtr ip-address show where the traffic stops. Test from another network to rule out a ban on your IP.
- 2
Check the provider's console and graphs
Dashboards show CPU, memory, disk and network: a plateau at 100% or a sudden drop dates the incident. Check the provider's status page for an ongoing incident.
- 3
Connect through the rescue console
The provider's VNC, KVM or serial console gives access to the server even when the network or SSH is down. That is where kernel messages and the login prompt appear.
- 4
Check the disk
df -h for space, df -i for inodes (a disk can be "full" of inodes with free space left). du -sh /var/log /var/lib/mysql /home/* to find what is taking up the space.
- 5
Check memory and the OOM killer
free -m for the current state, dmesg -T | grep -i 'out of memory' or journalctl -k | grep -i oom to find out whether the kernel killed processes, and which ones.
- 6
Check the firewall and fail2ban
iptables -L -n or ufw status for local rules, fail2ban-client status sshd to see banned IP addresses, and the security groups or network rules in the cloud console.
- 7
Examine load and logs
uptime and top show the load and the hungriest processes; journalctl -xe and /var/log/syslog or /var/log/messages detail recent system errors; last -x lists reboots.
Solutions
Free up disk space
Purge or rotate logs (journalctl --vacuum-size, logrotate), delete old backups, clear caches, then restart services. Set up a disk space alert.
Address memory exhaustion
Identify the responsible process, adjust its configuration (innodb_buffer_pool_size, PHP-FPM pm.max_children, Java heap), add swap as a safety net and, if needed, resize the server.
Fix the firewall
Lift the ban on your address (fail2ban-client set sshd unbanip), restore the rule allowing SSH and HTTP/HTTPS, fix the cloud security group.
Reboot cleanly
From the rescue console, stop services in order (application, then database), then reboot. If fsck runs at boot, let the check complete.
Contain an attack
Block the source IP addresses or countries, enable the provider's anti-DDoS protection, kill unknown processes then deal with the compromise before returning to service.
Restore or rebuild
If the system is unrecoverable, restore a snapshot or rebuild the server from a backup, ideally with automated configuration to move fast.
When should you call a professional?
- The server hosts production data (database, customer files) and you want to avoid any risky manipulation.
- The rescue console shows file system, kernel or "Out of memory" errors you do not know how to interpret.
- The server goes down again after every reboot.
- You suspect an intrusion: unknown processes, abnormal load, suspicious outbound connections.
- You have no recent backup and no documentation of the server configuration.
How Agencei can help
- 1
Remote diagnosis
Network tests from several points, analysis through the provider's rescue console, review of system and kernel logs to date and explain the incident.
- 2
Return to service
Resource clean-up, firewall fix, orderly service restart or controlled reboot, snapshot restore if needed.
- 3
Fixing the cause
Tuning of hungry services, log rotation, update or reconfiguration, clean-up and hardening in case of compromise.
- 4
Data verification
Integrity check of the database and files after a hard stop, table repair if needed.
- 5
Monitoring and backups
Alerts on disk, memory, load and availability, tested automatic backups and, where relevant, a more resilient infrastructure.
Frequently asked questions
Should I reboot the server right away?
Not before looking at the rescue console and, if possible, the logs. A reboot often fixes the symptom but wipes the clues, and a server with a full disk or a corrupted database may not come back up properly.
Ping responds but nothing else, what is going on?
The machine and the network work, but the services do not respond: firewall, stopped services, full disk or exhausted memory. The provider's rescue console makes it clear.
How do I know whether it is an incident at the provider?
Check its status page, emails and customer area. If the server is unreachable even from the rescue console and the usage graph stops abruptly, the incident is probably on their side.
Is my data lost?
Rarely because of a simple outage. A full disk or memory exhaustion does not destroy data. A database stopped abruptly may need repair, hence the importance of regular, tested backups.
How do I prevent this from happening again?
Monitoring with alerts (disk, memory, load, availability), log rotation, appropriate sizing, controlled updates, a documented firewall and automatic off-server backups.
Related services
IT Support
Application and server support, incident handling and clear service commitments.
See this serviceServer Migration
Full transfer of a dedicated server or VPS to a new machine, with OS upgrade and a controlled cutover.
See this serviceDevOps Consulting
Setting up CI/CD, infrastructure as code, monitoring and GitOps practices.
See this serviceIT Maintenance
Monitoring, updates, backups and continuous improvement of your applications in production.
See this serviceAWS Services
Design, deployment and optimisation of AWS infrastructure: compute, databases, storage, IAM, costs.
See this serviceTell us about your project
Describe your need in a few lines: we come back to you with a first analysis and the next steps.