Diagnose the network in ten minutes

"It must be the network" is the most said and least verified sentence in incidents. Most of the time it is the service. But it gets said because nobody on the team knows how to check.

This is the routine for checking. Six commands, in order, working up the layers. Each one eliminates a hypothesis.

First: understand the symptom#

Before the commands, the type of failure already tells you a lot:

→ Total timeout, nothing answers → layer 3 or 4. Routing, firewall, service down.

→ Connects and hangs → layer 4, or the service is not answering.

→ Certificate error → TLS.

→ HTTP response with an error code → layer 7. The network is fine; the problem is the application.

→ Works sometimes → packet loss, one bad node behind a load balancer, or DNS with multiple records and one of them broken.

That classification takes ten seconds and saves a lot.

Step 1: does the name resolve?#

dig api.example.com
dig +trace api.example.com

If it does not resolve, it is DNS. The investigation is over, and you already know who to talk to.

The +trace shows the full chain, from the root to the authoritative server. Useful when the answer is inconsistent between locations.

Look at the TTL on the response. If you just changed the record and the TTL is 1800 seconds, half your clients will keep hitting the old IP for half an hour.

The classic DNS failures:

→ A high TTL at migration time. Lower it in advance.

→ Eternal cache in the process. Many languages cache DNS in-process and ignore the TTL. The server changes IP and the application never notices until it restarts.

→ ndots on Kubernetes. The default is 5, which means a name with fewer than five dots is tried against several search suffixes before it resolves. One external call can become four or five queries. At volume, that takes down cluster DNS. Check with cat /etc/resolv.conf inside the pod.

Step 2: does it reach the destination?#

mtr -rwzbc 100 api.example.com

ping settles little, because many networks block or deprioritise ICMP. Prefer mtr, which shows latency and loss hop by hop.

How to read it: loss on an intermediate hop that does not propagate to the following ones is usually the router deprioritising ICMP, and can be ignored. Loss that starts at a hop and continues to the destination is a real problem on that stretch.

Step 3: does the port open?#

nc -zv api.example.com 443

If the name resolves and the port does not open, it is a firewall, a security group, a network policy, or the service is not listening.

In the cloud, this is the step that most often finds the problem, and the cause is usually a security group that allows egress but not ingress, or a NACL blocking the return traffic.

Step 4: does TLS complete?#

openssl s_client -connect api.example.com:443 -servername api.example.com

Here you see the certificate chain, the validity and the name presented.

The common problems: an expired certificate (still happens, a lot); an incomplete chain, which works in your browser, with the intermediate cached, and fails on the server that does not have it; and the wrong SNI, when the same IP hosts several domains.

The -servername exists precisely to test SNI.

Step 5: does HTTP answer, and where does the time go?#

curl -w "dns:%{time_namelookup} conn:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" -o /dev/null -s https://api.example.com/route

This is the most useful command on the list, because it breaks the time down by phase.

→ high time_namelookup → DNS

→ high time_connect → network latency or queueing at the destination

→ high time_appconnect → the TLS handshake

→ high time_starttransfer with the earlier ones low → the server is slow. The network is fine.

That last line is the one that most often ends the "it is the network" argument.

Run it twice in a row: the second should be faster because of the DNS cache.

Step 6: how does the connection look from the inside?#

ss -ti
ss -tan | awk '{print $1}' | sort | uniq -c

The first shows TCP connections in detail: window, estimated round trip time, and retransmissions. High retransmission means packet loss, and packet loss destroys throughput far beyond what the percentage suggests, because TCP reads it as congestion and shrinks the window.

The second counts the states, and it settles a classic confusion:

→ Lots of CLOSE_WAIT = your bug. The other side closed and your application never called close. Each one holds a descriptor.

→ Lots of TIME_WAIT = normal in a high volume service. It appears on the side that closed, and lasts about 60 seconds by design. It only becomes a problem when ephemeral ports run out, and the right fix is not to tune the kernel: it is to stop opening a new connection on every call.

The phantom bug: MTU#

One case worth mentioning because it is hard and recurring.

Symptom: the connection establishes, small requests work, and large requests hang with no error.

Cause: the path MTU is smaller than your host's, which is common with VPNs, tunnels and some cloud networks. The large packet needs fragmenting, the router sends an ICMP "fragmentation needed" (type 3, code 4), and somebody along the way blocks ICMP.

The sender never finds out. The packet disappears. And the symptom looks like the application.

Fix: MSS clamping on the router, or lowering the MTU. Diagnosis: ping -M do -s 1400, varying the size until you find where it stops getting through.

Save a portrait of normality#

The most useful advice in this article:

Run the six commands against your most critical service while it is healthy. Save the output in a versioned file.

The next time something breaks, you will have something to compare against, which is exactly the thing that is always missing at three in the morning.