The 4G Blog

In this post, I’ll document various network-related problems I’ve encountered, as well as steps and tools to troubleshoot them.

Common Issues

Loops – a loop happens when, through however many hops, devices, and cables, a device ends up plugged into itself. This could happen when both ends of the same cable are plugged into two different ports on a device, or when a device is plugged into another device twice, or any series of connections that result in a loop. Although STP (Spanning Tree Protocol) is supposed to prevent this from causing a problem, we’ve had limited success simply relying on that.

The problem with loops arises when broadcast information is put onto the network – this is a message that should be received by the device and then sent out on all appropriate ports. The problem is when that broadcast comes back through the loop on the other side – the network device sees it as a new broadcast, and so broadcasts it again. This causes massive network congestion, and in our experience generally comes across as network instability – the connection drops randomly, or is sometimes slow, etc. It can be difficult to track down since it’s an inherently-physical problem that has to be handled in-person.

The easiest troubleshooting is probably to ask, “Was something new plugged into the switch/device, or were any changes made to what’s plugged in?” Occasionally this is enough to find the troublesome cable. Other basic troubleshooting is simply to unplug cables and then plug them back in until the issues comes back. Additionally, cables can be traced – either physically or with a cable tracer/toner (see next post) to see if the signal comes back on a different line. When the troublesome cable is identified it can be removed.

Network port covers/caps can be put into unused ports on a network device to make this harder to accidentally do in the future.

IP-Assignment Issues

This comes in several forms. Two of the more common are, first, DHCP Exhaustion, where there are more devices on the network asking for an IP address than the scope allows, and secondly, Duplicate IPs, where two devices have the same IP. The first can be resolved by lowering the lease time (so disconnected IPs are freed up quicker) or expanding the scope to allow more devices. The second issue, where two devices have the same IP, arises when one (or both) devices are set static – usually when one device is static, but its IP address is within the DHCP scope so it’s assigned to something else. The best way to avoid this is either to make sure static IPs are outside the scope, or (more safely) to add DHCP reservations even for devices that  are static (which has the added benefit of assigning the correct IP if the device somehow loses its static settings and reverts to a more factory-default DHCP mode).

Routing Issues

        Occasionally, an issue arises where a device simply fails to communicate with the destination. This can be caused by a bad subnet mask, where the device has a faulty “understanding” of what its local network is, or it can be caused by routing issues caused either by bad (or missing) static routes, or a Default Gateway that is not actually on the local network. This is resolved by putting the correct information in the device.

Tools Referenced Above

• Ipconfig – this returns the IP information for the device. With the -all flag, it will return ALL information to have a network connection, as well as the device MAC address and other features.
• Ping – this is used to test connectivity to a given IP address. With the -a flag, it will attempt to resolve the hostname of the IP. With the -t flag, it will ping continuously until interrupted, in contrast with normal ping which a set number of attempts.
• Tracert/traceroute – mentioned in a previous post, this will return a list of all IPs encountered between the source and destination.
• Arp – this will return a list of MAC addresses and matching IP addresses.
• Nslookup – this command will query DNS for the hostname of a given IP address.

Physical Tools:

Cable tester & toner – in contrast to the above, these are physical tools used to determine a cable type, length, design, and check for faults and other issues (in the case of a tester). A toner will put a signal on the cable that, when paired with a probe, will return an audible sound allowing you to match one end of the cable to the other.

When troubleshooting a network problem, it’s important to follow the proper troubleshooting methodology before trying to fix something.

Methodology

First, identify clearly what the issue is. Be aware that users may not use terminology in the proper way – “we have no internet connection” doesn’t necessarily mean there’s an internet outage: this could also be caused by DNS failures, or DHCP scope exhaustion, and more – all of which would look like an internet outage. “I can’t start up my computer” would sound like a power/hardware issue, but if by that they mean that they can’t remote into a remote server that is very different from what they said. Therefore, it’s best to SEE what the issue precisely is, either by remoting into the PC or having the user read all errors off and walk through what they’re doing. It can also be important to verify if it’s just one user or all users/devices at a location.

Next, establish the probable theory of what’s wrong. If you can Ping an external IP but can’t Ping a domain name, it’s probably a DNS issue. If the device is getting an APIPA address (169.254.*.*), confirm that the network connection is good. Then check if the DHCP scope is exhausted – there just might not be any IP addresses left to lease to the device. If a user can’t log onto the PC because the domain isn’t available, check the network flow from the PC to the domain controller.

As an aside, I think it’s important to have a good mental map of the entire network flow. Give thought to the user’s inputs, and then consider the flow of data from start to finish and test each step. For a user who’s logged into a remote session and printing via port forwarding to a local printer, if the print is failing, you can immediately rule out an ISP outage – the user is still in the remote session. So consider the flow: Windows Print Spooler to the remote firewall, over the internet (either public or IPSec tunnel), to the local firewall, over the local network infrastructure, to the physical printer. Check if you can ping it locally. Can you send a test page locally? If so, then the issue is between the remote server and the local firewall. If you can’t, the issue is on the local network. Try a known-good cable and/or port. Confirm the printer is on.  The mental model of the packet flow is crucial to troubleshooting this.

(yes this is a common issue with our Tier-1 staff).

Finally, after resolving what seems to be the problem, test that it’s resolved. Aside from pings/etc., the best way is to actually confirm that the initial problem is now working. If so, you’re done. If not, you’ll then need to continue gathering information.

DOCUMENTATION!

Finally, perhaps the most crucial part – document your findings! In the simplest cases (a cord went bad, a connection was unplugged, the printer was offline), simply add a note to check that for the next user. That way something obvious won’t be repeatedly missed because someone thought it was TOO obvious – it’s important to remember that end users can be utterly confused by things that we take for granted within the industry – insider knowledge can lead to overlooking common issues AND create frustrated clients who may feel insulted or belittled or talked-down to. Adding a note to confirm those easy things – possibly even with a comment to the user of, “You may have already checked this, but can you confirm if…” can go a long way.

On the other hand, occasionally the problem is caused by something we’ve done wrong on our end in configuration – perhaps we set a DHCP reservation wrong (or forgot one), or we just have a fundamentally bad network design. In those cases, it’s important to document that as well so that, moving forward, the problems can be mitigated before they arise – and perhaps work can be done to clean up the technical debt of this potential problem lurking at other locations.

In my next post, I’ll give details on some specific common issues & how to resolve and test them.

(Yes, this entire post is written from hard-won experience…)