Packet Loss Monitoring and Troubleshooting

Packets, also called network packets, are tiny units of data sent and received over a network. Everything you do on the internet is made up of packets, from answering VoIP calls to sending emails.

Packets are tiny portions of the overall message sent or function executed on the network. Packets move through a wide range of different hubs—fiber optics, wireless routes, and copper cables—to reach their destination, following the most direct route. Taking the most direct route helps the network evenly distribute its load across multiple points, which increases operational efficiency and enhances performance. When packets successfully reach their destination, they receive a timestamp.

However, packet loss can occur when packets don’t successfully make it to their destinations. A packet is considered lost when it’s been delayed or misplaced on its way through the network. When a packet is delayed, it’s timed out. Your network will continue trying to deliver the packet, but when it arrives, it will most likely be incomplete. This process is called “retransmission timeout” or RTO. Your packets will arrive eventually, but they’ll be late and not usable.

Packet loss can slow down your network, negatively impact throughput and bandwidth, and cause bottlenecks. What’s more, packet loss can also be expensive. If you don’t get a handle on packet loss you may be forced to continue buying additional bandwidth and IT infrastructure to compensate for the lag.

Packet loss has many different causes. Sometimes packet loss can be caused by bad actors trying to attack your network, but it’s far more likely for packet loss to be the result of issues within your network. Here are the top ten causes of packet loss.

1. Network or link congestion: Network congestion is the most common cause of packet loss. Networks only have so much space available, and network use ebbs and flows throughout the day. At peak times when network use hits maximum capacity, like after lunch when many employees are logging back in to their computers at the same time, packet loss can be more likely.

   When a network is trying to play catch up and accommodate for maximum traffic, packets are the first things to get dropped in favor of more critical operations. Packets must wait to be delivered. Luckily, a good packet loss monitoring tool will try to complete packet transmissions by automatically resending packets until congestion clears up or slowing down transfer speeds across the network to give more packets a chance to make it through.

2. Overloaded devices: Sometimes the devices on your network are working too hard. In this case, the packets might make it to their destinations, but by then, the network is too weak to process the packets and it sends them back. It’s good if the device has a buffer in place to automatically put packets in a holding pattern until they can be resent, but these buffers can get filled up quickly and end up dropping excess packets anyway.
3. Inadequate IT infrastructure: Many network administrators are forced to cobble together a functioning network monitoring solution from multiple disparate technologies. Unless these tools are specifically designed for packet loss monitoring, opportunities to reduce packet loss are bound to fall through the cracks.
4. Faulty network hardware: Old or glitchy hardware that cannot scale with your growing enterprise can weaken the network and make it more susceptible to packet loss.
5. Poor Wi-Fi connection: Wi-Fi packet loss is common because when data is sent through the air, it’s easier for packets to get lost in transit. This becomes even more likely when packets are sent over long-distance internet connections.
6. Software bugs: Unpatched software is vulnerable to bugs with a negative impact on network performance and prevent it from adequately transmitting packets.
7. Firewalls: Sometimes the measures you take to protect your network can end up hindering it. An upstream firewall might be filtering only certain types of traffic, which can cause some websites or services to randomly load or fail.
8. Duplex mismatch: When two ends of a link are operating at different speeds, some packets will be lost on the link.
9. Excessive hops: The packet sender can choose the maximum number of hops a packet will go through en route to its destination, which is known as Time to Travel (TTL). Each time the packet passes through a router, the TTL drops by one and eventually reaches zero, at which point the packet will be dropped. If a packet gets rerouted through too many hops, the TTL will reach zero before the packet reaches its destination and the packet will be dropped.
10. Cyberattack: It’s not very likely, but sometimes packet loss is intentionally caused by bad actors. If you notice a sharp drop in packet success or a sudden, significant slowdown in network speed, you might be experiencing a cyberattack. Unfortunately, there’s not much you can do about packet loss if your network is already under attack. If you act quickly enough, you can use an access list to block the hacker’s IP address.

If you’re experiencing packet loss, it will typically be because of an internal network issue. The best way to drill down and start troubleshooting packet loss is to leverage a comprehensive packet loss tool.

The most straightforward way to monitor packet loss is to ping the element you want to monitor, like an IP address or website, and check for evidence of packet loss or latency.

If the element takes a long time to respond to the ping, this indicates a problem that needs investigating but might not amount to packet loss. If the element doesn’t respond to the ping at all, and you’ve checked to ensure everything is functioning properly on your end, you know packets aren’t reaching their destinations. You can also monitor packet loss by checking VoIP performance metrics and statistics for latency.

SolarWinds VoIP & Network Quality Manager (VNQM) is a powerful packet loss monitoring solution that gives you the real-time reporting, historical performance metrics, and Quality of Service (QoS) tools you need to effectively monitor your Network Service Assurance-capable network.

Quality of Service (QoS) settings can help you monitor packet loss and prevent it by ensuring network resources are triaged appropriately. QoS settings will divert resources to the devices and applications that need it first, like video calls and streamed content.

SolarWinds VNQM relies on Network Service Assurance operations to evaluate the performance of your network. With invaluable insights into latency, packet loss, jitter, MOS, and other QoS at your fingertips, VNQM provides you full visibility into the health and performance of your network.

SolarWinds VoIP & Network Quality Manager is an enterprise-grade packet loss monitor designed to take your monitoring efforts to the next level. Packet loss is one of the most prominent issues plaguing networks using VoIP, and network administrators need a specialized packet loss monitoring tool.

VNQM helps facilitate granular packet loss monitoring by allowing you to monitor packet loss, in real time, using key VoIP performance metrics indicating packet loss. It automatically generates and stores call detail records and compares them with WAN performance to help you pinpoint issues and fix packet loss. You can use these historical records in conjunction with real-time data to create VoIP deployment performance baselines against which you can compare future performance.

VNQM lets you search through VoIP call records filtered according to the top packet loss monitoring metrics, which adds another dimension to troubleshooting packet loss. With this feature, you can easily find where packet loss is moving beyond acceptable levels and act accordingly to fix packet loss before it spirals into sustained VoIP call problems and network performance issues.

One of SolarWinds VNQM’s most helpful features is its historical reports. Across all of the VoIP-enabled devices on your network, VNQM produces the following predefined historical reports:

• Calls: Last 24 Hours
• PRI Gateway Utilization: Last 24 Hours
• Jitter: Last 30 Days
• Latency: Last 30 Days
• MOS: Last 30 Days
• Packet Loss: Last 30 Days
• VoIP Nodes Availability: Last 30 Days

SolarWinds VNQM’s historical reports on packet loss are of particular note. By tracking all packet loss on your network over a 30-day period, VNQM helps you identify the average packet loss for call paths between source and destination. 

SolarWinds VNQM also allows you to set up instant alert notifications when the Mean Opinion Score (MOS) on any segment of your network falls below a certain threshold. As soon as an alert becomes triggered, you can tap into resources found on SolarWinds Platform Web Console and efficiently drill down on the underlying issue.

To reduce and fix packet loss, consider the following tips and best practices:

• Test your network’s connection: Packet loss can transpire when network connectivity falters. If you suspect packet loss, check that all network devices, cables, and connections function properly.
• Diagnose the root cause behind the packet loss: Businesses rely on enterprise-grade network monitoring tools to pinpoint the source of packet loss and drill down on the root cause. Network interference, interruption, and congestion can all lead to packet loss. Powerful network monitoring tools like VNQM cut through the noise, efficiently get to the heart of your packet loss problem and allow you to take action accordingly.
• Improve your hardware: In addition to network disruptions, hardware malfunctions or faulty software can also be causes of packet loss. Since any of these factors can induce packet loss, it makes sense why organizations should invest in high-quality network monitoring tools like VNQM. If it turns out your network hardware is the culprit because it is out-of-date or simply deficient vis-a-vis your network demands, it might be time to pursue hardware upgrades. Upgrading your routers, switches, and network hardware can help unlock performance gains and decrease packet loss.
• Assess your network configuration: Sub-optimal network configuration can also be responsible for packet loss. To optimize your network configuration, check that bandwidth has been appropriately allocated and that the right quality of service (QoS) settings are in place to support your network activity.
• Utilize a VPN: A virtual private network (VPN) can help minimize the impact of packet loss that occurs via the Internet. Designed with encryption and other security protocols, VPNs can bolster the dependability and quality of network communications.

SolarWinds VoIP & Network Quality Manager is a scalable packet loss monitoring tool that can help you create the network conditions that lead to optimal VoIP delivery.

VNQM includes many hallmark features that help you troubleshoot packet loss more efficiently and effectively. With an enterprise-grade VoIP monitoring solution like VNQM, you can track key VoIP call performance metrics surrounding call quality and packet loss. By making sense of termination call codes, MOS, latency data, and more, VNQM helps you quickly identify the underlying problem driving your packet loss.

VNQM maintains a comprehensive log of call records, which also comes in handy when determining the nature of any problem on your network. You can efficiently sort through records—isolating call quality, highlighting frequent errors, and filtering for other criteria—in cases where packet loss surpasses tolerable thresholds.

Once you add and properly configure your call manager devices to SolarWinds VNQM, VNQM gives you the tools to monitor network performance and troubleshoot packet loss effectively. SolarWinds VNQM collects and analyzes key performance metrics, distilling vital network data into intuitive and easy-to-consume visuals. VNQM’s graphs, charts, and tables not only help you monitor your network's current fitness but also help you track service quality network trends over more extended periods.

VNQM works in harmony with SolarWinds Network Performance Monitor (NPM). VNQM’s seamless integrations, real-time status updates, and cutting-edge presentation features empower you to troubleshoot packet loss with confidence.

With four licensing models to choose from—SLA 5, SLA 25, SLA 50, and SLA X—you can deploy SolarWinds VNQM on three levels. Ranging from SLA 5 on the low end to SLA X, which offers near-unlimited capacity, the license you opt for ultimately dictates the number of IP Phone levels, node levels, and Network Assurance site levels you have available to deploy.

Network Service Assurance Sites: Note that if a Cisco device is the source of a Network Service Assurance operation, it falls under your license.

IP Phones: Active IP phones connected to your Avaya Communications and Media Server or Cisco CallManger will be tallied under your license when the network is polled. While phones can be registered or unregistered, only registered phones count toward your license. Registered phones, unlike unregistered phones, are connected and therefore prioritized. In cases where registered IP phones exceed active VNQM licenses until you hit your licensed limit, the most recent registered phones will be monitored. All registered and unregistered phones are unmonitored after you’ve consumed your licensed IP phone levels.

Nodes: Every SNMP-enabled device you monitor fills up space on your licensed node capacity. Whether it’s an active Network Service Assurance device or a monitored call manager, in either case, it’ll count as a node license. Except for bidirectional setups, each source node consumes one node level. If a node is both the source and target, both would be counted under your license.