Sunday, October 04, 2015

Troubleshooting Packet drops in Solarflare 10G network card

If you experience packet drops while listening to UDP packet broadcast then the following extracts from the onload_stackdump command utility can help you
identify the Onload stack under memory pressure.

[root@paragpc ~]  onload_stackdump lots

You can monitor the rate of these drops for the Onload stack using the above command, for example in one of my machine the stackdump reported:

[root@paragpc ~] onload_stackdump lots | grep memory_pressure_drops
memory_pressure_drops: 81381

If this is increasing during bursts of data then increasing the value of EF_MAX_PACKETS to 65000 or higher could resolve the issue by providing more buffering but if this value is increasing steadily then it would only delay the point at which drops happen. You can set this by setting “EF_MAX_PACKETS=65000” in the environment or prefix the ‘onload’ command, e.g.:

[root@paragpc ~] EF_MAX_PACKETS=65000 onload <app_cmd_line>

Note that this is further restricted for TX and RX in order to avoid potential deadlocks where all the packet buffers are used on the RX and can’t be freed because data needs to be sent to do this. By default there is a 75% limit on each so if you use the maximum for RX it would leave 25% for TX.

Read more about the how to troubleshoot and configure onload here.

No comments:

Using Tcpdump to dump and read network traffic

Another Quick FYI tip. There are many network analyzer/reader utilities available on both Linux and Windows platform. There is of-course ...