Checking Packet Loss Due to Low Socket Buffers

In high-volume deployments you may experience UDP packets drops in the operating system socket buffers. They need to be large enough to handle the incoming and outgoing network traffic.

You can check if there are any packets lost on socket buffers in Linux kernel by execute the following command:

netstat -suna

Under Udp: section you should see:

Udp: xxxxxxxxxx packets received xxxxxxxxxx packets to unknown port received. xxxxxxxxxx packet receive errors xxxxxxxxxx packets sent 0 receive buffer errors 0 send buffer errors

If you see high counters in receive or send buffer errors, please see sections below.

Input Buffer Settings (Linux)

The default Linux kernel settings are not sufficient for high-volume packet rate. This can lead to dropped packets and data loss. We recommend that you change both the receive buffer size in NFO and the socket read buffer size in Linux kernel.

To set the size of the receive buffer to <N> bytes in NFO, add the following string to <nfo_home>/server/etc/server.cfg:

IT_RCVBUF <N>

The valid values for parameter N are 124928 through 56623104. The default value is 12582912.

To set the size of the socket read buffer in Linux kernel to <N> bytes for current session, execute (under root privileges) the following command in a console:

sysctl -w net.core.rmem_max N

To make this change persistent, add the following string to /etc/sysctl.conf:

net.core.rmem_max=<N>

Then run the following command to reload the settings from the file:

sysctl -p

To check what the socket read buffer size is currently used, execute the following command:

sysctl net.core.rmem_max

Notes:

1. NFO effectively uses the least size of those buffers.

2. NFO Virtual Appliance has the socket read buffer size 12582912 -- the default value for NFO receive buffer.

Output Buffer Settings (Linux)

NFO may produce a lot of syslog/JSON messages, sometimes in bursts (at the end of data collection interval). This is especially true when Top N is set to 0 (meaning to send all messages), or when several NFO Outputs are configured. This could lead to packet drops on the way out with default Linux kernel settings. We recommend that you change default and maximum socket send buffer size in Linux kernel.

To set the size of the default socket send buffer in Linux kernel to <N> bytes for the current session, execute (under root privileges) the following command in the console:

sysctl -w net.core.wmem_default <N>

To set the size of the maximum socket send buffer in Linux kernel to <N> bytes for the current session, execute (under root privileges) the following command in the console:

sysctl -w net.core.wmem_max <N>

To make this change persistent, add the following string to /etc/sysctl.conf:

net.core.wmem_max=<N> net.core.wmem_default=<N>

Then run the following command to reload the settings from the file:

sysctl -p

To check what the socket send buffer size is currently used, execute the following command:

sysctl net.core.wmem_default

Another way to reduce output packets loss on the socket buffers is to configure output throttling in NFO. To enable output throttling with rate N per second, add the following lines to<nfo_home>/server/etc/server.cfg:

THROTTLE_OUTPUT 1 THROTTLE_OUTPUT_RATE <N>