Performance Tuning and Scaling

To handle high-volume telemetry (e.g., >100k flows/sec), NFO requires specific optimizations at the Operating System and JVM levels. Without these adjustments, the OS may drop UDP packets before NFO can process them.

OS-Level Tuning (UDP Buffers)

By default, most Linux distributions have small UDP receive buffers. Under high load, the OS may drop packets before NFO can process them.

You can check if there are any packets lost on socket buffers in Linux kernel by execute the following command:

netstat -suna

Under Udp: section you should see:

   xxxxxxxxxx packets received
   xxxxxxxxxx packets to unknown port received.
   xxxxxxxxxx packet receive errors
   xxxxxxxxxx packets sent
   0 receive buffer errors
   0 send buffer errors

If you see high counters in receive or send buffer errors, please see sections below.

Recommended Buffer Settings

Default Linux kernel buffers are too small for high packet rates and lead to dropped packets. The settings below apply to any NFO node handling a high flow rate, whether it is a standalone instance, NFO Central, or a peer.

1. Raise the Kernel Buffer Limits

Create /etc/sysctl.d/nfo-custom.conf with the following:

net.core.rmem_max = 56623104
net.core.wmem_max = 12582912
net.core.wmem_default = 12582912

Apply it:

sysctl --system

The send buffer settings matter because NFO Central forwards received flows to its pool peers, and because any node producing a high volume of syslog or JSON output can overrun the send buffer.

net.core.rmem_default does not need changing, because NFO sets its own buffer size explicitly.

2. Raise the NFO Receive Buffer

Add the following line to <nfo_home>/server/etc/server.cfg:

IT_RCVBUF 56623104

56623104 is the maximum NFO accepts. Without this line NFO uses 12582912, which still helps once net.core.rmem_max is raised, but does not use the full kernel allowance.

Parameter	Valid range	Default
IT_RCVBUF	124928 to 56623104	12582912

note

NFO uses the smaller of the NFO receive buffer and the kernel limit, so both must be raised. The NFO Virtual Appliance ships with a socket read buffer of 12582912, matching the NFO default.

3. Restart the NFO Server

The buffer size is applied when the input socket is opened, so the new value takes effect only after a restart.

4. Verify

Run ss -uamn and find the NetFlow input port. The rb value under skmem should be approximately 113246208.

Linux doubles the requested buffer size to allow for its own bookkeeping, so rb reports twice what NFO asked for. The test is that rb is roughly double the IT_RCVBUF value, not that it matches an exact figure.

An rb of 425984 means the kernel limit did not apply

The kernel caps the requested size at net.core.rmem_max before doubling it. On a system left at the common default of 212992, that cap produces an rb of 425984 no matter how high IT_RCVBUF is set.

So if rb is still 425984 after step 2, the problem is step 1 rather than the NFO setting. Check that /etc/sysctl.d/nfo-custom.conf was loaded, and confirm with sysctl net.core.rmem_max.

Run netstat -suna again. The receive buffer errors and send buffer errors counters should stop growing.

If Drops Continue

Capture the following under load and send them to NetFlow Logic Support:

• netstat -suna
• cat /proc/net/softnet_stat
• top
• The NFO server and controller logs covering the period when drops occurred

Output Throttling

NFO can produce a high volume of syslog or JSON messages in bursts, particularly at the end of a data collection interval, when Top N is set to 0, or when several outputs are configured.

Where raising the send buffer is not enough, output throttling caps the rate at which NFO sends. To enable it at N messages per second, add the following lines to <nfo_home>/server/etc/server.cfg:

THROTTLE_OUTPUT 1
THROTTLE_OUTPUT_RATE <N>

JVM Memory Management

NFO runs on Java. For high-volume processing, the memory allocated to the NFO service (Heap Size) must be sufficient to handle the data in-flight.

• Configuration: Adjust the -Xmx parameter in the NFO configuration files.
• Guideline: High-volume instances need at least 8 GB of RAM dedicated to the JVM, depending on the number of active enrichments.

CPU Allocation

NFO is highly multi-threaded and benefits from additional cores as flow volume grows. It runs well on shared virtualized infrastructure, and most deployments do not require dedicated vCPUs.

At high flow rates, CPU contention on a heavily oversubscribed host can delay processing enough to cause drops. If drops persist after the buffer tuning above, check CPU ready time on VMware or steal time on KVM for the NFO guest. Sustained high values indicate the guest is waiting for physical CPU, and a CPU reservation for that VM will resolve it.