Skip to main content
Version: 2.10.1

Troubleshooting Drops with NFO Logs

If you see drop counts at Status page, you can find more information in NFO logs. The following table contains description and general explanation for packet drops.

Drop typeDescription
dropped by input threadsTotal number of packets received by NFO and dropped because they did not pass basic validation tests, e.g. packets are not one of known flow format – NetFlow v5/v9, sFlow, IPFIX, etc. Packets can also be dropped by input threads if NFO is unable to queue them for subsequent processing, e.g. out of memory, queue overflow, etc.
dropped by work threadsThe number of packets dropped when flow records is processed by Modules, e.g. there is no NFv9 or IPFIX Template for the flow record, or when there are other problems when processing flows, or when processed flows could not be placed in the Output queue. A small number of these drops is expected when NFO is restarted, while Templates are not yet received.
dropped by kron threadThe number of packets dropped by “Data collection interval” triggers caused by queue overflow.
dropped at outputThe number of packets dropped by NFO due to Output queue overflow.
dropped by QoSThe number of packets dropped by NFO internal Quality of Service mechanism to avoid congestions. These drops are also included in one of the drops statistics above.

Dropped by input threads

ProblemWhat to look for / Recommendation
Unknown flow format

Debug level:

"Input thread %d: NFv5 packet %u failed sanity check: %d\n"

"Input thread %d: NFv9 packet %u failed sanity check: %d\n"

"Input thread %d: IPFIX packet %u failed sanity check: %d\n"

"Input thread %d: sFlow packet failed sanity check: %d\n"

"Input thread %d: src addr: %s P2 FDR id: %u size: %u\n"

Flood level:

"Input thread %d: UNRECOGNIZED: src addr: %s size: %u\n"

Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2

NFO is unable to read packet from NFO input port

Debug level:

"Input thread %d: receiver failure %d\n"

Recommended action: Check health of the sockets opened by NFO for input ports

Insufficient memory when NFO reads packet from NFO input port

Error level:

"Input thread %d: failed to allocate buffer for derived objects\n"

Recommended action: Check free RAM availability on NFO host

Problem related to templates for NFv9 or IPFIX data records

Debug level:

"TFS CHECK: pkt longer than size in hdr: %u expected %u\n"

"TFS CHECK: pkt shorter than size in hdr: %u expected %u\n"

Recommended action: Check that all incoming NFv9/IPFIX packets meet specification

Problems when placing packets from input thread to work thread after validation. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"Input thread %d: worker thread %d QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Verbose level:

"Queue is full.\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped by work threads

ProblemWhat to look for
Problems when placing packets from work thread to output thread

Error level:

"Work thread %d: output thread %d QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Debug level:

"WT %d: failed to enqueue to output thread: %d\n"

"WT %d: failed to enqueue to output thread: %d\n"

Recommended action: Increase output threads count or configure output using faster network interface if available

Received discard status for a packet while processing it

Verbose level:

"Work thread %d: discarded message by status\n"

Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2. Check if all incoming data has corresponding templates (NFv9/IPFIX). Check if incoming templates meet minimal input fields requirements for enabled Modules.

Problems when placing packets from one work thread to another work thread. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"Worker thread %d: QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Debug level:

"WT %d: failed to enqueue derived message to work thread qos queue: %d\n" и

"WT %d: failed to enqueue derived message to work thread qos queue: %d\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Work thread overflow

Debug level:

"Work thread %d: failed to enqueue derived message: queue full\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped by kron threads

ProblemWhat to look for
Problems when placing packets from kron thread to work thread. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"KRON EVAL: %d: failed to enqueue message to work thread QoS queue: %d\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Work thread overflow

Verbose level:

"Queue is full.\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped at output

ProblemWhat to look for
Output thread overflow

Verbose level:

"Queue is full.\n"

Recommended action: Increase output threads count or configure output using faster network interface if available

Dropped by QoS

ProblemWhat to look for
Drops caused by NFO RED (https://en.wikipedia.org/wiki/Random_early_detection) implementationThese drops are included in one of the drops described before. There are no special logs for these drops.