Troubleshooting Drops with NFO Logs

If you see drop counts at Status page, you can find more information in NFO logs. The following table contains description and general explanation for packet drops.

Drop type	Description
dropped by input threads	Total number of packets received by NFO and dropped because they did not pass basic validation tests, e.g. packets are not one of known flow format – NetFlow v5/v9, sFlow, IPFIX, etc. Packets can also be dropped by input threads if NFO is unable to queue them for subsequent processing, e.g. out of memory, queue overflow, etc.
dropped by work threads	The number of packets dropped when flow records is processed by Modules, e.g. there is no NFv9 or IPFIX Template for the flow record, or when there are other problems when processing flows, or when processed flows could not be placed in the Output queue. A small number of these drops is expected when NFO is restarted, while Templates are not yet received.
dropped by cron thread	The number of packets dropped by “Data collection interval” triggers caused by queue overflow.
dropped at output	The number of packets dropped by NFO due to Output queue overflow.
dropped by QoS	The number of packets dropped by NFO internal Quality of Service mechanism to avoid congestions. These drops are also included in one of the drops statistics above.

Dropped by input threads

Problem	What to look for / Recommendation
Unknown flow format	Debug level: "Input thread %d: NFv5 packet %u failed sanity check: %d\n" "Input thread %d: NFv9 packet %u failed sanity check: %d\n" "Input thread %d: IPFIX packet %u failed sanity check: %d\n" "Input thread %d: sFlow packet failed sanity check: %d\n" "Input thread %d: src addr: %s P2 FDR id: %u size: %u\n" Flood level: "Input thread %d: UNRECOGNIZED: src addr: %s size: %u\n" Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2
NFO is unable to read packet from NFO input port	Debug level: "Input thread %d: receiver failure %d\n" Recommended action: Check health of the sockets opened by NFO for input ports
Insufficient memory when NFO reads packet from NFO input port	Error level: "Input thread %d: failed to allocate buffer for derived objects\n" Recommended action: Check free RAM availability on NFO host
Problem related to templates for NFv9 or IPFIX data records	Debug level: "TFS CHECK: pkt longer than size in hdr: %u expected %u\n" "TFS CHECK: pkt shorter than size in hdr: %u expected %u\n" Recommended action: Check that all incoming NFv9/IPFIX packets meet specification
Problems when placing packets from input thread to work thread after validation. You may also get +1 in dropped by QoS if RED queue is full	Error level: "Input thread %d: worker thread %d QoS queue failure %d\n" Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available Verbose level: "Queue is full.\n" Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped by work threads

Problem	What to look for
Problems when placing packets from work thread to output thread	Error level: "Work thread %d: output thread %d QoS queue failure %d\n" Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available Debug level: "WT %d: failed to enqueue to output thread: %d\n" "WT %d: failed to enqueue to output thread: %d\n" Recommended action: Increase output threads count or configure output using faster network interface if available
Received discard status for a packet while processing it	Verbose level: "Work thread %d: discarded message by status\n" Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2. Check if all incoming data has corresponding templates (NFv9/IPFIX). Check if incoming templates meet minimal input fields requirements for enabled Modules.
Problems when placing packets from one work thread to another work thread. You may also get +1 in dropped by QoS if RED queue is full	Error level: "Worker thread %d: QoS queue failure %d\n" Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available Debug level: "WT %d: failed to enqueue derived message to work thread qos queue: %d\n" и "WT %d: failed to enqueue derived message to work thread qos queue: %d\n" Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.
Work thread overflow	Debug level: "Work thread %d: failed to enqueue derived message: queue full\n" Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped by cron threads

Problem	What to look for
Problems when placing packets from cron thread to work thread. You may also get +1 in dropped by QoS if RED queue is full	Error level: "KRON EVAL: %d: failed to enqueue message to work thread QoS queue: %d\n" Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.
Work thread overflow	Verbose level: "Queue is full.\n" Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Dropped at output

Problem	What to look for
Output thread overflow	Verbose level: "Queue is full.\n" Recommended action: Increase output threads count or configure output using faster network interface if available

Dropped by QoS

Problem	What to look for
Drops caused by NFO RED (https://en.wikipedia.org/wiki/Random_early_detection) implementation	These drops are included in one of the drops described before. There are no special logs for these drops.

---