What Are Modules and Converters?
NetFlow data is notoriously voluminous. Traditionally, all NetFlow records generated by network devices are captured and stored for further interpretation. This exact process of capturing all NetFlow records, without understanding the significance of information contained in the records’ data, creates tremendous storage and data analysis problems. A mid-range 20 Gb device in a large office can process tens of thousands of network exchanges per second, which results in a hundred thousand NetFlow records per second. Assuming that each NetFlow record is 100 bytes long, storing data at this rate it would take 8.6TB of disk space every day. Even a smaller switch, router or firewall that processes 10 times less network connections produces 860GB of flow data every day.
By enabling appropriate Modules, you turn on specific functionality within NetFlow Optimizer. For example, you can monitor:
- your network conversations and hosts’ behavior
- your network devices
- malicious host (e.g. botnets, scanning hosts) communications to your data center
And many other use cases are expressed via the Modules.
NetFlow Optimizer Modules and Converters are designed to provide solutions for specific use cases and at the same time reduce the amount of data (without losing information veracity) that needs to be stored by orders of magnitude. The Modules and Converters are packaged into Module Set packages. Each Module consists of one or more content-based rules and one or more time-based triggers (called “Data Collection Interval”). Converters provide mechanisms for translating information emitted by the Modules into a format suitable for further processing. Please see Solutions at a Glance section below for more details.
Let us consider a typical example: a network administrator would like to know how the bandwidth of his Cisco ASA firewall is consumed. This is not possible using traditional Cisco ASA logging because setting logging at the Informational level severely impacts device performance. A better approach is to use Cisco ASA NetFlow Secure Event Logging (NSEL) but the sheer volume of NSEL data may overwhelm traditional NetFlow collectors. This is when NetFlow Optimizer’s Modules mechanism comes to the rescue.
The diagram below shows how data reduction is implemented in the Top Bandwidth Consumers Module. This Module employs an in-memory Map-Shuffle-Reduce algorithm. To report top 50 bandwidth consumers by each network device, the Module sums up bytes by source IP/Port, Destination IP/Port, and Protocol -- processing every single flow record over a short period of time (e.g. 30 seconds) (Map), then the data is sorted by accumulated bytes (Shuffle), and finally the top 50 records are retrieved (Reduce), converted to syslog, and sent to a SIEM system (e.g. Splunk Enterprise). Thus this Module processes thousands of flow records per second, and reports only top 50 bandwidth consumers every 30 seconds, which are typically responsible for 98%-99% of all traffic bandwidth consumption.
NFO Modules’ use cases are not limited to NetFlow Consolidation. Another diagram below shows how security NFO Module reports all malicious network conversation based on threat lists.
