NetFlow Optimizer™ and External Data Feeder Release Notes

What’s New in this Release

Build 2.7.1.1.21

Added Microsoft Azure NSG Flow Logs Support

Added ability to ingest and enrich Azure NSG Flow logs. This upgrade includes two NFO Modules; one to enrich Azure NSG Flow logs, and another to enrich and consolidate Azure NSG Flow logs with an option to report Top traffic

Customer Request/Ticket numbers: NFC-9504, NFC-9531, NFC-9657

Added NFO Modules for Google Cloud VPC Flow logs

  • Consolidate and optimize VPC Flow logs data thus enabling customers to store and index only a fraction of volume and at the same time gain all benefits of flow information without losing accuracy

  • Enrich basic VPC Flow logs with real-time information, such as VM name, etc

  • Improve performance

Customer Request/Ticket numbers: NFC-9445, NFC-9470, NFC-9473, NFC-9481, NFC-9490, NFC-9496, NFC-9502

Added Reporting of "Services" to AWS VPC Flow Logs

Now NFO detects if source or destination is one of AWS services (e.g. S3) and enriches flow records with this information.

Customer Request/Ticket numbers: NFC-9437

Added NFO Input Configuration for Public Clouds and Ability to Enable/Disable Inputs

  • Now NFO input ports could be enabled/disabled via GUI

  • Added configuration of AWS/GCP/Azure inputs in NFO GUI of Inputs panel

Customer Request/Ticket numbers: NFC-9547

Added New Options in Top Traffic Modules (Top Traffic, Top Packets, Top Connections, Top Host Pairs)

  • Now these Modules have an option to exclude ephemeral client source or destination port from reporting. The list of server side ports is configurable.

  • Enable / disable multiplying bytes, packets, and flow count by sampling rate. Any sampled NetFlow, IPFIX, or sFlow is supported.

Customer Request/Ticket numbers: NFC-5087, NFC-9137

Improve Visitors by Country Module

Added parameter to enable/disable reporting of denied flows.

Customer Request/Ticket numbers: NFC-9460

Added Cisco AVC Modules to NFO

Added Cisco Application Visibility and Control (AVC) (https://www.cisco.com/c/en/us/products/routers/avc-control.html) Modules to NFO packages.

Customer Request/Ticket numbers: NFC-9423

Improved SNMP polling capabilities

Allow users to add specific OIDs down to a table level.

Customer Request/Ticket numbers: NFC-9094

Added Cisco AnyConnect (aka NVM or nvzFlow) Support

Added support for Cisco IPFIX fields used in AnyConnect (Secure VPN access for remote workers).

Customer Request/Ticket numbers: NFC-9516

Upgraded Syslog Output Format to Comply with RFC 5424

NFO now uses RFC5424 (https://tools.ietf.org/html/rfc5424) when output sent as Syslog messages.

Customer Request/Ticket numbers: NFC-9492, NFC-9515

Allow Uploading Trusted Certificates via GUI

Add a user interface in Admin to upload trusted certs instead of dealing with CLI.

Customer Request/Ticket numbers: NFC-9202

NFO Usability Improvements

Various usability improvements and cosmetic enhancements.

Customer Request/Ticket numbers: NFC-9450, NFC-9506, NFC-9422

Upgraded NFO to Tomcat 9

Customer Request/Ticket numbers: NFC-9644

Upgraded NFO to Java 8 Update 261 (8u261)

Customer Request/Ticket numbers: NFC-9656

Build 2.7.0.0.264

Security update:

XML external entity (XXE) injection

This vulnerability allows an attacker to interfere with an application's processing of XML data. This vulnerability is closed in NFO 2.7.0.0.264. For prior releases please apply the following workaround:

  1. Edit /opt/flowintegrator/tomcat/webapps/ROOT/WEB-INF/web.xml file

  2. Find parameter: <init-param> <param-name>com.sun.jersey.config.feature.DisableXmlSecurity</param-name> <param-value>true</param-value> </init-param>

  3. Change <param-value> to false or remove the entire <init-param>section

  4. Restart tomcat service: service tomcat_nfo restart

Added Amazon Web Services (AWS) VPC Flow logs support

  • Ingest VPC Flow logs from AWS CloudWatch, Kinesis stream, or S3

  • Support processing VPC Flow logs from multiple AWS accounts, VPCs, and regions

  • Enrich native VPC Flow logs with real-time information, such as EC2 name, DNS name, and AWS region

  • Consolidate and optimize VPC Flow logs data thus enabling customers to store and index only a fraction of volume and at the same time gain all benefits of flow information without losing accuracy

Customer Request/Ticket numbers: NFC-9006, NFC-9251, NFC-9254, NFC-9260, NFC-9287, NFC-9300, NFC-9301, NFC-9302, NFC-9304, NFC-9305

Added Google Cloud VPC Flow logs support

Added ability to ingest, consolidate, and enrich GPC VPC Flow logs. Customer Request/Ticket numbers: NFC-9189

Added support for Cisco AVC

Cisco Application Visibility and Control (AVC) technology is now supported. AVC classifies more than 1400 applications, and reports them in IPFIX. AVC is available across routers, campus switches, access points, and wireless controllers. See https://www.cisco.com/c/en/us/products/routers/avc-control.html for details. Customer Request/Ticket numbers: NFC-8027

Improved SNMP polling capabilities

Added support for 'sparse augments'. Improved SNMP service performance. Added SNMP polling and traps statistics. Customer Request/Ticket numbers: NFC-8438, NFC-9081, NFC-9131, NFC-9133, NFC-9164

Support new MaxMind authentication

Change default URLs in all Modules with GeoIP enrichment to alow users to enter their own MaxMind subscription credentials. Customer Request/Ticket numbers: NFC-9293

Added user IP address and port to identify user when Palo Alto Networks device is not integrated with AD

When PAN device is not integrated with AD, all users reported as "na". To identify users for applications monitoring add user IP and port. Customer Request/Ticket numbers: NFC-9126

Security Modules: allow setting to include flow created and flow updated events

Add parameter to all Security Modules: "Enable reporting flow created and flow updated events". Default - disabled. Customer Request/Ticket numbers: NFC-9284

Added support for new sFlow extensions in Original Flow data

NFO sFlow support includes sFlow extended structures as of December 2019 (https://sflow.org/developers/structures.php). Customer Request/Ticket numbers: NFC-8885

Added ifAlias in SNMP polling Module (10003)

Added ifAlias (OID 1.3.6.1.2.1.31.1.1.1.18) to the output of SNMP polling Module (10003). Customer Request/Ticket numbers: NFC-9095

Added DNS names to Security Modules output

Added DNS names, if available, for source / destination IP addresses reported by Security Modules. Customer Request/Ticket numbers: NFC-9096

Implemented Heartbeat messages in Security Modules

Now Security Modules have an option to send a heartbeat message indicating that they are up and running. The message includes Module ID and the timestamp when corresponding threat lists were updated. Customer Request/Ticket numbers: NFC-9100

Package NFO and External Data Feeder (EDFN) together

As in most cases NFO and EDFN are installed together on the same machine, starting with this release EDFN is packaged together with NFO in one installer (rpm, tar.gz, and msi). Customer Request/Ticket numbers: NFC-9280

Build 2.6.0.1.1

Security update:

Remove SHA-1 ciphers

SHA-1 (Secure Hash Algorithm 1) has been known to be vulnerable to attacks. Digital certificate authorities have not been allowed to issue SHA-1-signed certificates since Jan. 1, 2016, although some exemptions have been made. Now SHA-1 ciphers are completely removed from NFO. Customer Request/Ticket numbers: NFC-8751

Implemented NetFlow Capture and Replay functionality

Now you can look back in time for security issues. NFO has an option to set a rolling period of time to capture flows, store these flows in memory or on disk, and replay them when a security event is detected in order to see the traffic that preceded the event. Customer Request/Ticket numbers: NFC-8839

Implemented Micro-segmentation Analytics Module

This Module is capable of processing NetFlow / IPFIX / sFlow from physical network devices as well as VMware Virtual Distributed Switch. It is used for analyzing “east-west” and “north-south” traffic and providing information for micro-segmentation planning. Customer Request/Ticket numbers: NFC-9038

If you had Micro-segmentation Module installed in previous NFO release, you need to reconfigure connection to vCenter after upgrading to NFO 2.6.

Implemented NSX Distributed Firewall (DFW) Monitoring

NSX Distributed Firewall is a hypervisor kernel-embedded firewall that provides visibility and control for virtualized workloads and networks. The new NFO modules for DFW report top bandwidth consumers, top destinations, top DFW policy violators, and top VMs with the most connections. Customer Request/Ticket numbers: NFC-8757

Implemented JSON output option

Now you have an option to choose whether NFO can be configured to produce output in Syslog or JSON format. NFO server.cfg file has two parameters:

REPLAY_OFD_OUTPUT JSON / SYSLOG – controls output format for Original Flow Data and Replay output.

MODULES_OUTPUT JSON / SYSLOG – controls output format for Original Flow Data and Replay output. Customer Request/Ticket numbers: NFC-8974, NFC-8999

Implemented NFO Modules ability to write output to disk

NFO Modules now can be requested with an option to write *flow data to disk (in addition to sending it out in syslog format) – available upon request. Customer Request/Ticket numbers: NFC-8579

Implemented support for BGP/BMP protocol to provide Autonomous System Paths

External Data Feeder for NFO has an Agent capable of providing Autonomous System Paths data retrieved in real time from edge devices that support BGP. It is used *flow data enrichment with AS Paths information. Customer Request/Ticket numbers: NFC-8561

Module: V2P Network Visibility – Enhancements

This Module correlates virtual overlay network and underlying physical network and virtual network operators to identify physical network devices impacting VM Applications performance. In this release we added the following: names for VDS interfaces, ifAlias field, VDS port group name, VM Host FQDN name. Added support for new IPV4 VDS templates. Removed LAN broadcast addresses from Path output (message 20183). Improve processing of *flows with SNMP indexes equal zero. Added ESXi physical adapter speeds to calculation utilization. Hide ifIPAddress field when value is 0.0.0.0. Customer Request/Ticket numbers: NFC-5744, NFC-6776, NFC-8700, NFC-8782, NFC-8783, NFC-8819, NFC-8820, NFC-8846, NFC-8847, NFC-8894.

Added support for Gentoo Linux

Gentoo Linux is now supported. Customer Request/Ticket numbers: NFC-8598

Added support for IPFIX field layer2OctetDeltaCount

Added support for IPFIX field layer2OctetDeltaCount as bytes Customer Request/Ticket numbers: NFC-8581

Added support for sFlow extensions in Original Flow data

NFO sFlow support includes sFlow extended structures as of February 2019 (https://sflow.org/developers/structures.php). Customer Request/Ticket numbers: NFC-8429

Enhance Microsegmentation Analytics for VMware vCenter Module

Implement integration with VMware NSX and vShield. Report VDS port groups. Customer Request/Ticket numbers: NFC-8755

Added ability for External Data Feeder for NFO to update multiple data sets

Now EDFN agent can handle several data sets. Update cron setting is still per agent. Customer Request/Ticket numbers: NFC-8930

Upgraded JDK8 to the latest build

Changed Oracle JDK 8u66 to Zulu OpenJDK 8u212. Customer Request/Ticket numbers: NFC-8968

Enhance NFv9/IPFIX Template persistence

Implemented Templates expiration. Default is 24 hours. Customer Request/Ticket numbers: NFC-7716, NFC-7717

Added FQDN name of DNS server to DNS Monitor Module (10004)

Added FQDN name field in Sysog/JSON output. Customer Request/Ticket numbers: NFC-8818

Expanded support of IPFIX variable length IEs

Added IPFIX variable length IEs support. Customer Request/Ticket numbers: NFC-7985

Performance: Implemented support for very large (several M recs) data sets

Improve performance of External Data Feeder and NFO. In this release we support unlimited size of in-memory data sets (tested with 7M records). In addition, data sets up to 3M records could be updated every 30 seconds. Customer Request/Ticket numbers: NFC-8614

Performance: Improve performance of streaming Modules

Streaming Modules performance (with *flow enrichment) was improved more than 3 times (300K records pes second in NFO 2.5.1 vs. 900K records per second in NFO 2.6 without a single drop). Customer Request/Ticket numbers: NFC-8560, NFC-8555

Performance: Improve performance of consolidation Modules

A single instance of NFO can now run up to 8 times more *flow consolidation Modules (NFO 2.5.1 vs NFO 2.6). Customer Request/Ticket numbers: NFC-8753

Build 2.5.1.0.43

Added support for SNMP Traps

NFO SNMP service supports SNMP Traps now. Customer Request/Ticket numbers: NFC-8334

Improved SNMP Polling performance

Implement GetBulk request for Table OIDs. Customer Request/Ticket numbers: NFC-8415

Added support for sFlow extensions in Original Flow data

NFO sFlow support includes sFlow extended structures as of February 2018 (https://sflow.org/developers/structures.php). Customer Request/Ticket numbers: NFC-8429

Added support for GeoIP enrichment using IP2Location databases

NFO Geo IP enrichment now has a choice: use MaxMind (GeoLite2-Country or GeoLite2-City) or IP2Location (DB1LITE for country level or DB5LITE for city level). Customer Request/Ticket numbers: NFC-8397

Added Cisco ASA support in V2P Module

Virtual to Physical (V2P) Network Visibility Module now is able to process Cisco ASA NSEL.

Customer Request/Ticket numbers: (NFC-8436)

Improved health score reporting in V2P Module

Report low traffic / low packet rate interfaces as having health score of 100. Customer Request/Ticket numbers: NFC-8456

Improved SYN-flood DDoS Attack detection

A number of enhancements were implemented in DDoS detection Module to improve reduction of false positives and increase the number of variations of DDoS attacks. Customer Request/Ticket numbers: NFC-8320

Improved SSDP Reflection DDoS Attack detection

A number of enhancements were implemented in DDoS detection Module to improve reduction of false positives and increase the number of variations of DDoS attacks. Customer Request/Ticket numbers: NFC-8381

Improved Visitors by Country Module

Use list of local subnets to determine internal IP addresses and exclude reporting local-to-local traffic. Customer Request/Ticket numbers: NFC-8264

Added support for IPFIX Private Enterprise Information Elements

NFO IPFIX Original Flow Data processing now has the ability to add and edit key names for any IPFIX field. Customer Request/Ticket numbers: NFC-8244

Improved usability of Module: SNMP Custom OID Sets Monitor

Customer Request/Ticket numbers: NFC-8092, NFC-8110, NFC-8179, NFC-8202, NFC-8203

Added support for Windows Server 2016

Customer Request/Ticket numbers: NFC-8162

Improve internal process.log format

NFO internal process.log format has been changed: remove units from numeric values. Customer Request/Ticket numbers: NFC-8414

Added client_ip to login success/fail messages in nf2sl.log

Customer Request/Ticket numbers: NFC-8507/ZEN-572

What’s Been Fixed in this Release

Build 2.7.1.1.21

Intermittent Error in FQDN Service

Affected Platforms: All Description: FQDN service intermittently raises errors when Google VPC Flow Logs Module 10301 is enabled.

Customer Request/Ticket numbers: NFC-9486

Bug in DNS Monitor Module does handling NetFlow v5

Affected Platforms: All Description: DNS Monitor Module does not produce output for NetFlow v5. NetFlow v9, IPFIX, and other *flow formats are working correctly.

Customer Request/Ticket numbers: NFC-9249

AWS Top Traffic Monitor intermitently reports 0 observation time interval

Affected Platforms: All Description: This Module intermitently reports 0 observation time interval.

Customer Request/Ticket numbers: NFC-9486

Various minor bug fixes

Build 2.7.0.0.264

VMware vCenter integration: unable to add 10Gibit pNIC

Affected Platforms: All Description: The following message is displayed:

Customer Request/Ticket numbers: NFC-9177

Build 2.6.0.1.1

Memory Leak after Known malicious hosts list has been updated

Affected Platforms: All Description: When known malicious hosts list is updated manually or via Updater, about 19MB of memory is not released.

Customer Request/Ticket numbers: NFC-7023

[Module 10103] Output produces separate syslog with non-table values when module is polling table data and scalar (non-table) data configured in the same OID set

Affected Platforms: All

Customer Request/Ticket numbers: NFC-8466

[Module 10103] Intermittent problem sending Module output

Affected Platforms: All

Customer Request/Ticket numbers: NFC-9120

Partial or complete lack of syslog output becuse of malformed KRON output

The Windows Filtering Platform prevents NFO Controller from a bind to a local port at some point on Windows Server 2016 platform

Affected Platforms: Windows 7/10, Windows Server 2012/2016 Description: When a block of a bind to a local port happens, NFO Controller warns on Status page that NFO Sever is unavailable and restarts it.

Customer Request/Ticket numbers: NFC-8505

Various bug fixes

Build 2.5.1.0.43

Fix syslog format to meet RFC-3164

NFO syslogs do not meet RFC-3164 requirements. Implement HOSTNAME field to follow TIMESTAMP field. Customer Request/Ticket numbers: NFC-3494

Remove unwanted error logs when External Data Feeder is restarted

When External Data Feeder is restarted, the following ERROR appears in nf2sl.log file: 2018-01-05 13:07:59,380 ERROR [JSR356Endpoint] NFO and External Data Feeder are working just fine. This error is removed to avoid unnecessary warnings. Customer Request/Ticket numbers: NFC-8362, ZEN-560

Top Traffic Monitor Module (10067) produces no output when Top N is set to 0 and deduplication is enabled

Customer Request/Ticket numbers: NFC-8325

Various bug fixes

Customer Request/Ticket numbers: NFC-3349, NFC-3604, NFC-4863, NFC-5332, NFC-5871, NFC-6058, NFC-6315, NFC-6540, NFC-7076, NFC-7836, NFC-7863, NFC-7879, NFC-7882, NFC-7886, NFC-7978, NFC-8014, NFC-8299, NFC-8357, NFC-8422, NFC-8439, NFC-8486

Known Issues

Build 2.5.1.0.43

Intermittent error: 'The server did not respond. Please check your connection'

This issue is related to Tomcat 9 with TLSv1.3. The root of this issue is OpenJDK bug:

Workaround: Disable TLSv1.3 and running with TLSv1.2 only.

Dashboard: statistics logging interval not displayed

Affected Platforms: All Description: Changing statistics logging interval, when changing the statistics-logging interval the statistics may not display based on the new value. Customer Request/Ticket numbers: NFC-2092 Workaround: Reset the statistics to the default of 10 seconds using the reset option.