Network Conversations Monitor (10062 / 20062)
Description
This Module reports consolidated network conversations.
Consolidation
This Module consolidates *flow records over a period of time - Data Collection Interval (DCI) which all have the same combination of the following fields:
- Exporter IP (*flow exporter for physical devices, VPC for AWS, Virtual Network for Azure, VCN for Oracle OCI, Subnetwork for Google cloud)
- Source IP address
- Destination IP address
- Source port number
- Destination port number
- Layer 3 protocol
- Reporter (for Google Flow logs)
Flow Stitching - Report Bidirectional Conversations
This Module optionally stitches client-server request-response flows, reporting bytes and packets server-to-client and client-to-server in separate fields: bytes_in/bytes_out and packetsin/packetsout.
Deduplication
Optionally the Module can report host pairs only from authoritative router/switch. Authoritative network device is determined as follows. The Module sums up bytes, packets, and connections between two hosts over data collection interval (parameter, default - 30 sec), reported by each flow exporter. An exporter with most connections (flows) for each host pair is considered authoritative, and host pair conversations reported by all other exporters are discarded.
State of Conversation
The state field indicates the state of the conversation and may have the following values: “B”: Begin, “C”: Continuing, “E”: End. The state is reported for each conversation at the end of Data Collection Interval (DCI).
The logic for the state field is:
-
The state is set to “B” when flows for a new host pair is observed fir the first time (or after we see them again after previous conversation has ended) during DCI.
-
The state is set to “C” if the conversation was reported in previous DCI and flows were observed in the current DCI.
-
The state is set to “E” when no flows were observed for the length of “Session report timeout” parameter.
note
Please note that the state of conversation is reliably reported when Top N is set to report all sessions, and not only top traffic.
Conversation Duration
Network Conversation duration is calculated as a difference between the earliest flow_start_time of a conversation when its state=B and the latest flow_end_time for state = C or E.
Direction
The direction field indicates in which direction (inbound or outbound) the network conversation is going. It is determined based on configuration of local subnets in the Module (List of local subnets for IPv4 and List of local IPv6 prefixes for IPv6).
The values are:
inbound for external src_ip and local dest_ip
outbound for local src_ip and external dest_ip
internal for local src_ip and local dest_ip
unknown for none of the above
Flow Enrichment
The following fields, not in the original flow records, are added to the Module output:
- FQDN name: Reverse-DNS lookup name of the src or dest, (src_host or dest_host)
- VM name: EC2 name for AWS, VM name for VMware, GCP, and Azure
- Security Reputation
- User identity
- Applications
Security Reputation
NFO matches source IP and destination IP against threat lists, and adds reputation information in the following fields:
- List name (threat_list_name)
- Reputation (reputation)
Where
- List name is the name of the list specified in configuration
- Reputation is provided by a threat list vendor or, for lists with malicious domain names (currently supported from AlienVault OTX), set by NFO using malicious domain name provided by the list vendor
note
Flows with reputation are always reported, even if they don't make it to Top N by traffic volume. Therefore, any communications with bad actors are never missed.
For more information, see Configuring Custom Threat List section in EDFN Administration Guide.
User Identity Enrichment
User identity is provided by EDFN agent User identity monitor and could be configured from this Module GUI. It builds two lookup lists:
- IPv4, idp, username
- IPv6, idp, username
Where
- idp is identity provider that reported IP address and username
- IPv4 is IP address from which the user logged on
- username is the name of the user or account name reported by identity provider
In this release the agent supports two methods:
- Integration with Active Directory Domain Controller
- Integration with Microsoft Azure Domain Controller
- Integration with Okta
- Integration with any identity management system reporting login/logout events via syslog
For more information, see Configuring User Identity Enrichment section in EDFN Administration Guide.
Module Parameters
To configure Module parameters click on 10062: Network Conversation Monitor.
Logic Parameters
| Parameter Name | Description | Comments |
|---|---|---|
| N – The number of unique sessions to report | The number of unique stitched sessions reported per NetFlow exporter (Top N). NOTE: Conversations with malicious hosts are reported regardless whether they are in Top N by traffic volume or not! | min = 0, max = 100000, default - 0 (0 indicates all sessions are reported) |
| Session report timeout, sec | This parameter is used to report the duration of a session, along with its state. Specify the number of seconds during which no flows are observed to determine when a network conversation is considered ended. Event is reported with state=”E” this many seconds after last flow_end_time | min = 0 sec, max = 600 sec, default - 60 sec. This parameter should be set to a value greater than network device flow-cache inactive timeout. NOTE: 0 is a special value to indicate that niether duration nor state fields are required, which results in better performance. |
| Report inactive sessions | If set to 1, report inactive session with 0 bytes/packets, even if there were no flows during DCI. If set to 0, inactive sessions are not reported | default - 0 |
| Report long flows with cumulative bytes and packets | If set to 1, cumulative bytes and packets are reported for long sessions. If set to 0, incremental bytes and packets are reported for long flows (state = “C”) | default - 0 |
| Enable (1) or disable (0) deduplication | If set to 1 (de-duplication enabled), the Module reports flows only from authoritative exporters | default - 0 |
| Enable (1) or disable (0) multiplying by sampling rate | If set to 1, when *flow is sampled (e.g. sFlow, sampled NetFlow/IPFIX), the sampling rate is used to multiply bytes and packets to report total traffic as statistical approximation | default - 0 |
| Default sampler rate | If sampling information is not available, use this rate to multiply bytes and packets to report total traffic as statistical approximation | default - 1 |
| Enable (1) or disable (0) reporting bidirectional conversations | If set to 1, stitch client-server flows reporting bytes_in and bytes_out, packets_in and packets_out in on consolidated message | default - 0 |
| Enable (1) or disable (0) reporting client port | If set to 1, the ephemeral client port number is reported. If set to 0, client port number is not taken into account for consolidation, and reported as 0 | default - 1 |
| Enable (1) or disable (0) reporting flow denied events | If set to 1, denied or rejected flows are reported. If set to 0, only allowed or accepted flows are reported | default - 1 |
| Enable (1) or disable (0) collecting application info from devices reporting it | If set to 1, application collector is enabled. If set to 0, applications are reported only in flows containing app fields | default - 1 |
| Enable (1) or disable (0) enriching other devices with application info | If set to 1, application collector is used to enrich all flows, even if they did not have application fields | default - 1 |
| Output filename for application info | Path to application collector file. Used for troubleshooting purposes | default - ../../logs/app_info.log |
| Enable (1) or disable (0) generating end of conversation events | If set to 1, events at the end of conversations (state=E) are created and reporteed. If set to 0, events with stary=E are not reported | default - 0 |
| Enable (1) or disable (0) SNMP enrichment | If set to 1, the Module will call SNMP service to enrich flow data with sysName and interface names. If set to 0, SNMP service is not called | default - 1 |
Data Consolidation Parameter
| Parameter Name | Description | Comments |
|---|---|---|
| Data Collection Interval, sec | Module logic execution interval | min = 5 sec, max = 86400 sec, default - 30 sec |
Data Sets and Enrichment Parameters
| Parameter Name | Description | Comments |
|---|---|---|
| List of known server destination port numbers | List of server destination ports to be used to determine which host is a client and which is a server. If the list is empty, the server is the one with a smaller port number. This parameter is ignored for unidirectional flows. | This parameter is pre-loaded with values from: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml |
| List of local subnets | Used to identify direction for IPv4 traffic: inbound or outbound or internal | default - { 10.0.0.0,8; 172.16.0.0,12; and 192.168.0.0,16 } |
| List of local IPv6 prefixes | Used to identify direction for IPv6 traffic: inbound or outbound or internal | default - fc00:0:0:0:0:0:0:0,7 |
| AWS EC2 instances list | List of EC2 instances with IPs and VPC names and other information | Provided by EDFN agent. Please see Amazon VPC Flow Logs for details |
| AWS VPC IPv4 Routes | List of AWS VPC IPv4 routes | Provided by EDFN agent. Please see Amazon VPC Flow Logs for details |
| AWS VPC IPv6 Routes | List of AWS VPC IPv6 routes | Provided by EDFN agent. Please see Amazon VPC Flow Logs for details |
| IPv4 address ranges for cloud services | List of IPv4 address ranges for cloud services | Provided by EDFN agent |
| IPv6 address ranges for cloud services | List of IPv6 address ranges for cloud services | Provided by EDFN agent |
| GCP VM instances list | List of Google cloud VM names | Provided by EDFN agent |
| GCP IPv4 routes list | List of Google cloud routes | Provided by EDFN agent |
| Azure VM Instances | List of Azure VM names | Provided by EDFN agent |
| Azure IPv4 Routes | List of Azure IPv4 routes | Provided by EDFN agent |
| Azure IPv6 Routes | List of Azure IPv6 routes | Provided by EDFN agent |
| List of Users by IPv4 address | List of Users logged in from IPv4 addresses | Provided by EDFN agent. Please see User Identity Agent configurations for details |
| List of Users by IPv6 address | List of Users logged in from IPv6 addresses | Provided by EDFN agent. Please see User Identity Agent configurations for details |
| Applications override list | List of App IDs and names to override applications reported by NetFlow/IPFIX | Created manually |
| List of Application names to be ignored | List of App names provided by DPI engines not to be collected / reported | Created manually. default - { incomplete; not-applicable; unknown-udp; unknown-tcp; unknown-p2p } |
| Custom Applications list | List of Custom Applications: IPv4 Address, Port, Protocol, Application Name, etc. | Created manually |
| Custom Threat list | List of public and private IP addresses with reputation known to be malicious hosts | Provided by EDFN agent. Please see Configuring Custom Threat List for details |
| IPv4 address block and country code | Mapping of country codes to IP addresses blocks | Provided by EDFN agent, which uses the MaxMind Country database as a source |
| IPv4 address block and city location | Mapping of city and country codes to IP addresses blocks | Provided by EDFN agent, which uses the MaxMind City database as a source |
| List of vCenter Virtual Machines | List of VMs, including: VDS IPv4 address, VM IPv4 address, VM IPv6 address, VDS Port ID, vNIC key, Port Group name, VM name, VM MoRef, VM instance UUID, vCenter UUID | Provided by EDFN agent by connecting to one or several vCenters |
| List of BD subnets to Tenant mapping | List of Cisco ACI Bridge domains and Tenants: IPv4 Address, IPv4 Address Mask, BD name, Tenant name | Provided by EDFN agent. Please see Configuring Cisco ACI Bridge Domain Subnets for details |
| AS Numbers IPv4 Blocks | Mapping of Autonomous System Number to IP addresses blocks | Provided by EDFN agent, which uses the MaxMind ASN database as a source |
| AS Numbers IPv6 Blocks | Mapping of Autonomous System Number to IP addresses blocks | Provided by EDFN agent, which uses the MaxMind ASN database as a source |
Converter Configuration
To configure Converter click on Converter for Network Conversation Monitor.
Output Fields
To configure output fields, including the order, click on List of output fields.
Input
NetFlow v5, v9, IPFIX, sFlow, Citrix AppFlow, AWS VPC Flow Logs, Microsoft Azure NSG Flow logs, Oracle Cloud Flow Logs.
Required NetFlow Fields
| Information Element (IE) | IE id | IE size (Bytes) | Description |
|---|---|---|---|
| sourceIPv4Address or sourceIPv6Address | 8 or 27 | 4 or 16 | The IPv4 or IPv6 source address in the IP packet header |
| destinationIPv4Address or destinationIPv6Address | 12 or 28 | 4 or 16 | The IPv4 or IPv6 destination address in the IP packet header |
| protocolIdentifier | 4 | 1 | The value of the protocol number in the IP packet header. The protocol number identifies the IP packet payload type. Protocol numbers are defined in the IANA Protocol Numbers registry |
| sourceTransportPort | 7 | 2 | The source port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the source port number given in the respective header |
| destinationTransportPort | 11 | 2 | The destination port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the destination port number given in the respective header |
Syslog/JSON Message Fields
| Key | Field Description | Comment |
|---|---|---|
| nfc_id | Message type identifier | string, nfc_id=20062 |
| flow_type | Type of Flow | string, e.g. NFv5, NFv9, sFlow, IPFIX, AWS, Azure, OCI, ... |
| exp_ip | NetFlow exporter IPv4 address | IPv4_address (for public clouds added for compatibility with other flows) |
| exp_ip6 | NetFlow exporter IPv6 address | IPv6_address (for public clouds added for compatibility with other flows) |
| input_snmp | Input SNMP index | number |
| output_snmp | Output SNMP index | number |
| exp_name | Exporter name | string |
| input_if_name | Input interface name | string |
| input_if_alias | Input interface alias | string |
| output_if_name | Output interface name | string |
| output_if_alias | Output interface alias | string |
| protocol | Transport Protocol | number, e.g. TCP = 6, UDP = 17 |
| src_ip | Source IPv4 address | IPv4 address |
| src_ip6 | Source IPv6 address | IPv6 address |
| src_port | Source transport port | number |
| src_tos | Source type of service | number |
| src_asn | Source AS | number |
| src_cc | Source Country Code | string |
| src_region | Source region | string |
| src_city | Source city | string |
| src_lon | Source longitude | number |
| src_lat | Source latitude | number |
| src_mac | Source MAC address | string, e.g. e0:46:9a:2b:83:13 |
| src_cloud_region | Cloud source region | string |
| src_cloud_service | Cloud source service | string |
| src_host | Source host name | string, included when FQDN is on |
| src_vlan | Source VLAN | string |
| src_vm_name | Source VM name or AWS EC2 instance or OCI Compute Instance name | string |
| src_vpc_name | Source VPC name | string |
| src_subnet_name | Source subnet name | string |
| src_tenant_name | Cisco ACI source tenant | string |
| src_bd_name | Cisco ACI source bridge domain | string |
| dest_ip | Destination IPv4 address | IPv4 address |
| dest_ip6 | Destination IPv6 address | IPv6 address |
| dest_port | Destination transport port | number |
| dest_tos | Destination type of service | number |
| dest_asn | Destination AS | number |
| dest_cc | Destination Country Code | string |
| dest_region | Destination region | string |
| dest_city | Destination city | string |
| dest_lon | Destination longitude | number |
| dest_lat | Destination latitude | number |
| dest_vm_name | Destination VM name or AWS EC2 instance or OCI Compute Instance name | string |
| dest_vpc_name | Destination VPC name | string |
| dest_subnet_name | Destination subnet name | string |
| dest_tenant_name | Cisco ACI destination tenant | string |
| dest_bd_name | Cisco ACI destination bridge domain | string |
| dest_cloud_region | Cloud destination region | string |
| dest_cloud_service | Cloud destination service | string |
| dest_mac | Destination MAC address | string, e.g. e0:46:9a:2b:83:13 |
| dest_vlan | Destination VLAN | string |
| dest_host | Destination host name | string, included when FQDN is on |
| tcp_flag | TCP flags | string, e.g. SYN,ACK,FIN |
| packets_in | Packets in the flow received by destination IP from source IP | number |
| bytes_in | Total number of Layer 3 bytes in the packets of the flow received by destination IP from source IP | number |
| packets_out | Packets in the flow sent by destination IP to source IP | number |
| bytes_out | Total number of Layer 3 bytes in the packets of the flow sent by destination IP to source IP | number |
| flow_count | Number of consolidated flows reported in this event | number |
| action [^1] | Flow action | string, The action is determined from IPFIX element 233 - firewallEvent and NFv9 / IPFIX element 89 - forwardingStatus |
| state | Flow state | string, B = Begin, C = Continuing, E = End |
| latency | As reported in flow records in msec | number |
| duration | Session duration - unidirectional / Conversation duration - bidirectional. Reported in sec | number |
| direction | Direction of the flow, if reported, or direction determined based on local subnets | string, inbound (local IP address is dest), outbound (local IP address is src ), internal (both, src and dest IP addresses are local), unknown (both src and dest IP addresses are not local) |
| netscaler_client_retrans_count | NetScaler client TCP retransmission count | integer |
| netscaler_server_retrans_count | NetScaler server TCP retransmission count | integer |
| netscaler_client_rtt | NetScaler client Rount-Trip time, msec | integer |
| netscaler_server_rtt | NetScaler server Rount-Trip time, msec | integer |
| idp | IDP for the user | string |
| username | User name provided by EDFN Agent ( UserName Type 371 - upcoming) | string |
| app_id | Application ID (Type 95) | string, Class Eng. ID:Selector ID (see Section 4 https://www.rfc-editor.org/rfc/rfc6759.html) |
| app_name | Application Name (Type 96) or proprietary IPFIX or NFv9 elements (Palo Alto Networks, NetScaler) | string |
| app_desc | Application Description (Type 94) or proprietary IPFIX or NFv9 elements (Palo Alto Networks, NetScaler) | string |
| app_engine_id | Application (Classification) Engine ID | string, Class Eng. ID description for part 1 of Type 95 (Type 101 - upcoming) |
| threat_list_name | The name of a cybersecurity threat list | string |
| reputation | Reputation from the treat list | string |
| aws_vpc_id | AWS VPC identifier | string |
| aws_vpc_name | AWS VPC name | string |
| aws_interface_id | AWS Interface Id | string |
| aws_account_id | AWS Account Id | string |
| gcp_reporter | GCP VPC Flow logs Reporter | string, SRC or DEST |
| gcp_exp | GCP VPC Flow logs Exporter. Calculated field based on reporter = SRC or DEST | string, Project ID/VPC/Subnet |
| gcp_subnet_id | GCP Subnet ID | string |
| src_vm_ip_pub | Source EC2 or OCI Instance public IPv4 address | IPv4 address |
| aws_src_inst_id | Source EC2 instance id | string, e.g. i-390d7032 or i-0c0a6ac75d9d87b7e |
| gcp_src_project_id | GCP Source Project ID | string |
| gcp_src_vm_zone | GCP Source VM Zone | string |
| azure_src_subs_id | Azure Source Subscription ID | string |
| azure_src_subs_name | Azure Source Subscription Name | string |
| azure_src_nsg_name | Azure Source NSG Name | string |
| azure_src_vnet_name | Azure Source Virtual Network Name | string |
| azure_src_res_grp_name | Azure Source Resource Group Name | string |
| dest_vm_ip_pub | Destination EC2 or OCI Instance public IPv4 address | IPv4 address |
| aws_dest_inst_id | Destination EC2 instance id | string |
| gcp_dest_project_id | GCP Destination Project ID | string |
| gcp_dest_vm_zone | GCP Destination VM Zone | string |
| azure_dest_subs_id | Azure Destination Subscription ID | string |
| azure_dest_subs_name | Azure Destination Subscription Name | string |
| azure_dest_nsg_name | Azure Destination NSG Name | string |
| azure_dest_vnet_name | Azure Destination Virtual Network Name | string |
| azure_dest_res_grp_name | Azure Destination Resource Group Name | string |
| oci_src_vcn_name | Source VCN name | string |
| oci_src_vcn_name | Source VCN name | string |
| oci_src_compartment_name | Source Compartment name | string |
| oci_src_tenancy_name | Source Tenancy name | string |
| oci_dest_vcn_name | Destination VCN name | string |
| oci_dest_compartment_name | Destination Compartment name | string |
| oci_dest_tenancy_name | Destination Tenancy name | string |
| flow_start_time | Start time of the first consolidated flow | time |
| flow_end_time | End of the last consolidated flow | time |
| t_int | Observation time interval, msec | number |
[^1] Action is reported as follows:
action=Rfor firewallEvent 0 (ignored), 2 (deleted), and 3 (denied), and Rejected cloud flow logsaction=Afor firewallEvent 1 (created), 4 (alert), and 5 (update), and Allowed cloud flow logsaction=Ufor forwardingStatus 00 (unknown)action=Ffor forwardingStatus 01 (forwarded)action=Dfor forwardingStatus 10 (dropped)action=Cfor forwardingStatus 11 (consumed)