Network Conversations Monitor (10062 / 20062)

Description

This Module reports consolidated network conversations.

Consolidation

This Module consolidates *flow records over a period of time - Data Collection Interval (DCI) which all have the same combination of the following fields:

• Exporter IP (*flow exporter for physical devices, VPC for AWS, Virtual Network for Azure, Subnetwork for Google cloud)
• Source IP address
• Destination IP address
• Source port number
• Destination port number
• Layer 3 protocol
• Reporter (for Google Flow logs)

Flow Stitching - Report Bidirectional Conversations

This Module optionally stitches client-server request-response flows, reporting bytes and packets server-to-client and client-to-server in separate fields: bytes_in/bytes_out and packetsin/packetsout.

Deduplication

Optionally the Module can report host pairs only from authoritative router/switch. Authoritative network device is determined as follows. The Module sums up bytes, packets, and connections between two hosts over data collection interval (parameter, default - 30 sec), reported by each flow exporter. An exporter with most connections (flows) for each host pair is considered authoritative, and host pair conversations reported by all other exporters are discarded.

State of Conversation

The state field indicates the state of the conversation and may have the following values: “B”: Begin, “C”: Continuing, “E”: End. The state is reported for each conversation at the end of Data Collection Interval (DCI).

The logic for the state field is:

• The state is set to “B” when flows for a new host pair is observed fir the first time (or after we see them again after previous conversation has ended) during DCI.

• The state is set to “C” if the conversation was reported in previous DCI and flows were observed in the current DCI.

• The state is set to “E” when no flows were observed for the length of “Session report timeout” parameter.

note

Please note that the state of conversation is reliably reported when Top N is set to report all sessions, and not only top traffic.

Conversation Duration

Network Conversation duration is calculated as a difference between the earliest flow_start_time of a conversation  when its state=B and the latest flow_end_time for state = C or E.

Direction

The direction field indicates in which direction (inbound or outbound) the network conversation is going. It is determined based on configuration of local subnets in the Module (List of local subnets for IPv4 and List of local IPv6 prefixes for IPv6).

The values are:

inbound for external src_ip and local dest_ip

outbound for local src_ip and external dest_ip

internal for local src_ip and local dest_ip

unknown for none of the above

Flow Enrichment

The following fields, not in the original flow records, are added to the Module output:

• FQDN name: Reverse-DNS lookup name of the src or dest, (src_host or dest_host)
• VM name: EC2 name for AWS, VM name for VMware, GCP, and Azure
• Security Reputation
• User identity
• Applications

Security Reputation

NFO matches source IP and destination IP against threat lists, and adds reputation information in the following fields:

• List name (threat_list_name)
• Reputation (reputation)

Where

• List name is the name of the list specified in configuration
• Reputation is provided by a threat list vendor or, for lists with malicious domain names (currently supported from AlienVault OTX), set by NFO using malicious domain name provided by the list vendor

note

Flows with reputation are always reported, even if they don't make it to Top N by traffic volume. Therefore, any communications with bad actors are never missed.

For more information, see Configuring Custom Threat List section in EDFN Administration Guide.

User Identity Enrichment

User identity is provided by EDFN agent User identity monitor and could be configured from this Module GUI. It builds two lookup lists:

• IPv4, idp, username
• IPv6, idp, username

Where

• idp is identity provider that reported IP address and username
• IPv4 is IP address from which the user logged on
• username is the name of the user or account name reported by identity provider

In this release the agent supports two methods:

• Integration with Active Directory Domain Controller
• Integration with Microsoft Azure Domain Controller
• Integration with Okta
• Integration with any identity management system reporting login/logout events via syslog

For more information, see Configuring User Identity Enrichment section in EDFN Administration Guide.

Module Parameters

To configure Module parameters click on 10062: Network Conversation Monitor.

Logic Parameters

Parameter Name	Description	Comments
N – The number of unique sessions to report	The number of unique stitched sessions reported per NetFlow exporter (Top N). NOTE: Conversations with malicious hosts are reported regardless whether they are in Top N by traffic volume or not!	min = 0, max = 100000, default - 0 (0 indicates all sessions are reported)
Session report timeout, sec	This parameter is used to report the duration of a session, along with its state. Specify the number of seconds during which no flows are observed to determine when a network conversation is considered ended. Event is reported with state=”E” this many seconds after last flow_end_time	min = 0 sec, max = 600 sec, default - 60 sec. This parameter should be set to a value greater than network device flow-cache inactive timeout. NOTE: 0 is a special value to indicate that niether duration nor state fields are required, which results in better performance.
Report inactive sessions	If set to 1, report inactive session with 0 bytes/packets, even if there were no flows during DCI. If set to 0, inactive sessions are not reported	default - 0
Report long flows with cumulative bytes and packets	If set to 1, cumulative bytes and packets are reported for long sessions. If set to 0, incremental bytes and packets are reported for long flows (state = “C”)	default - 0
Enable (1) or disable (0) deduplication	If set to 1 (de-duplication enabled), the Module reports flows only from authoritative exporters	default - 0
Enable (1) or disable (0) multiplying by sampling rate	If set to 1, when *flow is sampled (e.g. sFlow, sampled NetFlow/IPFIX), the sampling rate is used to multiply bytes and packets to report total traffic as statistical approximation	default - 0
Default sampler rate	If sampling information is not available, use this rate to multiply bytes and packets to report total traffic as statistical approximation	default - 1
Enable (1) or disable (0) reporting bidirectional conversations	If set to 1, stitch client-server flows reporting bytes_in and bytes_out, packets_in and packets_out in on consolidated message	default - 0
Enable (1) or disable (0) reporting client port	If set to 1, the ephemeral client port number is reported. If set to 0, client port number is not taken into account for consolidation, and reported as 0	default - 1
Enable (1) or disable (0) reporting flow denied events	If set to 1, denied or rejected flows are reported. If set to 0, only allowed or accepted flows are reported	default - 1
Enable (1) or disable (0) collecting application info from devices reporting it	If set to 1, application collector is enabled. If set to 0, applications are reported only in flows containing app fields	default - 1
Enable (1) or disable (0) enriching other devices with application info	If set to 1, application collector is used to enrich all flows, even if they did not have application fields	default - 1
Output filename for application info	Path to application collector file. Used for troubleshooting purposes	default - ../../logs/app_info.log
Enable (1) or disable (0) generating end of conversation events	If set to 1, events at the end of conversations (state=E) are created and reporteed. If set to 0, events with stary=E are not reported	default - 0

Data Consolidation Parameter

Parameter Name	Description	Comments
Data Collection Interval, sec	Module logic execution interval	min = 5 sec, max = 86400 sec, default - 30 sec

Data Sets and Enrichment Parameters

Parameter Name	Description	Comments
List of known server destination port numbers	List of server destination ports to be used to determine which host is a client and which is a server. If the list is empty, the server is the one with a smaller port number. This parameter is ignored for unidirectional flows.	This parameter is pre-loaded with values from: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml
List of local subnets	Used to identify direction for IPv4 traffic: inbound or outbound or internal	default - { 10.0.0.0,8; 172.16.0.0,12; and 192.168.0.0,16 }
List of local IPv6 prefixes	Used to identify direction for IPv6 traffic: inbound or outbound or internal	default - fc00:0:0:0:0:0:0:0,7
AWS EC2 instances list	List of EC2 instances with IPs and VPC names and other information	Provided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS VPC IPv4 Routes	List of AWS VPC IPv4 routes	Provided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS VPC IPv6 Routes	List of AWS VPC IPv6 routes	Provided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS IPv4 Ranges	List of AWS IPv4 ranges	Provided by EDFN agent
AWS IPv6 Ranges	List of AWS IPv6 ranges	Provided by EDFN agent
GCP VM instances list	List of Google cloud VM names	Provided by EDFN agent
GCP IPv4 routes list	List of Google cloud routes	Provided by EDFN agent
Azure VM Instances	List of Azure VM names	Provided by EDFN agent
Azure IPv4 Routes	List of Azure IPv4 routes	Provided by EDFN agent
Azure IPv6 Routes	List of Azure IPv6 routes	Provided by EDFN agent
List of Users by IPv4 address	List of Users logged in from IPv4 addresses	Provided by EDFN agent. Please see User Identity Agent configurations for details
List of Users by IPv6 address	List of Users logged in from IPv6 addresses	Provided by EDFN agent. Please see User Identity Agent configurations for details
Applications override list	List of App IDs and names to override applications reported by NetFlow/IPFIX	Created manually
List of Application names to be ignored	List of App names provided by DPI engines not to be collected / reported	Created manually. default - { incomplete; not-applicable; unknown-udp; unknown-tcp; unknown-p2p }
Custom Applications list	List of Custom Applications: IPv4 Address, Port, Protocol, Application Name, etc.	Created manually
Custom Threat list	List of public and private IP addresses with reputation known to be malicious hosts	Provided by EDFN agent. Please see Configuring Custom Threat List for details
IPv4 address block and country code	Mapping of country codes to IP addresses blocks	Provided by EDFN agent, which uses the MaxMind Country database as a source
IPv4 address block and city location	Mapping of city and country codes to IP addresses blocks	Provided by EDFN agent, which uses the MaxMind City database as a source
List of vCenter Virtual Machines	List of VMs, including: {VDS IPv4 address, VM IPv4 address, VM IPv6 address, VDS Port ID, vNIC key, Port Group name, VM name, VM MoRef, VM instance UUID, vCenter UUID}	Provided by EDFN agent by connecting to one or several vCenters
List of BD subnets to Tenant mapping	List of Cisco ACI Bridge domains and Tenants: {IPv4 Address, IPv4 Address Mask, BD name, Tenant name}	Created manually
AS Numbers IPv4 Blocks	Mapping of Autonomous System Number to IP addresses blocks	Provided by EDFN agent, which uses the MaxMind ASN database as a source
AS Numbers IPv6 Blocks	Mapping of Autonomous System Number to IP addresses blocks	Provided by EDFN agent, which uses the MaxMind ASN database as a source

Converter Parameters

To configure Converter parameters click on Converter for Network Conversation Monitor.

Parameter Name	Description	Comments
Output Destination	Enable or disable output destination options: 0 – syslog/JSON, 1 – AWS S3, 2 – both	default - 0
AWS S3 account credentials	Path to AWS account credentials file	e.g. root/.aws/credentials-s3
AWS S3 bucket name	Name of S3 bucket to send data to. You need to create this bucket in your AWS environment	string
AWS S3 NFO Folder	Name of AWS S3 folder (directory). This folder will be created by NFO	string
AWS S3 NFO Filename	Filename pattern used when S3 files are created	string, e.g nfo.log.gz creates S3 files as timestamp-nfo.log.gz
Output Fields	Configure Output Fields	default - all fields

Output Fields

To configure output fields, including the order, click on List of output fields.

Set Access to your AWS Accounts for S3 Output

Create an account in your AWS environment with with the following permissions:

s3:PutObject and s3:GetBucketLocation.

For example, inline policy may look like the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "NfoS3Permissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name"
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

note

Do not use an account you use to read VPC Flow logs.

Create an AWS credentials file, e.g. credentials-s3. It should be placed on the machine where NFO is installed. Use the IAM User public and secret access key to create a file as follows:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

Change file permissions to read only for root user (if NFO is running as root): chmod 400 credentials-s3.

Set path to this file, for example: /root/.aws/credentials-s3

Environment Variables for AWS S3 Output

When this Module writes output to AWS S3, the file is closed when one of the following occurs:

1. On file rotation interval timeout
2. The file has the number of records specified in chunk size
3. The number of bytes in the file reached buffer size

These parameters are controlled by the following environment variables.

Variable Name	Description	Comment
NFO_M_10062_S3_OUTPUT_BUFFER_SIZE	S3 output buffer size, bytes	Min - 32768, max - 16777216, default - 4194304
NFO_M_10062_S3_OUTPUT_THREADS	S3 output threads count	Min - 1, max - 64, default - 8
NFO_M_10062_S3_OUTPUT_QUEUE_LEN	S3 output queue length, records	Min - 1000, max - 512000, default - 10000
NFO_M_10062_S3_OUTPUT_CHUNK_SIZE	S3 output file chunk size, records	Min - 1, max - 1000000, default - 100000
NFO_M_10062_S3_OUTPUT_ROTATION_INT	S3 output file rotation interval, msec	Min - 1000, max - 3600000, default - 30000

Input

NetFlow v5, v9, IPFIX, sFlow, AWS VPC Flow Logs, Microsoft Azure NSG Flow logs.

Required NetFlow Fields

| IE id | IE size (Bytes) | Description | Information Element (IE) | | :--- |:--- |:--- | 8 or 27 |4 or 16 | The IPv4 or IPv6 source address in the IP packet header | sourceIPv4Address or sourceIPv6Address | | 12 or 28 |4 or 16 |The IPv4 or IPv6 destination address in the IP packet header | destinationIPv4Address or destinationIPv6Address | | 4| 1 | The value of the protocol number in the IP packet header. The protocol number identifies the IP packet payload type. Protocol numbers are defined in the IANA Protocol Numbers registry | protocolIdentifier | | 7| 2 | The source port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the source port number given in the respective header | sourceTransportPort | | 11 | 2 | The destination port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the destination port number given in the respective header | destinationTransportPort |

Syslog/JSON Message Fields

Key	Field Description	Comment
nfc_id	Message type identifier	string,nfc_id=20062
flow_type	Type of Flow	string, e.g. NFv5, NFv9, sFlow, IPFIX, AWS, Azure...
exp_ip	NetFlow exporter IPv4 address	IPv4_address (for public clouds added for compatibility with other flows)
input_snmp	Input SNMP index	number
output_snmp	Output SNMP index	number
exp_name	Exporter name	string
input_if_name	Input interface name	string
input_if_alias	Input interface alias	string
output_if_name	Output interface name	string
output_if_alias	Output interface alias	string
protocol	Transport Protocol	number, e.g. TCP = 6, UDP = 17
src_ip	Source IPv4 address	IPv4 address
src_ip6	Source IPv6 address	IPv6 address
src_port	Source transport port	number
src_tos	Source type of service	number
src_as	Source AS	number
src_cc	Source Country Code	string
src_region	Source region	string
src_city	Source city	string
src_lon	Source longitude	number
src_lat	Source latitude	number
src_mac	Source MAC address	string, e.g. e0:46:9a:2b:83:13
src_cloud_region	Cloud source region	string
src_host	Source host name	string, included when FQDN is on
src_vlan	Source VLAN	string
src_vm_name	Source VM name or AWS EC2 instance name	string
src_vpc_name	Source VPC name	string
src_subnet_name	Source subnet name	string
src_tenant_name	Cisco ACI source tenant	string
src_bd_name	Cisco ACI source bridge domain	string
dest_ip	Destination IPv4 address	IPv4 address
dest_ip6	Destination IPv6 address	IPv6 address
dest_port	Destination transport port	number
dest_tos	Destination type of service	number
dest_as	Destination AS	number
dest_cc	Destination Country Code	string
dest_region	Destination region	string
dest_city	Destination city	string
dest_lon	Destination longitude	number
dest_lat	Destination latitude	number
dest_vm_name	Destination VM name or AWS EC2 instance name	string
dest_vpc_name	Destination VPC name	string
dest_subnet_name	Destination subnet name	string
dest_tenant_name	Cisco ACI destination tenant	string
dest_bd_name	Cisco ACI destination bridge domain	string
dest_cloud_region	Cloud destination region	string
dest_mac	Destination MAC address	string, e.g. e0:46:9a:2b:83:13
dest_vlan	Destination VLAN	string
dest_host	Destination host name	string, included when FQDN is on
tcp_flag	TCP flags	string, e.g. SYN,ACK,FIN
packets_in	Packets in the flow received by destination IP from source IP	number
bytes_in	Total number of Layer 3 bytes in the packets of the flow received by destination IP from source IP	number
packets_out	Packets in the flow sent by destination IP to source IP	number
bytes_out	Total number of Layer 3 bytes in the packets of the flow sent by destination IP to source IP	number
flow_count	Number of consolidated flows reported in this event	number
action [^1]	Flow action	string, The action is determined from IPFIX element 233 - firewallEvent and NFv9 / IPFIX element 89 - forwardingStatus
state	Flow state	string, B = Begin, C = Continuing, E = End
latency	As reported in flow records in msec	number
duration	Session duration - unidirectional / Conversation duration - bidirectional. Reported in sec	number
direction	Direction of the flow, if reported, or direction determined based on local subnets	string, inbound (local IP address is dest), outbound (local IP address is src ), internal (both, src and dest IP addresses are local), unknown (both src and dest IP addresses are not local)
idp	IDP for the user	string
username	User name provided by EDFN Agent ( UserName Type 371 - upcoming)	string
app_id	Application ID (Type 95)	string, Class Eng. ID:Selector ID (see Section 4 https://www.rfc-editor.org/rfc/rfc6759.html)
app_name	Application Name (Type 96)	string
app_desc	Application Description (Type 94)	string
app_engine_id	Application (Classification) Engine ID	string, Class Eng. ID description for part 1 of Type 95 (Type 101 - upcoming)
threat_list_name	The name of a cybersecurity threat list	string
reputation	Reputation from the treat list	string
aws_vpc_id	AWS VPC identifier	string
aws_vpc_name	AWS VPC name	string
aws_interface_id	AWS Interface Id	string
aws_account_id	AWS Account Id	string
gcp_reporter	GCP VPC Flow logs Reporter	string, SRC or DEST
gcp_exp	GCP VPC Flow logs Exporter. Calculated field based on reporter = SRC or DEST	string, Project ID/VPC/Subnet
gcp_subnet_id	GCP Subnet ID	string
aws_src_ip_pub	Source EC2 instance public IPv4 address	IPv4 address
aws_src_inst_id	Source EC2 instance id	string, e.g. i-390d7032 or i-0c0a6ac75d9d87b7e
gcp_src_project_id	GCP Source Project ID	string
gcp_src_vm_zone	GCP Source VM Zone	string
azure_src_subs_id	Azure Source Subscription ID	string
azure_src_subs_name	Azure Source Subscription Name	string
azure_src_nsg_name	Azure Source NSG Name	string
azure_src_vnet_name	Azure Source Virtual Network Name	string
azure_src_res_grp_name	Azure Source Resource Group Name	string
aws_dest_ip_pub	Destination EC2 instance public IPv4 address	IPv4 address
aws_dest_inst_id	Destination EC2 instance id	string
gcp_dest_project_id	GCP Destination Project ID	string
gcp_dest_vm_zone	GCP Destination VM Zone	string
azure_dest_subs_id	Azure Destination Subscription ID	string
azure_dest_subs_name	Azure Destination Subscription Name	string
azure_dest_nsg_name	Azure Destination NSG Name	string
azure_dest_vnet_name	Azure Destination Virtual Network Name	string
azure_dest_res_grp_name	Azure Destination Resource Group Name	string
flow_start_time	Start time of the first consolidated flow	time
flow_end_time	End of the last consolidated flow	time
t_int	Observation time interval, msec	number

[^1] Action is reported as follows:

• action=R for firewallEvent 0 (ignored), 2 (deleted), and 3 (denied), and Rejected cloud flow logs
• action=A for firewallEvent 1 (created), 4 (alert), and 5 (update), and Allowed cloud flow logs
• action=U for forwardingStatus 00 (unknown)
• action=F for forwardingStatus 01 (forwarded)
• action=D for forwardingStatus 10 (dropped)
• action=C for forwardingStatus 11 (consumed)