Network Conversations Monitor (10062 / 20062)

Description

This Module reports consolidated network conversations.

Consolidation

This Module consolidates *flow records over a period of time - Data Collection Interval (DCI) which all have the same combination of the following fields:
    Exporter IP (*flow exporter for physical devices, VPC for AWS, Virtual Network for Azure, Subnetwork for Google cloud)
    Source IP address
    Destination IP address
    Source port number
    Destination port number
    Layer 3 protocol
    Reporter (for Google Flow logs)

Flow Stitching - Report Bidirectional Conversations

This Module optionally stitches client-server request-response flows, reporting bytes and packets server-to-client and client-to-server in separate fields: bytes_in/bytes_out and packetsin/packetsout.

Deduplication

Optionally the Module can report host pairs only from authoritative router/switch. Authoritative network device is determined as follows. The Module sums up bytes, packets, and connections between two hosts over data collection interval (parameter, default = 30 sec), reported by each flow exporter. An exporter with most connections (flows) for each host pair is considered authoritative, and host pair conversations reported by all other exporters are discarded.

Conversation Duration

Network Conversation duration is based on flow_start_time and flow_end_time and calculated as follows:
For TCP session duration is reported when the session is terminated. It is calculated as the time between the flow_start_time of the SYN flow and the flow_end_time of the FIN/RST flow. Session report timeout interval is a parameter to report TCP sessions that did not get FIN/RST within this time after last flow of the conversation is received.
For non-TCP protocols duration is reported with event state=”E” and is calculated as the time interval between flow_start_time of the first flow and flow_end_time of the last flow.

State of Conversation

The state field indicates the state of the conversation and may have the following values: “B”: Begin, “C”: Continuing, “E”: End. The state is reported for each conversation at the end of Data Collection Interval (DCI).
The logic for the state field is:
For TCP protocol TCP flags are used to report the state. The state is set to “B” when NFO received a flow with “SYN” or the first flow for the confirmation. The state is set to “C” for continuing conversations. Finally, the state is set to “E” when FIN/RST flow is received or TCP session times out. When a session begins and ends withing DCI, the state is set to "E".
For UDP (and other) protocol: The state is set to “B” when flows for a new host pair is observed during DCI. The set is set to “C” if the conversation was reported in previous DCI and flows were observed in subsequent DCIs. The set is set to “E” when no flows were observed for the length of “Session report timeout” parameter.

Direction

The direction field indicates in which direction (inbound or outbound) the network conversation is going. It is determined based on configuration of local subnets in the Module (List of local subnets for IPv4 and List of local IPv6 prefixes for IPv6).
The values are:
inbound for external src_ip and local dest_ip outbound for local src_ip and external dest_ip internal for local src_ip and local dest_ip unknown for none of the above

Flow Enrichment

The following fields, not in the original flow records, are added to the Module output:
    FQDN name: Reverse-DNS lookup name of the src or dest, (src_host or dest_host)
    VM name: EC2 name for AWS, VM name for VMware, GCP, and Azure

Module Parameters

To configure Module parameters click on Network Conversation Monitor.
Parameter Name
Description
Comments
Data Collection Interval, sec
Module logic execution interval
min = 5 sec, max = 86400 sec, default = 300 sec
N – The number of unique sessions to report
The number of unique stitched sessions reported per NetFlow exporter
min = 0, max = 100000, default = 0 (0 indicates all sessions are reported)
Session report timeout, sec
The number of seconds that an inactive flow remains unreported. Event is reported with state=”E” this many seconds after last flow_end_time
min = 10 sec, max = 600 sec, default = 60 sec. This parameter should be set to a value greater than network device flow-cache inactive timeout
Report inactive sessions
If set to 1, report inactive session with 0 bytes/packets, even if there were no flows during DCI. If set to 0, inactive sessions are not reported
Default = 0
Report long flows with cumulative bytes and packets
If set to 1, cumulative bytes and packets are reported for long sessions. If set to 0, incremental bytes and packets are reported for long flows (state = “C”)
Default = 0
Enable (1) or disable (0) deduplication
If set to 1 (de-duplication enabled), the Module reports flows only from authoritative exporters
Default = 0
Enable (1) or disable (0) multiplying by sampling rate
If set to 1, when *flow is sampled (e.g. sFlow, sampled NetFlow/IPFIX), the sampling rate is used to multiply bytes and packets to report total traffic as statistical approximation
Default = 0
Default sampler rate
If sampling information is not available, use this rate to multiply bytes and packets to report total traffic as statistical approximation
Default = 1
Report bidirectional conversations
If set to 1, stitch client-server flows reporting bytes_in and bytes_out, packets_in and packets_out
Default = 0
Enable(1) or disable (0) reporting client port
If set to 1, the ephemeral client port number is reported. If set to 0, client port number is not taken into account for consolidation, and reported as 0
Default = 1
List of known server destination port numbers
List of server destination ports to be used to determine which host is a client and which is a server. If the list is empty, the server is the one sending more traffic than receiving. This parameter is ignored for unidirectional flows
List of local subnets
Used to identify direction for IPv4 traffic: inbound or outbound
default = 10.0.0.0,8; 172.16.0.0,12; and 192.168.0.0,16
List of local IPv6 prefixes
Used to identify direction for IPv6 traffic: inbound or outbound
default - fc00:0:0:0:0:0:0:0,7

Converter Parameters

To configure Converter parameters click on NConverter for Network Conversation Monitor.
Parameter Name
Description
Comments
Output Destination
Enable or disable output destination options:
0 – syslog/JSON, 1 – AWS S3, 2 – both
Default = 0
AWS S3 account credentials
Path to AWS account credentials file
e.g. root/.aws/credentials-s3
AWS S3 bucket name
Name of S3 bucket to send data to. You need to create this bucket in your AWS
<string>
AWS S3 NFO Folder
Name of AWS S3 folder (directory). This folder will be created by NFO
<string>
AWS S3 Region
Region where S3 bucket is created
<string>, e.g us-west-1
Output Fields
Configure Output Fields
Default - all fields

Output Fields

To configure output fields, including the order, click on List of output fields.

Set Access to your AWS Accounts for S3 Output

Create an account in your AWS environment with AmazonS3FullAccess permissions policy.
Do not use an account you use to read VPC Flow logs.
Create an AWS credentials file, e.g. credentials-s3. It should be placed on the machine where NFO is installed. Use the IAM User public and secret access key to create a file as follows:
1
[default]
2
aws_access_key_id = your_access_key_id
3
aws_secret_access_key = your_secret_access_key
Copied!
Change file permissions to read only for root user (if NFO is running as root): chmod 400 credentials. The Module reads the file and takes all profiles from it. The Module expects that each account has only one profile.
Set path to this file, for example: /root/.aws/credentials-s3

Environment Variables for AWS S3 Output

When this Module writes output to AWS S3, the file is closed when one of the following occurs:
    1.
    On file rotation interval timeout
    2.
    The file has the number of records specified in chunk size
    3.
    The number of bytes in the file reached buffer size
These parameters are controlled by the following environment variables.
Variable Name
Description
Comment
NFO_M_10062_S3_OUTPUT_BUFFER_SIZE
S3 output buffer size, bytes
Min - 32768, max - 16777216, default - 4194304
NFO_M_10062_S3_OUTPUT_THREADS
S3 output threads count
Min - 1, max - 64, default - 8
NFO_M_10062_S3_OUTPUT_QUEUE_LEN
S3 output queue length, records
Min - 1000, max - 512000, default - 10000
NFO_M_10062_S3_OUTPUT_CHUNK_SIZE
S3 output file chunk size, records
Min - 1, max - 1000000, default - 100000
NFO_M_10062_S3_OUTPUT_ROTATION_INT
S3 output file rotation interval, msec
Min - 1000, max - 3600000, default - 30000

Input

NetFlow v5, v9, IPFIX, sFlow, AWS VPC Flow Logs, Microsoft Azure NSG Flow logs.

Required NetFlow Fields

Information Element (IE)
IE id
IE size, B
Description
sourceIPv4Address or sourceIPv6Address
8 or 27
4 or 16
The IPv4 or IPv6 source address in the IP packet header
destinationIPv4Address or destinationIPv6Address
12 or 28
4 or 16
The IPv4 or IPv6 destination address in the IP packet header
protocolIdentifier
4
1
The value of the protocol number in the IP packet header. The protocol number identifies the IP packet payload type. Protocol numbers are defined in the IANA Protocol Numbers registry
sourceTransportPort
7
2
The source port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the source port number given in the respective header
destinationTransportPort
11
2
The destination port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the destination port number given in the respective header

Syslog/JSON Message Fields

Key
Field Description
Comment
nfc_id
Message type identifier
string,nfc_id=20062
flow_type
Type of Flow
string, e.g. NFv5, NFv9, sFlow, IPFIX, AWS, Azure...
exp_ip
NetFlow exporter IPv4 address
IPv4_address (for public clouds added for compatibility with other flows)
input_snmp
Input SNMP index
number
output_snmp
Output SNMP index
number
protocol
Transport Protocol
number, e.g. TCP = 6, UDP = 17
src_ip
Source IPv4 address
IPv4 address
src_ip6
Source IPv6 address
IPv6 address
src_port
Source transport port
number
src_mac
Source MAC address
string, e.g. e0:46:9a:2b:83:13
src_vlan
Source VLAN
string
src_host
Source host name
string, included when FQDN is on
src_region
Cloud Source Region
string
src_vm_name
Source VM name or AWS EC2 instance name
string
src_vpc_name
Source VPC Name
string
dest_ip
Destination IPv4 address
IPv4 address
dest_ip6
Destination IPv6 address
IPv6 address
dest_port
Destination transport port
number
dest_mac
Destination MAC address
string, e.g. 00:30:48:d7:24:3e
dest_vlan
Destination VLAN
string
dest_host
Destination host name
string, included when FQDN is on
dest_region
Cloud Destination Region
string
dest_vm_name
Destination VM name or AWS EC2 instance name
string
dest_vpc_name
Destination VPC Name
string
tcp_flag
TCP Flags
string, e.g. SYN,ACK,FIN
packets_in
Packets in the flow received
number
bytes_in
Total number of Layer 3 bytes in the packets of the flow received
number
packets_out
Packets in the flow sent
number
bytes_out
Total number of Layer 3 bytes in the packets of the flow sent
number
flow_count
Number of consolidated Flows
number
action
Flow Action
string, A = Accepted or Allowed, R = Rejected or Blocked or Denied
state
Flow state
string, B = Begin, C = Continuing, E = End
latency
As reported in *flow records in msec
number
duration
Session duration - unidirectional / Conversation duration - bidirectional. Reported in sec
number
direction
Direction of the flow, if reported, or direction of the first flow of the conversation
string, inbound (local IP address is dest), outbound (local IP address is src ), internal (both, src and dest IP addresses are local), unknown (both src and dest IP addresses are not local)
idp
IDP for the user
string
user
User name provided by EDFN Agent ( UserName Type 371 - upcoming)
string
app_id
Application ID (Type 95)
string, Class Eng. ID:Selector ID (see Section 4 https://www.rfc-editor.org/rfc/rfc6759.html)
app_name
Application Name (Type 96)
string
app_desc
Application Description (Type 94)
string
app_engine_id
Application (Classification) Engine ID
string, Class Eng. ID description for part 1 of Type 95 (Type 101 - upcoming)
threat_list_name
The name of a cybersecurity threat list
string
reputation
Reputation from the treat list
string
aws_vpc_id
AWS VPC identifier
string
aws_vpc_name
AWS VPC name
string
aws_interface_id
AWS Interface Id
string
aws_account_id
AWS Account Id
string
gcp_reporter
GCP VPC Flow logs Reporter
string, SRC or DEST
gcp_exp
GCP VPC Flow logs Exporter. Calculated field based on reporter = SRC or DEST
string, Project ID/VPC/Subnet
gcp_subnet_id
GCP Subnet ID
string
aws_src_ip_pub
Source EC2 instance public IPv4 address
IPv4 address
aws_src_inst_id
Source EC2 instance id
string, e.g. i-390d7032 or i-0c0a6ac75d9d87b7e
gcp_src_project_id
GCP Source Project ID
string
gcp_src_vm_zone
GCP Source VM Zone
string
gcp_src_subnet_name
GCP Source Subnet Name
string
azure_src_subs_id
Azure Source Subscription ID
string
azure_src_subs_name
Azure Source Subscription Name
string
azure_src_nsg_name
Azure Source NSG Name
string
azure_src_vnet_name
Azure Source Virtual Network Name
string
azure_src_subnet_name
Azure Source Subnet Name
string
azure_src_res_grp_name
Azure Source Resource Group Name
string
aws_dest_ip_pub
Destination EC2 instance public IPv4 address
IPv4 address
aws_dest_inst_id
Destination EC2 instance id
string
gcp_dest_project_id
GCP Destination Project ID
string
gcp_dest_vm_zone
GCP Destination VM Zone
string
gcp_dest_subnet_name
GCP Destination Subnet Name
string
azure_dest_subs_id
Azure Destination Subscription ID
string
azure_dest_subs_name
Azure Destination Subscription Name
string
azure_dest_nsg_name
Azure Destination NSG Name
string
azure_dest_vnet_name
Azure Destination Virtual Network Name
string
azure_dest_subnet_name
Azure Destination Subnet Name
string
azure_dest_res_grp_name
Azure Destination Resource Group Name
string
flow_start_time
Start time of the first consolidated flow
time
flow_end_time
End of the last consolidated flow
time
t_int
Observation time interval, msec
number
Last modified 1mo ago