Skip to main content
Version: 2.10.1

Network Conversations Monitor (10062 / 20062)

Description

This Module reports consolidated network conversations.

Consolidation

This Module consolidates *flow records over a period of time - Data Collection Interval (DCI) which all have the same combination of the following fields:

  • Exporter IP (*flow exporter for physical devices, VPC for AWS, Virtual Network for Azure, Subnetwork for Google cloud)
  • Source IP address
  • Destination IP address
  • Source port number
  • Destination port number
  • Layer 3 protocol
  • Reporter (for Google Flow logs)

Flow Stitching - Report Bidirectional Conversations

This Module optionally stitches client-server request-response flows, reporting bytes and packets server-to-client and client-to-server in separate fields: bytes_in/bytes_out and packetsin/packetsout.

Deduplication

Optionally the Module can report host pairs only from authoritative router/switch. Authoritative network device is determined as follows. The Module sums up bytes, packets, and connections between two hosts over data collection interval (parameter, default - 30 sec), reported by each flow exporter. An exporter with most connections (flows) for each host pair is considered authoritative, and host pair conversations reported by all other exporters are discarded.

State of Conversation

The state field indicates the state of the conversation and may have the following values: “B”: Begin, “C”: Continuing, “E”: End. The state is reported for each conversation at the end of Data Collection Interval (DCI).

The logic for the state field is:

  • The state is set to “B” when flows for a new host pair is observed fir the first time (or after we see them again after previous conversation has ended) during DCI.

  • The state is set to “C” if the conversation was reported in previous DCI and flows were observed in the current DCI.

  • The state is set to “E” when no flows were observed for the length of “Session report timeout” parameter.

note

Please note that the state of conversation is reliably reported when Top N is set to report all sessions, and not only top traffic.

Conversation Duration

Network Conversation duration is calculated as a difference between the earliest flow_start_time of a conversation  when its state=B and the latest flow_end_time for state = C or E.

Direction

The direction field indicates in which direction (inbound or outbound) the network conversation is going. It is determined based on configuration of local subnets in the Module (List of local subnets for IPv4 and List of local IPv6 prefixes for IPv6).

The values are:

inbound for external src_ip and local dest_ip

outbound for local src_ip and external dest_ip

internal for local src_ip and local dest_ip

unknown for none of the above

Flow Enrichment

The following fields, not in the original flow records, are added to the Module output:

  • FQDN name: Reverse-DNS lookup name of the src or dest, (src_host or dest_host)
  • VM name: EC2 name for AWS, VM name for VMware, GCP, and Azure
  • Security Reputation
  • User identity
  • Applications

Security Reputation

NFO matches source IP and destination IP against threat lists, and adds reputation information in the following fields:

  • List name (threat_list_name)
  • Reputation (reputation)

Where

  • List name is the name of the list specified in configuration
  • Reputation is provided by a threat list vendor or, for lists with malicious domain names (currently supported from AlienVault OTX), set by NFO using malicious domain name provided by the list vendor
note

Flows with reputation are always reported, even if they don't make it to Top N by traffic volume. Therefore, any communications with bad actors are never missed.

For more information, see Configuring Custom Threat List section in EDFN Administration Guide.

User Identity Enrichment

User identity is provided by EDFN agent User identity monitor and could be configured from this Module GUI. It builds two lookup lists:

  • IPv4, idp, username
  • IPv6, idp, username

Where

  • idp is identity provider that reported IP address and username
  • IPv4 is IP address from which the user logged on
  • username is the name of the user or account name reported by identity provider

In this release the agent supports two methods:

  • Integration with Active Directory Domain Controller
  • Integration with Microsoft Azure Domain Controller
  • Integration with Okta
  • Integration with any identity management system reporting login/logout events via syslog

For more information, see Configuring User Identity Enrichment section in EDFN Administration Guide.

Module Parameters

To configure Module parameters click on 10062: Network Conversation Monitor.

Logic Parameters

Parameter NameDescriptionComments
N – The number of unique sessions to reportThe number of unique stitched sessions reported per NetFlow exporter (Top N).

NOTE: Conversations with malicious hosts are reported regardless whether they are in Top N by traffic volume or not!
min = 0, max = 100000, default - 0 (0 indicates all sessions are reported)
Session report timeout, secThis parameter is used to report the duration of a session, along with its state. Specify the number of seconds during which no flows are observed to determine when a network conversation is considered ended. Event is reported with state=”E” this many seconds after last flow_end_timemin = 0 sec, max = 600 sec, default - 60 sec. This parameter should be set to a value greater than network device flow-cache inactive timeout.

NOTE: 0 is a special value to indicate that niether duration nor state fields are required, which results in better performance.
Report inactive sessionsIf set to 1, report inactive session with 0 bytes/packets, even if there were no flows during DCI. If set to 0, inactive sessions are not reporteddefault - 0
Report long flows with cumulative bytes and packetsIf set to 1, cumulative bytes and packets are reported for long sessions. If set to 0, incremental bytes and packets are reported for long flows (state = “C”)default - 0
Enable (1) or disable (0) deduplicationIf set to 1 (de-duplication enabled), the Module reports flows only from authoritative exportersdefault - 0
Enable (1) or disable (0) multiplying by sampling rateIf set to 1, when *flow is sampled (e.g. sFlow, sampled NetFlow/IPFIX), the sampling rate is used to multiply bytes and packets to report total traffic as statistical approximationdefault - 0
Default sampler rateIf sampling information is not available, use this rate to multiply bytes and packets to report total traffic as statistical approximationdefault - 1
Enable (1) or disable (0) reporting bidirectional conversationsIf set to 1, stitch client-server flows reporting bytes_in and bytes_out, packets_in and packets_out in on consolidated messagedefault - 0
Enable (1) or disable (0) reporting client portIf set to 1, the ephemeral client port number is reported. If set to 0, client port number is not taken into account for consolidation, and reported as 0default - 1
Enable (1) or disable (0) reporting flow denied eventsIf set to 1, denied or rejected flows are reported. If set to 0, only allowed or accepted flows are reporteddefault - 1
Enable (1) or disable (0) collecting application info from devices reporting itIf set to 1, application collector is enabled. If set to 0, applications are reported only in flows containing app fieldsdefault - 1
Enable (1) or disable (0) enriching other devices with application infoIf set to 1, application collector is used to enrich all flows, even if they did not have application fieldsdefault - 1
Output filename for application infoPath to application collector file. Used for troubleshooting purposesdefault - ../../logs/app_info.log
Enable (1) or disable (0) generating end of conversation eventsIf set to 1, events at the end of conversations (state=E) are created and reporteed. If set to 0, events with stary=E are not reporteddefault - 0

Data Consolidation Parameter

Parameter NameDescriptionComments
Data Collection Interval, secModule logic execution intervalmin = 5 sec, max = 86400 sec, default - 30 sec

Data Sets and Enrichment Parameters

Parameter NameDescriptionComments
List of known server destination port numbersList of server destination ports to be used to determine which host is a client and which is a server. If the list is empty, the server is the one with a smaller port number. This parameter is ignored for unidirectional flows.This parameter is pre-loaded with values from: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml
List of local subnetsUsed to identify direction for IPv4 traffic: inbound or outbound or internaldefault - { 10.0.0.0,8; 172.16.0.0,12; and 192.168.0.0,16 }
List of local IPv6 prefixesUsed to identify direction for IPv6 traffic: inbound or outbound or internaldefault - fc00:0:0:0:0:0:0:0,7
AWS EC2 instances listList of EC2 instances with IPs and VPC names and other informationProvided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS VPC IPv4 RoutesList of AWS VPC IPv4 routesProvided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS VPC IPv6 RoutesList of AWS VPC IPv6 routesProvided by EDFN agent. Please see Amazon VPC Flow Logs for details
AWS IPv4 RangesList of AWS IPv4 rangesProvided by EDFN agent
AWS IPv6 RangesList of AWS IPv6 rangesProvided by EDFN agent
GCP VM instances listList of Google cloud VM namesProvided by EDFN agent
GCP IPv4 routes listList of Google cloud routesProvided by EDFN agent
Azure VM InstancesList of Azure VM namesProvided by EDFN agent
Azure IPv4 RoutesList of Azure IPv4 routesProvided by EDFN agent
Azure IPv6 RoutesList of Azure IPv6 routesProvided by EDFN agent
List of Users by IPv4 addressList of Users logged in from IPv4 addressesProvided by EDFN agent. Please see User Identity Agent configurations for details
List of Users by IPv6 addressList of Users logged in from IPv6 addressesProvided by EDFN agent. Please see User Identity Agent configurations for details
Applications override listList of App IDs and names to override applications reported by NetFlow/IPFIXCreated manually
List of Application names to be ignoredList of App names provided by DPI engines not to be collected / reportedCreated manually. default - { incomplete; not-applicable; unknown-udp; unknown-tcp; unknown-p2p }
Custom Applications listList of Custom Applications: IPv4 Address, Port, Protocol, Application Name, etc.Created manually
Custom Threat listList of public and private IP addresses with reputation known to be malicious hostsProvided by EDFN agent. Please see Configuring Custom Threat List for details
IPv4 address block and country codeMapping of country codes to IP addresses blocksProvided by EDFN agent, which uses the MaxMind Country database as a source
IPv4 address block and city locationMapping of city and country codes to IP addresses blocksProvided by EDFN agent, which uses the MaxMind City database as a source
List of vCenter Virtual MachinesList of VMs, including: {VDS IPv4 address, VM IPv4 address, VM IPv6 address, VDS Port ID, vNIC key, Port Group name, VM name, VM MoRef, VM instance UUID, vCenter UUID}Provided by EDFN agent by connecting to one or several vCenters
List of BD subnets to Tenant mappingList of Cisco ACI Bridge domains and Tenants: {IPv4 Address, IPv4 Address Mask, BD name, Tenant name}Created manually
AS Numbers IPv4 BlocksMapping of Autonomous System Number to IP addresses blocksProvided by EDFN agent, which uses the MaxMind ASN database as a source
AS Numbers IPv6 BlocksMapping of Autonomous System Number to IP addresses blocksProvided by EDFN agent, which uses the MaxMind ASN database as a source

Converter Parameters

To configure Converter parameters click on Converter for Network Conversation Monitor.

Parameter NameDescriptionComments
Output Destination

Enable or disable output destination options:

0 – syslog/JSON, 1 – AWS S3, 2 – both

default - 0
AWS S3 account credentialsPath to AWS account credentials filee.g. root/.aws/credentials-s3
AWS S3 bucket nameName of S3 bucket to send data to. You need to create this bucket in your AWS environmentstring
AWS S3 NFO FolderName of AWS S3 folder (directory). This folder will be created by NFOstring
AWS S3 NFO FilenameFilename pattern used when S3 files are createdstring, e.g nfo.log.gz creates S3 files as timestamp-nfo.log.gz
Output FieldsConfigure Output Fieldsdefault - all fields

Output Fields

To configure output fields, including the order, click on List of output fields.

Set Access to your AWS Accounts for S3 Output

Create an account in your AWS environment with with the following permissions:

s3:PutObject and s3:GetBucketLocation.

For example, inline policy may look like the following:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "NfoS3Permissions",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::your-bucket-name"
"arn:aws:s3:::your-bucket-name/*"
]
}
]
}
note

Do not use an account you use to read VPC Flow logs.

Create an AWS credentials file, e.g. credentials-s3. It should be placed on the machine where NFO is installed. Use the IAM User public and secret access key to create a file as follows:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

Change file permissions to read only for root user (if NFO is running as root): chmod 400 credentials-s3.

Set path to this file, for example: /root/.aws/credentials-s3

Environment Variables for AWS S3 Output

When this Module writes output to AWS S3, the file is closed when one of the following occurs:

  1. On file rotation interval timeout
  2. The file has the number of records specified in chunk size
  3. The number of bytes in the file reached buffer size

These parameters are controlled by the following environment variables.

Variable NameDescriptionComment
NFO_M_10062_S3_OUTPUT_BUFFER_SIZES3 output buffer size, bytesMin - 32768, max - 16777216, default - 4194304
NFO_M_10062_S3_OUTPUT_THREADSS3 output threads countMin - 1, max - 64, default - 8
NFO_M_10062_S3_OUTPUT_QUEUE_LENS3 output queue length, recordsMin - 1000, max - 512000, default - 10000
NFO_M_10062_S3_OUTPUT_CHUNK_SIZES3 output file chunk size, recordsMin - 1, max - 1000000, default - 100000
NFO_M_10062_S3_OUTPUT_ROTATION_INTS3 output file rotation interval, msecMin - 1000, max - 3600000, default - 30000

Input

NetFlow v5, v9, IPFIX, sFlow, AWS VPC Flow Logs, Microsoft Azure NSG Flow logs.

Required NetFlow Fields

IE idIE size (Bytes)Description
8 or 274 or 16The IPv4 or IPv6 source address in the IP packet header
12 or 284 or 16The IPv4 or IPv6 destination address in the IP packet header
41The value of the protocol number in the IP packet header. The protocol number identifies the IP packet payload type. Protocol numbers are defined in the IANA Protocol Numbers registry
72The source port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the source port number given in the respective header
112The destination port identifier in the transport header. For the transport protocols UDP, TCP, and SCTP, this is the destination port number given in the respective header

Syslog/JSON Message Fields

KeyField Description  Comment  
nfc_idMessage type identifierstring,nfc_id=20062  
flow_type  Type of Flow  string, e.g. NFv5, NFv9, sFlow, IPFIX, AWS, Azure...  
exp_ipNetFlow exporter IPv4 address  IPv4_address (for public clouds added for compatibility with other flows)
input_snmpInput SNMP index  number
output_snmpOutput SNMP index  number
exp_nameExporter namestring
input_if_nameInput interface namestring
input_if_aliasInput interface aliasstring
output_if_nameOutput interface namestring
output_if_aliasOutput interface aliasstring
protocolTransport Protocolnumber, e.g. TCP = 6, UDP = 17
src_ipSource IPv4 addressIPv4 address  
src_ip6Source IPv6 addressIPv6 address  
src_port  Source transport port  number
src_tos  Source type of service  number
src_as  Source AS  number
src_ccSource Country Codestring
src_regionSource regionstring
src_citySource citystring
src_lonSource longitudenumber
src_latSource latitudenumber
src_macSource MAC addressstring, e.g. e0:46:9a:2b:83:13
src_cloud_regionCloud source regionstring
src_host  Source host name  string, included when FQDN is on  
src_vlan  Source VLANstring
src_vm_name  Source VM name or AWS EC2 instance namestring
src_vpc_name  Source VPC namestring
src_subnet_name  Source subnet namestring
src_tenant_nameCisco ACI source tenantstring
src_bd_nameCisco ACI source bridge domainstring
dest_ipDestination IPv4 address  IPv4 address  
dest_ip6Destination IPv6 address  IPv6 address  
dest_port  Destination transport portnumber
dest_tos  Destination type of service  number
dest_as  Destination AS  number
dest_ccDestination Country Codestring
dest_regionDestination regionstring
dest_cityDestination citystring
dest_lonDestination longitudenumber
dest_latDestination latitudenumber
dest_vm_name  Destination VM name or AWS EC2 instance namestring
dest_vpc_name  Destination VPC namestring
dest_subnet_name  Destination subnet namestring
dest_tenant_nameCisco ACI destination tenantstring
dest_bd_nameCisco ACI destination bridge domainstring
dest_cloud_regionCloud destination regionstring
dest_macDestination MAC addressstring, e.g. e0:46:9a:2b:83:13
dest_vlan  Destination VLANstring
dest_host  Destination host name  string, included when FQDN is on  
tcp_flag  TCP flags  string, e.g. SYN,ACK,FIN  
packets_inPackets in the flow received by destination IP from source IP  number
bytes_in  Total number of Layer 3 bytes in the packets of the flow received by destination IP from source IPnumber
packets_outPackets in the flow sent by destination IP to source IPnumber
bytes_out  Total number of Layer 3 bytes in the packets of the flow sent by destination IP to source IP  number
flow_countNumber of consolidated flows reported in this eventnumber
action 1 Flow actionstring, The action is determined from IPFIX element 233 - firewallEvent and NFv9 / IPFIX element 89 - forwardingStatus
state  Flow statestring, B = Begin, C = Continuing, E = End
latencyAs reported in flow records in msecnumber
durationSession duration - unidirectional / Conversation duration - bidirectional. Reported in secnumber
direction  Direction of the flow, if reported, or direction determined based on local subnetsstring, inbound (local IP address is dest), outbound (local IP address is src ), internal (both, src and dest IP addresses are local), unknown (both src and dest IP addresses are not local)
idpIDP for the user  string
usernameUser name provided by EDFN Agent ( UserName Type 371 - upcoming)  string
app_idApplication ID (Type 95)  string, Class Eng. ID:Selector ID (see Section 4 https://www.rfc-editor.org/rfc/rfc6759.html)  
app_name  Application Name (Type 96)string
app_desc  Application Description (Type 94)  string
app_engine_idApplication (Classification) Engine IDstring, Class Eng. ID description for part 1 of Type 95 (Type 101 - upcoming)
threat_list_name  The name of a cybersecurity threat liststring
reputation  Reputation from the treat liststring
aws_vpc_idAWS VPC identifierstring
aws_vpc_name  AWS VPC name  string
aws_interface_id  AWS Interface Id  string
aws_account_idAWS Account Idstring
gcp_reporter  GCP VPC Flow logs Reporterstring, SRC or DEST  
gcp_expGCP VPC Flow logs Exporter. Calculated field based on reporter = SRC or DEST  string, Project ID/VPC/Subnet
gcp_subnet_idGCP Subnet ID  string
aws_src_ip_pub  Source EC2 instance public IPv4 addressIPv4 address  
aws_src_inst_id  Source EC2 instance idstring, e.g. i-390d7032 or i-0c0a6ac75d9d87b7e
gcp_src_project_id  GCP Source Project ID  string
gcp_src_vm_zone  GCP Source VM Zonestring
azure_src_subs_idAzure Source Subscription ID  string
azure_src_subs_name  Azure Source Subscription Namestring
azure_src_nsg_name  Azure Source NSG Name  string
azure_src_vnet_name  Azure Source Virtual Network Name  string
azure_src_res_grp_name  Azure Source Resource Group Name  string
aws_dest_ip_pub  Destination EC2 instance public IPv4 address  IPv4 address  
aws_dest_inst_idDestination EC2 instance idstring
gcp_dest_project_id  GCP Destination Project IDstring
gcp_dest_vm_zoneGCP Destination VM Zonestring
azure_dest_subs_id  Azure Destination Subscription ID  string
azure_dest_subs_nameAzure Destination Subscription Namestring
azure_dest_nsg_name  Azure Destination NSG Namestring
azure_dest_vnet_nameAzure Destination Virtual Network Namestring
azure_dest_res_grp_nameAzure Destination Resource Group Name  string
flow_start_time  Start time of the first consolidated flow  time  
flow_end_timeEnd of the last consolidated flow  time  
t_int  Observation time interval, msecnumber

1 Action is reported as follows:

  • action=R for firewallEvent 0 (ignored), 2 (deleted), and 3 (denied), and Rejected cloud flow logs
  • action=A for firewallEvent 1 (created), 4 (alert), and 5 (update), and Allowed cloud flow logs
  • action=U for forwardingStatus 00 (unknown)
  • action=F for forwardingStatus 01 (forwarded)
  • action=D for forwardingStatus 10 (dropped)
  • action=C for forwardingStatus 11 (consumed)