Azure Top Traffic Monitor (10467 / 20467)

Description

This Module identifies Azure VMs with the most traffic. It consolidates NSG Flow Logs records over a period of time (Data Collection Interval) which all have the same combination of the following fields:

• Source IP address
• Destination IP address
• Source port number
• Destination port number
• Layer 3 protocol

This information is provided per Virtual Network (Exporter). The Module also enriches them with Azure data not reported in NSG Flow Logs natively.

De-duplication: optionally the Module can report consolidated flows only from authoritative Virtual Network. Authoritative NSG is determined as follows. The Module sums up bytes, packets, and connections between two communicating peers over data collection interval reported by each Virtual Network. A Virtual Network with most connections (flows) for each consolidated flow is considered authoritative, and flows reported for the same two peers by all other Virtual Networks are discarded.

Parameters

Parameter Name	Description	Comments
Data Collection Interval, sec	Module logic execution interval	min = 5 sec, max = 1800 sec, default = 30 sec
N – number of reported hosts	The number of top hosts reported per Virtual Network	min = 0, max = 100000, default = 50 (0 indicates all hosts are reported)
Share of total traffic reported, %	Reported percent of total traffic per Virtual Network	e.g. 98 - indicates that reported consolidated flows consuming 98% of total NetFlow exporter traffic; min = 1%, max = 100%, default = 95%. Not more than N consolidated flows will be reported
Enable(1) or disable (0) reporting by authoritative exporters only	If set to 1 (de-duplication enabled), the Module reports flows only from authoritative Virtual Networks (exporters)	default = 0
Azure VM Instances	VMs with IPv4, IPv6, Subscription ID, Subscription Name, VM Name, NSG Name, Virtual Network Name, etc	Provided by EDF agent
Azure IPv4 Routes	IP range, source and destination Virtual Network hash	Provided by EDF agent
Azure IPv6 Routes	IP range, source and destination Virtual Network hash	Provided by EDF agent
Azure IPv4 Ranges	IPv4 ranges, Service name, Region	Provided by EDF agent
Azure IPv6 Ranges	IPv6 ranges, Service name, Region	Provided by EDF agent

Input

Azure NSG Flow Logs

Syslog/JSON Message Fields

Key	Field Description	Comments
nfc_id	Message type identifier	“nfc_id=20467”
exp_ip	Exporter Ipv4 address	IPv4 address (added for compatibility with other flows)
protocol	Transport Protocol (TCP = 6, UDP = 17)	number
direction	The direction of the traffic flow	string
decision	Whether traffic was allowed or denied	string, valid values are “A” for allowed and “D” for denied
src_ip	Source VM instance IPv4 address	IPv4 address
[src_ip6]	Source VM instance Ipv6 address	IPv6 address
[src_host]	Source host name	string, included when FQDN is on
[src_subs_id]	Source Subscription ID	string
[src_subs_name]	Source Subscription Name	string
[src_vm_name]	Source VM name	string
[src_nsg_name]	Source NSG name	string
[src_vnet_name]	Source Virtual Network name	string
[src_subnetwork_name]	Source Subnet name	string
[src_service]	Source Service	string
[src_region]	Source Region	string
[src_res_grp_name]	Source Resource Group Name	string
src_port	Source port number	number
dest_ip	Destination VM instance IPv4 address	IPv4 address
[dest_ip6]	Destination VM instance Ipv6 address	IPv6 address
[dest_host]	Destination host name	string, included when FQDN is on
[dest_subs_id]	Destination Subscription ID	string
[dest_subs_name]	Destination Subscription Name	string
[dest_vm_name]	Destination VM name	string
[dest_nsg_name]	Destination NSG name	string
[dest_vnet_name]	Destination Virtual Network name	string
[dest_subnetwork_name]	Destination Subnet name	string
[dest_service]	Destination Service	string
[dest_region]	Destination Region	string
[dest_res_grp_name]	Destination Resource Group Name	string
dest_port	Destination port number	number
packets_in	Total number of packets in the consolidated flows from the source to the destination	number
bytes_in	Total number of Layer 3 bytes in the packets of the consolidated flows from the source to the destination	number
packets_out	Total number of packets in the consolidated flows from the destination to the source	number
bytes_out	Total number of Layer 3 bytes in the packets of the consolidated flows from the destination to the source	number
flow_count	Number of consolidated flows	number
percent_of_total	Percent of Total (bytes)	decimal, e.g. 25.444% is 25.444
flow_start_time	Start time of the first consolidated flow	time
flow_end_time	End of the last consolidated flow	time
t_int	Observation time interval, msec	number