AWS Top Traffic Monitor (10267 / 20267)

Description

This Module identifies EC2 instances with the most traffic. It consolidates VPC Flow Logs records over a period of time (Data Collection Interval) which all have the same combination of the following fields:
    Source IP address
    Destination IP address
    Source port number
    Destination port number
    Layer 3 protocol
This information is provided per VPC ID. The Module also enriches them with AWS data not reported in VPC Flow Logs natively.
De-duplication: optionally the Module can report consolidated flows only from authoritative VPC. Authoritative VPC is determined as follows. The Module sums up bytes, packets, and connections between two EC2 instances over data collection interval reported by each VPC. A VPC with most connections (flows) for each consolidated flow is considered authoritative, and flows reported for the same two EC2 instances by all other VPCs are discarded.

Parameters

Parameter Name
Description
Comments
Data Collection Interval, sec
Module logic execution interval
min = 5 sec, max = 1800 sec, default = 30 sec
N – number of reported hosts
The number of top hosts reported per NetFlow exporter
min = 0, max = 100000, default = 50 (0 indicates all hosts are reported)
Share of total traffic reported, %
Reported percent of total traffic per VPC
e.g. 98 - indicates that reported consolidated flows consuming 98% of total NetFlow exporter traffic; min = 1%, max = 100%, default = 95%
Not more than N consolidated flows will be reported
Enable(1) or disable (0) reporting by authoritative exporters only
If set to 1 (de-duplication enabled), the Module reports flows only from authoritative VPCs (exporters)
default = 0
EC2 Instances
EC2 instances with IPs and VPC names and other information
Provided by EDF agent
VPC IPv4 Routes
AWS VPC IPv4 routes
Provided by EDF agent
VPC IPv6 Routes
AWS VPC IPv6 routes
Provided by EDF agent
AWS IPv4 Ranges
IPv4 ranges, AWS name, Region
Provided by EDF agent
AWS IPv6 Ranges
IPv6 ranges, AWS name, Region
Provided by EDF agent

Input

Amazon AWS Flow Logs ingested from CloudWatch or Kinesis stream or S3.

Syslog/JSON Message Fields

Key
Field Description
Comments
nfc_id
Message type identifier
“nfc_id=20267”
exp_ip
NetFlow exporter Ipv4 address
<IPv4 address> (added for compatibility with other flows)
[vpc_id]
VPC identifier
<string>
[vpc_name]
VPC name
<string>
interface_id
Interface ID
<string>
account_id
Account ID
<string>
protocol
Transport Protocol (TCP = 6, UDP = 17)
<number>
src_ip
Source EC2 instance IPv4 address
<IPv4 address>
[src_ip6]
Source EC2 instance Ipv6 address
<IPv6 address>
[src_host]
Source host name
<string>, included when FQDN is on
src_service
AWS Service
<string>, e.g "S3"
[src_ip_pub]
Source EC2 instance public IPv4 address
<IPv4 address>
[src_inst_id]
Source EC2 instance id
<string>, e.g i-390d7032 or i-0c0a6ac75d9d87b7e
[src_inst_name]
Source EC2 instance name
<string>
src_region
AWS Source Availability Zone (Region)
<string>
src_port
Source EC2 instance port number
<number>
dest_ip
Destination EC2 instance IPv4 address
<IPv4 address>
[dest_ip6]
Destination EC2 instance IPv6 address
<IPv6 address>
[dest_host]
Destination host name
<string>, included when FQDN is on
dest_service
AWS Service
<string>, e.g "S3"
[dest_ip_pub]
Destination EC2 instance public IPv4 address
<IPv4 address>
[dest_inst_id]
Destination EC2 instance id
<string>
[dest_inst_name]
Destination EC2 instance name
<string>
dest_region
AWS Source Availability Zone (Region)
<string>
dest_port
Destination EC2 instance port number
<number>
tcp_flag
TCP Flags
<string>, e.g. “SYN,ACK,FIN”
packets_in
Packets in the flow
<number>
bytes_in
Total number of Layer 3 bytes in the packets of the flow received
<number>
flow_count
Number of consolidated Flows
<number>
percent_of_total
Percent of Total (bytes)
<decimal>, e.g. 25.444% is 25.444
vpcflow_action
VPC Flow Action
<string>, “ACCEPTED” / ”REJECTED”
vpcflow_type
VPC Flow Type
<string>
subnet_id
Subnet ID
<string>
flow_start_time
Start time of the first consolidated flow
<time>
flow_end_time
End of the last consolidated flow
<time>
t_int
Observation time interval, msec
<number>
Last modified 2mo ago