AWS Top Traffic Monitor (10267 / 20267)
Description
This Module identifies EC2 instances with the most traffic. It consolidates VPC Flow Logs records over a period of time (Data Collection Interval) which all have the same combination of the following fields:
- Source IP address
- Destination IP address
- Source port number
- Destination port number
- Layer 3 protocol
This information is provided per VPC ID. The Module also enriches them with AWS data not reported in VPC Flow Logs natively.
De-duplication: optionally the Module can report consolidated flows only from authoritative VPC. Authoritative VPC is determined as follows. The Module sums up bytes, packets, and connections between two EC2 instances over data collection interval reported by each VPC. A VPC with most connections (flows) for each consolidated flow is considered authoritative, and flows reported for the same two EC2 instances by all other VPCs are discarded.
Parameters
| Parameter Name | Description | Comments |
|---|---|---|
| Data Collection Interval, sec | Module logic execution interval | min = 5 sec, max = 1800 sec, default = 30 sec |
| N – number of reported hosts | The number of top hosts reported per NetFlow exporter | min = 0, max = 100000, default = 50 (0 indicates all hosts are reported) |
| Share of total traffic reported, % | Reported percent of total traffic per VPC | e.g. 98 - indicates that reported consolidated flows consuming 98% of total NetFlow exporter traffic; min = 1%, max = 100%, default = 95%. Not more than N consolidated flows will be reported |
| Enable(1) or disable (0) reporting by authoritative exporters only | If set to 1 (de-duplication enabled), the Module reports flows only from authoritative VPCs (exporters) | default = 0 |
| EC2 Instances | EC2 instances with IPs and VPC names and other information | Provided by EDF agent |
| VPC IPv4 Routes | AWS VPC IPv4 routes | Provided by EDF agent |
| VPC IPv6 Routes | AWS VPC IPv6 routes | Provided by EDF agent |
| AWS IPv4 Ranges | IPv4 ranges, AWS name, Region | Provided by EDF agent |
| AWS IPv6 Ranges | IPv6 ranges, AWS name, Region | Provided by EDF agent |
Input
Amazon AWS Flow Logs ingested from CloudWatch or Kinesis stream or S3.
Syslog/JSON Message Fields
| Key | Field Description | Comments |
|---|---|---|
| nfc_id | Message type identifier | “nfc_id=20267” |
| exp_ip | NetFlow exporter Ipv4 address | <IPv4 address> (added for compatibility with other flows) |
| [vpc_id] | VPC identifier | <string> |
| [vpc_name] | VPC name | <string> |
| interface_id | Interface ID | <string> |
| account_id | Account ID | <string> |
| protocol | Transport Protocol (TCP = 6, UDP = 17) | <number> |
| src_ip | Source EC2 instance IPv4 address | <IPv4 address> |
| [src_ip6] | Source EC2 instance Ipv6 address | <IPv6 address> |
| [src_host] | Source host name | <string>, included when FQDN is on |
| src_service | AWS Service | <string>, e.g "S3" |
| [src_ip_pub] | Source EC2 instance public IPv4 address | <IPv4 address> |
| [src_inst_id] | Source EC2 instance id | <string>, e.g i-390d7032 or i-0c0a6ac75d9d87b7e |
| [src_inst_name] | Source EC2 instance name | <string> |
| src_region | AWS Source Availability Zone (Region) | <string> |
| src_port | Source EC2 instance port number | <number> |
| dest_ip | Destination EC2 instance IPv4 address | <IPv4 address> |
| [dest_ip6] | Destination EC2 instance IPv6 address | <IPv6 address> |
| [dest_host] | Destination host name | <string>, included when FQDN is on |
| dest_service | AWS Service | <string>, e.g "S3" |
| [dest_ip_pub] | Destination EC2 instance public IPv4 address | <IPv4 address> |
| [dest_inst_id] | Destination EC2 instance id | <string> |
| [dest_inst_name] | Destination EC2 instance name | <string> |
| dest_region | AWS Source Availability Zone (Region) | <string> |
| dest_port | Destination EC2 instance port number | <number> |
| tcp_flag | TCP Flags | <string>, e.g. “SYN,ACK,FIN” |
| packets_in | Packets in the flow | <number> |
| bytes_in | Total number of Layer 3 bytes in the packets of the flow received | <number> |
| flow_count | Number of consolidated Flows | <number> |
| percent_of_total | Percent of Total (bytes) | <decimal>, e.g. 25.444% is 25.444 |
| vpcflow_action | VPC Flow Action | <string>, “ACCEPTED” / ”REJECTED” |
| vpcflow_type | VPC Flow Type | <string> |
| subnet_id | Subnet ID | <string> |
| flow_start_time | Start time of the first consolidated flow | <time> |
| flow_end_time | End of the last consolidated flow | <time> |
| t_int | Observation time interval, msec | <number> |