GCP Top Traffic Monitor (10367 / 20367)
Description
This Module identifies GCP VM instances with the most traffic. It consolidates VPC Flow Logs records over a period of time (Data Collection Interval) which all have the same combination of the following fields:
- Source IP address
- Destination IP address
- Source port number
- Destination port number
- Layer 3 protocol
- Reporter
This information is provided per Subnetwork. The Module also enriches them with GCP data not reported in base VPC Flow Logs (when metadata fields are disabled).
De-duplication: optionally the Module can report consolidated flows only from authoritative Subnetwork. Authoritative Subnetwork is determined as follows. The Module sums up bytes, packets, and connections between two GCP VM instances over data collection interval reported by each Subnetwork. A Subnetwork with most connections (flows) for each consolidated flow is considered authoritative, and flows reported for the same two GCP VM instances by all other Subnetworks are discarded.
Parameters
| Parameter Name | Description | Comments |
|---|---|---|
| Data Reporting Interval, sec | Module logic execution interval | min = 5 sec, max = 1800 sec, default = 30 sec |
| N – number of reported hosts | The number of top hosts reported per NetFlow exporter | min = 0, max = 100000, default = 50 (0 indicates all hosts are reported) |
| Share of total traffic reported, % | Reported percent of total traffic per VPC | e.g. 98 - indicates that reported consolidated flows consuming 98% of total NetFlow exporter traffic; min = 1%, max = 100%, default = 95%. Not more than N consolidated flows will be reported |
| Enable(1) or disable (0) reporting by authoritative exporters only | If set to 1 (de-duplication enabled), the Module reports flows only from authoritative VPCs (exporters) | default = 0 |
| Compute Engine VM Instances | VMs with IPs, project ID, zone, name, and VPC names | Provided by EDF agent |
| Compute Engine IPv4 Routes | IP range, source and destination subnetwork IDs, Subnetwork name | Provided by EDF agent |
Input
GCP Flow Logs
Syslog/JSON Message Fields
| Key | Field Description | Comments |
|---|---|---|
| nfc_id | Message type identifier | “nfc_id=20367” |
| exp_ip | NetFlow exporter Ipv4 address | <IPv4 address> (added for compatibility with other flows) |
| reporter | GCP VPC Flow logs Reporter | <string>, "SRC" or "DEST" |
| gcp_exp | GCP VPC Flow logs Exporter. Calculated field based on reporter = SRC or DEST | <string>, "Project ID/VPC/Subnet" |
| protocol | Transport Protocol ( TCP = 6, UDP = 17) | <number> |
| src_ip | Source host IPv4 address | <IPv4 address> |
| [src_ip6] | Source host Ipv6 address | <IPv6 address> |
| [src_host] | Source host name | <string>, included when FQDN is on |
| [src_project_id] | Source Project ID | <string> |
| [src_vm_name] | Source VM name | <string> |
| [src_vm_zone] | Source VM Zone | <string> |
| [src_vpc_name] | Source VPC Name | <string> |
| src_port | Source port number | <number> |
| dest_ip | Destination host IPv4 address | <IPv4 address> |
| [dest_ip6] | Destination host IPv6 address | <IPv6 address> |
| [dest_host] | Destination host name | <string>, included when FQDN is on |
| [dest_project_id] | Destination Project ID | <string> |
| [dest_vm_name] | Destination VM name | <string> |
| [dest_vm_zone] | Destination VM Zone | <string> |
| [dest_vpc_name] | Destination VPC Name | <string> |
| dest_port | Destination EC2 instance port number | <number> |
| tcp_flag | TCP Flags | <string>, e.g. “SYN,ACK,FIN” |
| packets_in | Packets in the flow | <number> |
| bytes_in | Total number of Layer 3 bytes in the packets of the flow received | <number> |
| flow_count | Number of consolidated Flows | <number> |
| percent_of_total | Percent of Total (bytes) | <decimal>, e.g. 25.444% is 25.444 |
| subnet_id | Subnet ID | <string> |
| flow_start_time | Start time of the first consolidated flow | <time> |
| flow_end_time | End of the last consolidated flow | <time> |
| t_int | Observation time interval, msec | <number> |