GCP VPC Flow Logs Configuration

Prerequisites

Google Cloud Platform (GCP) generates VPC Flow logs for each subnetwork.

GCP has to be configured to send VPC Flow Logs to the EDFN agent. The agent uses GCP service account for retrieving enrichment data, such as VM names, subnetwork names, VPC names, etc., and for flow logs ingestion. VPC Flow logs are received using Pub/Sub service.

This section has instructions how to configure service account, enable flow logs and configure Pub/Sub subscription.

Configuration Steps

1. Configure GCP Service Account
2. Configure GCP VPC Flow logs
3. Configure EDFN Agent for ingestion of Google VPC Flow Logs
4. Enable Network Conversations Module (10062) for Google VPC Flow Logs reporting

Configure GCP Service Account

NetFlow Optimizer and EDFN support ingestion of GCP VPC Flow Logs using a service account.

For more information on GCP VPC Flow Logs, visit https://cloud.google.com/vpc/docs/using-flow-logs.

Service Account Configuration

There are two ways to configure EDFN Agent access to GCP: use a service account assigned to a Compute Engine instance, or provide a credentials file. In both cases you need to create a service account or use an existing one, and grant it the following IAM roles:

• Compute Network Viewer
• Pub/Sub Subscriber

Using Service Account Assigned to EDFN Compute Engine Instance

When EDFN is installed on Compute Engine instance, this method is preferred. This method does not require the credentials file. Make sure that appropriate IAM roles are granted to the service account and at least following Cloud API access scopes are set for the EDFN instance:

• Cloud Pub/Sub – Enabled
• Compute Engine – Read Only

To create the service account:

1. Open GCP console
2. In the navigation pane choose IAM & Admin -> Service Accounts (https://console.cloud.google.com/iam-admin/serviceaccounts), and then choose Create service account
3. Enter the service account name and then choose Create
4. Select the following roles: Compute Engine -> Compute Network Viewer, Pub/Sub -> Pub/Sub Subscriber. Choose Continue
5. Access Key is not needed for this authentication method, so don’t create any keys
6. Click Done

To assign the service account to the instance (https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes):

1. Navigate to the Compute Engine -> VM instances page: https://console.cloud.google.com/compute/instances
2. Click the VM instance name where EDFN is installed
3. Click the Stop button. Wait for the instance to be stopped
4. Click the Edit button
5. Scroll down to the Service Account section
6. From the dropdown menu, select the desired service account
7. In the Access scopes section set values: Cloud Pub/Sub – Enabled, Compute Engine – Read Only
8. Save changes

Providing Service Account Credentials

Use this option if your EDFN is installed on premises. This method requires an access key in the json credentials file.

To create a service account:

1. Open GCP console
2. In the navigation pane choose IAM & Admin -> Service Accounts (https://console.cloud.google.com/iam-admin/serviceaccounts), and then choose Create service account
3. Enter the service account name and then choose Create
4. Select following roles: Compute Engine -> Compute Network Viewer, Pub/Sub -> Pub/Sub Subscriber. Choose Continue
5. Click Create Key, leave JSON selected and click Create
6. Click Done
7. Copy json file with credentials to the EDFN installation server, the agent will use it for GCP services access
8. Change file permissions to read only for root user (if EDFN is running as root): chmod 400

Multiple Projects Configuration

If flow logs have to be collected from several projects, perform the following steps:

1. Open GCP console, select project to ingest VPC Flow Logs from
2. In the navigation pane choose IAM & Admin -> IAM (https://console.cloud.google.com/iam-admin/iam), and then choose Add
3. Enter the service account email address
4. Select the following Compute Engine -> Compute Network Viewer role
5. Click Save

Configure GCP VPC Flow Logs

VPC flow logs are configured on VPC network details page or on Subnetwork details page.

1. Open GCP console
2. In the navigation pane choose VPC Network -> VPC Networks (https://console.cloud.google.com/networking/networks/list), and then choose Network for which flow logs should be enabled
3. Choose one or several subnetworks, then on Flow Logs dropdown choose Configure
4. Configure Aggregation Interval, Include metadata and Sample rate
5. Click Save

VPC flow logs are stored in the Cloud Logging. You have to export logs to Pub/Sub: https://cloud.google.com/logging/docs/export. To export flow logs using Logs Viewer perform the following steps (for details visit https://cloud.google.com/logging/docs/export/configure_export_v2):

1. Open GCP console
2. In the navigation pane choose Logging -> Logs Router (https://console.cloud.google.com/logs/exports), and then choose Create sink
3. Select GCE Subnetwork in the first pull-down menu
4. Select compute.googleapis.com/vpc_flows in the second pull-down menu. Important, only vpc_flow logs must be selected!
5. On the right panel enter the sink name, select Pub/Sub as a destination service, then create a new Cloud Pub/Sub topic or use an existing one. Also logs can be exported to a Cloud Pub/Sub topic in another project
6. Click Create sink

When sink and Cloud Pub/Sub topic are created, you have to create a subscription. EDFN agent will receive flow logs using this subscription. Perform the following steps for the subscription creation:

1. In the navigation panel choose Pub/Sub -> Topics (https://console.cloud.google.com/cloudpubsub/topic/list), and then choose the topic where flow logs are exported
2. Select Create Subscription (simple subscription) in the Create subscription pull-down
   a. Enter subscription ID
   b. Delivery type: Pull
   c. Acknowledgement deadline: 60 seconds
   d. Retain acknowledged message: leave unchecked
   e. Click Create
3. When subscription is created, you’ll be navigated to the Subscription details page. Copy Subscription name for EDFN agent configuration, it looks like following template: projects/{projectID}/subscriptions/{subscriptionID}

In case of successful configuration, if the EDFN agent is not yet running, you will see the Unacked message count increasing on the subscription details page.

Configure EDFN Agent

When you click on Google VPC Flow Logs in NFO Input summary panel you will be presented with the following configuration screen.

On this screen you can configure the following parameters:

Cron Schedule

GCP VPC Flow Logs processing includes data enrichment with fields such as VM names, subnetwork names, VPC names, etc. This information is updated on cron schedule set here.

Credentials File Path

Set path to the service account credentials file here. Credentials file should be in the json format. If not provided and EDFN is installed on Compute Engine instance, the service assigned to the instance is used.

Project ID

Enter your main project ID here.

Subscription Name

Enter Pub/Sub subscription name or ID. Subscription name expected format is: projects/{projectID}/subscriptions/{subscriptionID}, where {projectID} may be same as Project ID from the previous parameter or it could be different.

Subscribers Count

Enter the desired number of parallel connections to the subscription.

Concurrency

Enter the number of parallel message processors per connection (subscriber).

Append Metadata

EDFN agent can append VM metadata to flow records processed by the Network Conversations Monitor (Module 10062). Possible values: 0 — metadata is not added; 1 — metadata is added if available.

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

Projects

Main project ID is automatically added after you press Run now button. Main project should not be deleted. When more than one Project is going to be monitored, additional project IDs may be added. Service account must have access (Compute Network Viewer, Compute Viewer) to all projects in the list. Also when VPC Flow Logs from undefined project is received, project ID is automatically added to the list. The list is updated according to the agent’s cron schedule interval (by default once per hour).

Verify GCP Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of VPC subnetworks which generates VPC Flow logs and associated Project ID, Region, and IP address range. This action will close the input configuration. It may take from several seconds to a few minutes to retrieve the list.

Open the IPFIX Exporters section to review and assign an exporter IP to each subnetwork. By default NFO will use the first IP from the range as an exporter IP. This IP will be reported as the exp_ip field in syslog events, for compatibility with physical network device flow reporting in visualizations and alerting.

You can change the exporter IP to better identify your subnetworks, especially if you have several subnetworks with same or overlapping IP address ranges.

Network Conversations Module Output Fields (Module 10062)

The table below maps native GCP VPC Flow Logs fields to their corresponding output field names in the Network Conversations Monitor (Module 10062). GCP VPC Flow Logs are structured as JSON log entries delivered via Pub/Sub — field names below reflect the jsonPayload structure. For the complete Module 10062 field reference, see Network Conversations Monitor.

GCP VPC Flow Logs Native Field	Module 10062 Output Field	Description
	nfc_id	Message type identifier (nfc_id=20062)
	flow_type	Type of flow (GCP)
	exp_ip	Subnetwork exporter IPv4 address, assigned per-subnetwork in the EDFN agent
	gcp_exp	GCP flow exporter identifier — calculated as Project ID/VPC/Subnet from the reporter field
reporter	gcp_reporter	Which endpoint logged the flow: SRC (source VM) or DEST (destination VM)
connection.src_ip	src_ip	Source IPv4 or IPv6 address
connection.src_port	src_port	Source port number
connection.dest_ip	dest_ip	Destination IPv4 or IPv6 address
connection.dest_port	dest_port	Destination port number
connection.protocol	protocol	IANA protocol number
bytes_sent	bytes_in	Bytes sent from the reporter's perspective
packets_sent	packets_in	Packets sent from the reporter's perspective
start_time	flow_start_time	Flow start time
end_time	flow_end_time	Flow end time
src_instance.vm_name	src_vm_name	Source VM instance name
src_instance.project_id	gcp_src_project_id	Source project ID
src_instance.zone	gcp_src_vm_zone	Source VM zone (e.g. us-central1-a)
src_instance.region	src_cloud_region	Source cloud region (e.g. us-central1)
dest_instance.vm_name	dest_vm_name	Destination VM instance name
dest_instance.project_id	gcp_dest_project_id	Destination project ID
dest_instance.zone	gcp_dest_vm_zone	Destination VM zone
dest_instance.region	dest_cloud_region	Destination cloud region
src_vpc.vpc_name	src_vpc_name	Source VPC name
src_vpc.subnetwork_name	src_subnet_name	Source subnet name
dest_vpc.vpc_name	dest_vpc_name	Destination VPC name
dest_vpc.subnetwork_name	dest_subnet_name	Destination subnet name
resource.labels.subnetwork_id	gcp_subnet_id	Source subnetwork ID
	src_cloud_service	Cloud source service (e.g. Compute Engine)
	dest_cloud_service	Cloud destination service
	src_asn	Source BGP Autonomous System Number
	src_cc	Source country code (GeoIP-derived, ISO 3166-1 alpha-2)
	src_region	Source region (GeoIP-derived, city level)
	src_city	Source city (GeoIP-derived)
	src_lon	Source longitude (GeoIP-derived)
	src_lat	Source latitude (GeoIP-derived)
	dest_asn	Destination BGP Autonomous System Number
	dest_cc	Destination country code (GeoIP-derived)
	dest_region	Destination region (GeoIP-derived, city level)
	dest_city	Destination city (GeoIP-derived)
	dest_lon	Destination longitude (GeoIP-derived)
	dest_lat	Destination latitude (GeoIP-derived)
	direction	Session origin: inbound, outbound, internal, or unknown
	state	Conversation lifecycle state: B (Begin), C (Continuing), E (End)
	duration	Conversation duration in seconds
	flow_count	Number of consolidated flows reported in this event
	t_int	Observation time interval (msec)
	threat_list_name	Name of the matched cybersecurity threat list
	reputation	Reputation score from the threat list
	idp	Identity provider for the user
	username	Username from identity enrichment (AD, Entra ID, Okta)
	app_id	Application ID
	app_name	Application name
	app_desc	Application description

The following native GCP VPC Flow Logs fields are not mapped by Module 10062:

GCP VPC Flow Logs Native Field	Notes
timestamp	Log entry timestamp (RFC3339) — not mapped (use flow_start_time / flow_end_time instead)
rtt_msec	TCP round-trip time reported by GCP — not mapped
src_location.continent	GCP-native continent enrichment — not mapped (NFO uses its own GeoIP enrichment)
src_location.country	GCP-native country enrichment — not mapped
src_location.region	GCP-native region enrichment — not mapped
src_location.city	GCP-native city enrichment — not mapped
dest_location.continent	GCP-native destination continent — not mapped
dest_location.country	GCP-native destination country — not mapped
dest_location.region	GCP-native destination region — not mapped
dest_location.city	GCP-native destination city — not mapped
logName	Cloud Logging log name — not mapped
resource.type	Always gce_subnetwork — not mapped
resource.labels.project_id	Resource label project ID — not mapped (use gcp_src_project_id / gcp_dest_project_id)
resource.labels.location	Resource label location — not mapped