Configuring AWS VPC Flow Logs Enrichment

Integrating NetFlow Optimizer with AWS for VPC flow logs enrichment enhances the value of flow data by providing additional context and insights into network traffic within AWS environments. By enriching VPC flow logs with information such as EC2 instance names, regions, and services, organizations can gain a deeper understanding of network activity and accelerate the investigation process. With this enriched data, security teams can quickly identify anomalous behavior, detect potential threats, and respond more effectively to security incidents. Ultimately, this integration streamlines operations, improves visibility, and enhances the overall security posture of AWS environments.

note

To configure integration with User Identity system, on the left navigation bar select Modules, open Network Conversations Monitor set by clicking on ..., and click on Module configuration 10062: Network Conversations Monitor. Scroll down to EDFN Agent Amazon VPC Flow Logs and click on it.

You will be presented with the following configuration screen.

Configure EDFN Agent

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO AWS VPC Flow Logs processing includes data enrichment with fields such as EC2 instance names, VPC names, Regions, Services, etc. This information is updated on cron schedule set here.

Assume Role Session Duration

Set this parameter is you use IAM Role method to access your AWS environment. The EDFN agent uses temporary credentials for AWS access. By default, the AWS temporary session has a one hour duration. If all delegated roles have a longer or shorter session duration, you can set this parameter value from 15 minutes to 12 hours.

KCL Metrics Level

Set this parameter (Kinesis Client Library metrics level) to SUMMARY or DETAILED to investigate Kinesis Stream processing problems. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Logs Request Interval

By default, VPC Flow Logs are requested every 30 seconds. Set this parameter to a frequency you want. Valid values are from 1 second to 1 day.

S3 Concurrency

This parameter is used when VPC Flow Logs are ingested from S3. It may be increased (default = 2), when S3 objects are processed too slowly. This can be verified in your AWS on queue monitoring panel (SQS console – select queue – choose Monitoring tab).

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

IPFIX Records Rate Limit

This is NFO internal parameter – maximum IPFIX records per second. Default value 0 (means unlimited). If you see NFO server dropped messages (NetFlow Optimizer -> Status), this parameter could be set to another value, for example, 10000 records per second.

Set Access to your AWS Accounts

There are two alternative ways to access your AWS accounts where VPC Flow Logs are configured.

1. AWS Credentials Tab

Use this method if you have a list of independent AWS accounts. The EDFN Agent expects that each account has only one profile. On "AWS Credentials" tab add a row for each account:

Access key ID and Secret access key are mandatory.

These credentials are available here: AWS console > IAM > Users > select User name > Security credentials tab. "Profile name" and "Default region" are optional.

The EDNF agent monitors all available regions to gather information about your EC2 instances and VPCs. To retrieve the account ID and list of available regions, the agent makes API calls using “Default region”. If it isn't provided, "us-east-1" is used.

2. IAM Role Tab

Use this method if you create IAM Role to delegate access across AWS accounts. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html for details.

Create a list of your ARNs and optionally External IDs as follows:

arn:aws:iam::999999999999:role/Role1,
arn:aws:iam::999999999998:role/Role1,
arn:aws:iam::999999999997:role/Role2,externalID-2

and enter it here:

info

Do not enter ARN assigned to NFO instance!

Verify AWS Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of accounts and associated VPCs (IPFIX exporters section), Kinesis Streams, CloudWatch Log groups, and S3 buckets (with associated queues).

Open the IPFIX Exporters section to review and assign an exporter IP to each VPC. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Enable VPC Flow Logs Input

Depending on your AWS VPC Flow Logs collection, you can enable one or more of the following configurations.

note

Please note that to avoid duplicate VPC Flow logs ingestion, the EDFN Agent for each VPC will use Kinesis Stream, if configured, before attempting to get logs from the CloudWatch Log group.

Kinesis Streams

Open the Kinesis Streams section to enable VPC Flow logs ingestion using Kinesis Streams. Set Enhanced Fan-Out option, if necessary.

To investigate Kinesis Stream processing problems, change the parameter “KCL metrics level” – Kinesis Client Library metrics level: valid values are NONE, SUMMARY and DETAILED. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Log Groups

Open the CloudWatch Log groups to enable VPC Flow logs ingestion using the CloudWatch API.

By default, VPC Flow Logs are requested every 30 seconds according to the “CloudWatchLogs request interval” parameter.

S3 and SQS

Open the “S3 and SQS” section to enable reading VPC Flow Logs from S3 using SQS messages notifications.

AWS Modules and Network Conversations Module Output Fields

The table below provides a side-by-side comparison of fields available in native VPC flow logs and NFO enriched output fields from various Modules.

VPC Flow Logs Native Fields	Enriched 20201 Output	Enriched 20267 Output	Enriched 20062 Output	Field Description
	nfc_id	nfc_id	nfc_id	Message type identifier
			flow_type	Type of flow
	exp_ip	exp_ip	exp_ip	Subnet virtual router ipv4 address
account-id	account_id	account_id	aws_account_id	AWS account ID
action	vpcflow_action	vpcflow_action	action	Accept or reject
bytes	bytes_in	bytes_in	bytes_in	Number of bytes recorded in the capture window.
dstaddr	dest_ip	dest_ip	dest_ip	Ipv4 or ipv6 of the destination.
dstport	dest_port	dest_port	dest_port	Port number of the destination.
end	flow_end_time	flow_end_time	flow_end_time	End time of the capture window in unix epoch seconds.
instance-id	aws_src_inst_id aws_dest_inst_id	aws_src_inst_id aws_dest_inst_id	aws_src_inst_id aws_dest_inst_id	Source or destination instance ID
interface-id	interface_id	interface_id	aws_interface_id	Interface ID
packets	packets_in	packets_in	packets_in	Number of packets recorded in the capture window.
pkt-dstaddr	dest_ip			The packet-level (origin) destination IP address.
pkt-srcaddr	src_ip			The packet-level (origin) source IP address.
protocol	protocol	protocol	protocol	Iana protocol number.
srcaddr	src_ip	src_ip	src_ip	Ipv4 or ipv6 address of the source.
srcport	src_port	src_port	src_port	Port number of the source
start	flow_start_time	flow_start_time	flow_start_time	Start time of the capture window in unix epoch seconds.
subnet-id	subnet_id	subnet_id		Subent ID of the interface (201 and 267 modules support this field, 62 – ignores)
tcp-flags	tcp_flag	tcp_flag	tcp_flag	TCP flags
type	vpcflow_type	vpcflow_type		Type of traffic: IPv4
vpc-id	vpc_id	vpc_id	aws_vpc_id	VPC ID
	src_inst_name	src_inst_name	src_vm_name	Name of the source instance
	dest_inst_name	dest_inst_name	dest_vm_name	Name of the destination instance
	src_ip_pub	src_ip_pub	src_ip_pub	Source EC2 instance public IPv4 address
	dest_ip_pub	dest_ip_pub	dest_ip_pub	Destination EC2 instance public IPv4 address
			src_region	Source GEO region
	src_region	src_region	src_cloud_region	Cloud source region
			dest_region	Destination GEO region
	dest_region	dest_region	dest_cloud_region	Cloud destination region
	src_service	src_service	src_cloud_service	Cloud source service
	dest_service	dest_service	dest_cloud_service	Cloud destination service
		bytes_out	bytes_out	Number of bytes in reply when client-server flows are stitched
		packets_out	packets_out	Number of packets in reply when client-server flows are stitched
			exp_name	Name of vpc flow exporter
			direction	Direction of originating flow