Skip to main content
Version: 2.10.2

Configuring AWS VPC Flow Logs Enrichment

Integrating NetFlow Optimizer with AWS for VPC flow logs enrichment enhances the value of flow data by providing additional context and insights into network traffic within AWS environments. By enriching VPC flow logs with information such as EC2 instance names, regions, and services, organizations can gain a deeper understanding of network activity and accelerate the investigation process. With this enriched data, security teams can quickly identify anomalous behavior, detect potential threats, and respond more effectively to security incidents. Ultimately, this integration streamlines operations, improves visibility, and enhances the overall security posture of AWS environments.

note

To configure integration with User Identity system, on the left navigation bar select Modules, open Network Conversations Monitor set by clicking on ..., and click on Module configuration 10062: Network Conversations Monitor. Scroll down to EDFN Agent Amazon VPC Flow Logs and click on it.

You will be presented with the following configuration screen.

Configure EDFN Agent

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO AWS VPC Flow Logs processing includes data enrichment with fields such as EC2 instance names, VPC names, Regions, Services, etc. This information is updated on cron schedule set here.

Assume Role Session Duration

Set this parameter is you use IAM Role method to access your AWS environment. The EDFN agent uses temporary credentials for AWS access. By default, the AWS temporary session has a one hour duration. If all delegated roles have a longer or shorter session duration, you can set this parameter value from 15 minutes to 12 hours.

KCL Metrics Level

Set this parameter (Kinesis Client Library metrics level) to SUMMARY or DETAILED to investigate Kinesis Stream processing problems. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Logs Request Interval

By default, VPC Flow Logs are requested every 30 seconds. Set this parameter to a frequency you want. Valid values are from 1 second to 1 day.

S3 Concurrency

This parameter is used when VPC Flow Logs are ingested from S3. It may be increased (default = 2), when S3 objects are processed too slowly. This can be verified in your AWS on queue monitoring panel (SQS console – select queue – choose Monitoring tab).

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

IPFIX Records Rate Limit

This is NFO internal parameter – maximum IPFIX records per second. Default value 0 (means unlimited). If you see NFO server dropped messages (NetFlow Optimizer -> Status), this parameter could be set to another value, for example, 10000 records per second.

Set Access to your AWS Accounts

There are two alternative ways to access your AWS accounts where VPC Flow Logs are configured.

1. AWS Credentials Tab

Use this method if you have a list of independent AWS accounts. The EDFN Agent expects that each account has only one profile. On "AWS Credentials" tab add a row for each account:

Access key ID and Secret access key are mandatory.

These credentials are available here: AWS console > IAM > Users > select User name > Security credentials tab. "Profile name" and "Default region" are optional.

The EDNF agent monitors all available regions to gather information about your EC2 instances and VPCs. To retrieve the account ID and list of available regions, the agent makes API calls using “Default region”. If it isn't provided, "us-east-1" is used.

2. IAM Role Tab

Use this method if you create IAM Role to delegate access across AWS accounts. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html for details.

Create a list of your ARNs and optionally External IDs as follows:

arn:aws:iam::999999999999:role/Role1,
arn:aws:iam::999999999998:role/Role1,
arn:aws:iam::999999999997:role/Role2,externalID-2

and enter it here:

info

Do not enter ARN assigned to NFO instance!

Verify AWS Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of accounts and associated VPCs (IPFIX exporters section), Kinesis Streams, CloudWatch Log groups, and S3 buckets (with associated queues).

Open the IPFIX Exporters section to review and assign an exporter IP to each VPC. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Enable VPC Flow Logs Input

Depending on your AWS VPC Flow Logs collection, you can enable one or more of the following configurations.

note

Please note that to avoid duplicate VPC Flow logs ingestion, the EDFN Agent for each VPC will use Kinesis Stream, if configured, before attempting to get logs from the CloudWatch Log group.

Kinesis Streams

Open the Kinesis Streams section to enable VPC Flow logs ingestion using Kinesis Streams. Set Enhanced Fan-Out option, if necessary.

To investigate Kinesis Stream processing problems, change the parameter “KCL metrics level” – Kinesis Client Library metrics level: valid values are NONE, SUMMARY and DETAILED. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Log Groups

Open the CloudWatch Log groups to enable VPC Flow logs ingestion using the CloudWatch API.

By default, VPC Flow Logs are requested every 30 seconds according to the “CloudWatchLogs request interval” parameter.

S3 and SQS

Open the “S3 and SQS” section to enable reading VPC Flow Logs from S3 using SQS messages notifications.

AWS Modules and Network Conversations Module Output Fields

The table below provides a side-by-side comparison of fields available in native VPC flow logs and NFO enriched output fields from various Modules.

VPC Flow Logs Native FieldsEnriched 20201 OutputEnriched 20267 OutputEnriched 20062 OutputField Description
nfc_idnfc_idnfc_idMessage type identifier
flow_typeType of flow
exp_ipexp_ipexp_ipSubnet virtual router ipv4 address
account-idaccount_idaccount_idaws_account_idAWS account ID
actionvpcflow_actionvpcflow_actionactionAccept or reject
bytesbytes_inbytes_inbytes_inNumber of bytes recorded in the capture window.
dstaddrdest_ipdest_ipdest_ipIpv4 or ipv6 of the destination.
dstportdest_portdest_portdest_portPort number of the destination.
endflow_end_timeflow_end_timeflow_end_timeEnd time of the capture window in unix epoch seconds.
instance-idaws_src_inst_id aws_dest_inst_idaws_src_inst_id aws_dest_inst_idaws_src_inst_id aws_dest_inst_idSource or destination instance ID
interface-idinterface_idinterface_idaws_interface_idInterface ID
packetspackets_inpackets_inpackets_inNumber of packets recorded in the capture window.
pkt-dstaddrdest_ipThe packet-level (origin) destination IP address.
pkt-srcaddrsrc_ipThe packet-level (origin) source IP address.
protocolprotocolprotocolprotocolIana protocol number.
srcaddrsrc_ipsrc_ipsrc_ipIpv4 or ipv6 address of the source.
srcportsrc_portsrc_portsrc_portPort number of the source
startflow_start_timeflow_start_timeflow_start_timeStart time of the capture window in unix epoch seconds.
subnet-idsubnet_idsubnet_idSubent ID of the interface (201 and 267 modules support this field, 62 – ignores)
tcp-flagstcp_flagtcp_flagtcp_flagTCP flags
typevpcflow_typevpcflow_typeType of traffic: IPv4
vpc-idvpc_idvpc_idaws_vpc_idVPC ID
src_inst_namesrc_inst_namesrc_vm_nameName of the source instance
dest_inst_namedest_inst_namedest_vm_nameName of the destination instance
src_ip_pubsrc_ip_pubsrc_ip_pubSource EC2 instance public IPv4 address
dest_ip_pubdest_ip_pubdest_ip_pubDestination EC2 instance public IPv4 address
src_regionSource GEO region
src_regionsrc_regionsrc_cloud_regionCloud source region
dest_regionDestination GEO region
dest_regiondest_regiondest_cloud_regionCloud destination region
src_servicesrc_servicesrc_cloud_serviceCloud source service
dest_servicedest_servicedest_cloud_serviceCloud destination service
bytes_outbytes_outNumber of bytes in reply when client-server flows are stitched
packets_outpackets_outNumber of packets in reply when client-server flows are stitched
exp_nameName of vpc flow exporter
directionDirection of originating flow