AWS VPC Flow Logs Input Configuration

General Settings

When you click on Amazon VPC Flow Logs in NFO Input summary panel you will be presented with the the following configuration screen.

On this screen you can configure the following parameters:

Cron Schedule

NFO AWS VPC Flow Logs processing includes data enrichment with fields such as EC2 instance names, VPC names, Regions, Services, etc. This information is updated on cron schedule set here.

AWS Credentials File

There are two alternative ways to access your AWS accounts where VPC Flow Logs are configured: AWS Credentials and IAM Roles. Set path to AWS Credentials file, if you use this method to access your AWS environments. If you use IAM Roles, leave it blank.

Assume Role Session Duration

Set this parameter is you use IAM Role method to access your AWS environment. The EDFN agent uses temporary credentials for AWS access. By default, the AWS temporary session has a one hour duration. If all delegated roles have a longer or shorter session duration, you can set this parameter value from 15 minutes to 12 hours.

KCL metrics level

Set this parameter (Kinesis Client Library metrics level) to SUMMARY or DETAILED to investigate Kinesis Stream processing problems. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Logs request interval

By default, VPC Flow Logs are requested every 30 seconds according. Set this parameter to a frequency you want. Valid values are from 1 second to 1 day.

S3 concurrency

This parameter is used when VPC Flow Logs are ingested from S3. It may be increased (default = 2), when S3 objects are processed too slowly. This can be verified in your AWS on queue monitoring panel (SQS console – select queue – choose Monitoring tab).

Max IPFIX packet size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

IPFIX records rate limit

This is NFO internal parameter – maximum IPFIX records per second. Default value 0 (means unlimited). If you see NFO server dropped messages (NetFlow Optimizer -> Status), this parameter could be set to another value, for example, 10000 records per second.

Set Access to your AWS Accounts

There are two alternative ways to access your AWS accounts where VPC Flow Logs are configured.

1. AWS Credentials

Use this method if you have a list of independent AWS accounts. Create an AWS credentials file, e.g. credentials. It should be placed on the machine where EDFN is installed. Use the IAM User public and secret access key to create a file as follows:

[account_1]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
..........
[account_N]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

Change file permissions to read only for root user (if EDFN is running as root): chmod 400 credentials. The Agent reads the file and takes all profiles from it. The Agent expects that each account has only one profile.

Set path to this file, for example: /root/.aws/credentials

2. IAM Role

Use this method if you create IAM Role to delegate access across AWS accounts. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html for details.

Create a list of your ARNs and optionally External IDs as follows:

arn:aws:iam::999999999999:role/Role1,
arn:aws:iam::999999999998:role/Role1,
arn:aws:iam::999999999997:role/Role2,externalID-2

and enter it here (NOTE: do not enter ARN assigned to NFO instance):

AWS Region Configuration

When the EDFN is installed on an EC2 Instance, you can skip this section.

The EDNF agent monitors all available regions to gather information about your EC2 instances and VPCs. To retrieve the account ID and list of available regions, the agent makes API calls using “default” region.

When EDNF is installed outside of Amazon, the default region must be configured!

Set the AWS Region in the AWS config file on your local system, located at:

  • ~/.aws/config on Linux, macOS, or Unix

  • C:\Users\USERNAME\.aws\config on Windows

This file should contain lines in the following format:

[default]
region = your_aws_region

Substitute your desired AWS Region (for example, “us-west-2”) for your_aws_region.

Verify AWS Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of accounts and associated VPCs (IPFIX exporters section), Kinesis Streams, CloudWatch Log groups, and S3 buckets (with associated queues).

Open the IPFIX Exporters section to review and assign an exporter IP to each VPC. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Enable VPC Flow Logs Input

Depending on your AWS VPC Flow Logs collection, you can enable one or more of the following configurations.

Please note that to avoid duplicate VPC Flow logs ingestion, the EDFN Agent for each VPC will use Kinesis Stream, if configured, before attempting to get logs from the CloudWatch Log group.

Kinesis Streams

Open the Kinesis Streams section to enable VPC Flow logs ingestion using Kinesis Streams. Set Enhanced Fan-Out option, if necessary.

To investigate Kinesis Stream processing problems, change the parameter “KCL metrics level” – Kinesis Client Library metrics level: valid values are NONE, SUMMARY and DETAILED. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Log Groups

Open the CloudWatch Log groups to enable VPC Flow logs ingestion using the CloudWatch API.

By default, VPC Flow Logs are requested every 30 seconds according to the “CloudWatchLogs request interval” parameter.

S3 and SQS

Open the “S3 and SQS” section to enable reading VPC Flow Logs from S3 using SQS messages notifications.