Skip to main content
Version: Next

AWS VPC Flow Logs Configuration

Prerequisites

AWS VPC Flow Logs are generated by an ENI (network interface) in your VPC, whether it is attached to an EC2 instance, a service, or a task in AWS Fargate.

NFO and EDFN support ingestion of VPC Flow Logs from both S3 and CloudWatch. A single NFO instance can ingest VPC Flow Logs from S3 for certain VPCs while simultaneously receiving VPC Flow Logs from CloudWatch for other VPCs in your AWS environment.

important

There are two methods to authenticate EDFN with AWS:

  • Using IAM Roles: Use this option if your NFO/EDFN is installed on an EC2 instance.
  • Using AWS User: Use this option if your NFO/EDFN is installed on-premises or outside of AWS.

Configuration Steps

  1. Configure AWS VPC Flow Logs
  2. Configure AWS User and Policy
  3. Configure EDFN Agent for ingestion of AWS VPC Flow Logs
  4. Enable Network Conversations Module (10062) for AWS VPC Flow Logs reporting

Configure AWS VPC Flow Logs

To publish VPC Flow Logs to S3 visit: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html.

To publish VPC Flow Logs to CloudWatch Logs visit: https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html.

Kinesis Stream Configuration

When VPC Flow logs are stored in the CloudWatch Logs, NFO can receive them using CWL API, or, subscribe to a Kinesis stream.

There are two ways to configure logs publishing to a Kinesis stream:

  • use CloudFormation template or
  • manually create required resources using Amazon console and CLI

CloudFormation

A CloudWatch Logs group and Kinesis stream can be created per region using a CloudFormation template. Download the template from:

https://nfo-ami.s3.us-west-2.amazonaws.com/2.10.0/NFOCwlKinesisSubscription.template

Open the Amazon CloudFormation console and create a stack using these instructions.

During the stack creation, the following parameters may be changed:

  • CloudWatch Logs group: Logs retention period in days (default 30 days)
  • Kinesis stream: Data retention period in hours (default 24 hours, min 24 hours, max 168 hours)
  • Kinesis stream: Number of shards (default 1)

The CloudFormation template automatically creates the following resources:

  • NFLVpcFlowLogRole_<stack_name>_<region_id> – the IAM role for publishing Flow Logs to CloudWatch Logs.
  • NFLCwlToKinesisRole_<stack_name>_<region_id> – the IAM role for publishing Flow logs from CloudWatch Logs group to a Kinesis Stream.
  • NFLVpcFlowLogGroup_<stack_name>_<region_id> – the CloudWatch Logs group for Flow Logs.
  • NFLVpcKinesisStream_<stack_name>_<region_id> – Kinesis Stream.
  • NFLVpcFlowLogsSubscriptionFilter – the CloudWatch Logs group NFLVpcFlowLogGroup subscription filter. All Flow Logs are published to the Kinesis Stream NFLVpcKinesisStream.

Where <stack_name> is substituted as a CloudFormation stack name and <region_id> as a current region ID.

After that you only have to create VPC Flow Logs for your VPCs, subnets, or network interfaces (see https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-cwl-create-flow-log for details).

Select NFLVpcFlowLogGroup_<stack_name>_<region_id> as the destination and NFLVpcFlowLogRole_<stack_name>_<region_id> as the IAM role.

Amazon Console

If the CloudFormation template isn’t suitable for you, all these resources can be created using the Amazon console or CLI. For example, you may want to collect Flow Logs from several Amazon accounts in one Kinesis Stream and so on.

  1. Create an IAM role for publishing Flow Logs to CloudWatch Logs (see https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-iam for details).

  2. Create a Flow Log that publishes to CloudWatch Logs (see https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-cwl.html#flow-logs-cwl-create-flow-log for details).

  3. Configure Kinesis Stream.

    The EDFN agent is working as a Kinesis steam consumer, and it can be with or without Enhanced Fan-Out, see Reading Data from Amazon Kinesis Data Streams https://docs.aws.amazon.com/streams/latest/dev/building-consumers.html for details.

    If you decide not to use Enhanced fan-out, you may deregister the consumer EDFN_KINESIS_VPC_FLOW_LOG_CONSUMER manually. To do this, please navigate to AWS console - Services - Kinesis - Data Streams, then select your stream(s), open the Enhanced fan-out tab, and deregister EDFN_* consumer.

    Also, if you decide to use CloudWatch Logs API instead of Kinesis stream consumer, you may remove unused streams too.

  4. Create CloudWatch Logs Kinesis subscription filter. After that CloudWatch log group will publish VPC Flow Logs to the Kinesis stream.

    The subscription can be created using CLI (see Subscription Filters with Kinesis https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#DestinationKinesisExample for details). On step 7 the filter-pattern parameter may be left "" (empty string). To create the subscription using AWS console:

    1. Create IAM Role with the following policy:
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Action": "kinesis:PutRecord",
    "Resource": "kinesis-stream-arn",
    "Effect": "Allow"
    },
    {
    "Action": "iam:PassRole",
    "Resource": "iam-role-arn",
    "Effect": "Allow"
    }
    ]
    }
    1. Navigate CloudWatch Log groups and select log group for streaming.
    2. Open "Subscription filters" tab and select "Create Kinesis subscription filter".
    3. On the opened form enter Kinesis stream, IAM Role, filter name and you may select Log format "Amazon VPC Flow Logs"
    4. Click "Start streaming" button
note

Enhanced fan-out consumers may result in additional charges from AWS.

S3 and SQS Configuration

VPC Flow logs may be sent to an S3 bucket. For more information visit Publishing Flow Logs to Amazon S3 https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html. The EDFN agent doesn’t monitor the S3 bucket for new object creation directly. It assumes that the S3 bucket sends “s3:ObjectCreated:*” notifications to the Simple Queue Service (SQS).

There are two ways to configure S3 bucket and SQS:

  • use CloudFormation template
  • manually create required resources using Amazon console or CLI

CloudFormation

All required resources may be created using Amazon CloudFormation template. The CloudFormation template is available here: https://nfo-ami.s3.us-west-2.amazonaws.com/2.10.0/nfo_aws_s3_sqs_template.json

Create a stack using the template. Open Amazon CloudFormation console and create the stack following these instructions: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stack.html

  1. Navigate to CloudFormation, select Stacks in the left panel.
  2. Click “Create stack” and select “With new resources (standard).
  3. Enter Amazon S3 URL or upload the template.
  4. Enter the stack name.
  5. Enter custom parameters:
    1. Policy name – IAM policy name for EDFN agent, this is the optional parameter and may be left empty. For example, when S3/SQS is configured in several regions you need to create a single policy.
    2. Role name – IAM role name for EC2 instance where NFO/EDFN is installed. This is also the optional parameter and may be left empty.
    3. S3 bucket name – name of the bucket to which VPC flow logs will be published.
    4. SQS name – name of the queue to which bucket object creation events will be published.
  6. Click next.
  7. Validate options on the opened page and then click next.
  8. Review and submit.

After these steps CloudFormation creates from 2 to 4 resources: S3 bucket, SQS queue, IAM policy, and IAM role. IAM policy and role have deletion policy RetainExceptOnCreate, so please remove them manually when you will not need them anymore. S3 bucket also cannot be removed if it is not empty.

Finally, you only have to create VPC Flow Logs for your VPCs, subnets, or network interfaces (see https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html for details) and enter S3 bucket ARN as the destination. ARN can be copied from the bucket Properties page or it is also available on the Output tab of the stack.

Amazon Console

If the CloudFormation template doesn't work for you, you have the option to create all these resources using the Amazon console or CLI.

note

EDFN agent expects that the queue account has access to “s3:GetObject” action. The S3 bucket region and SQS region must be the same.

To configure SQS:

  1. Sign in to the Amazon SQS console https://console.aws.amazon.com/sqs/.
  2. Change the region to the S3 bucket’s region.
  3. Choose Create New Queue.
  4. Enter the Queue name.
  5. Leave Standard queue selected.
  6. Choose Configure Queue.
  7. Change Receive Message Wait Time from 0 to 20 seconds.
  8. Choose Create Queue.

When the queue is created, add permission to the queue allows the S3 bucket to send messages to it:

{
"Version": "2012-10-17",
"Id": "QUEUE-PERMISSION-POLICY-ID",
"Statement": [
{
"Sid": "STATEMENT-ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SQS:SendMessage"
],
"Resource": "arn:aws:sqs:REGION:ACCOUNT-ID:QUEUE-NAME",
"Condition": {
"ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:BUCKET-NAME" }
}
}
]
}

When the queue permissions are configured, add S3 bucket notifications:

  1. Sign in to the Amazon S3 console https://console.aws.amazon.com/s3/.
  2. Select the bucket and choose Properties.
  3. Choose events and press Add notification.
  4. Enter the event name.
  5. In the Events section, select All object create events.
  6. Optionally enter prefix and suffix. For example, suffix may be “.log.gz”.
  7. Choose the SQS destination and select your queue name or enter the queue ARN. The SQS and bucket region must be the same.
  8. Choose Save.

Configure AWS User and Policy

You have two choices to configure EDFN Agent to access AWS and ingest VPC Flow Logs: using IAM Role or using AWS account. In both cases you need to create an AWS Permissions Policy - see below.

Using IAM Roles

Use this option if your NFO/EDFN is installed on an EC2 instance. Make sure that an appropriate IAM role is attached to the EC2 instance running EDFN.

note

The AWS Credentials tab must be empty when using IAM role authentication.

Using AWS User

Use this option if your NFO/EDFN is installed on premises or outside of your AWS cloud. This method requires an access key, a secret access key and optionally default region.

AWS Permissions Policy

Create an IAM Policy to allow the EDFN Agent to access AWS resources. The policy can be created automatically when the CloudFormation template is used for S3/SQS configuration. To create such a policy, open the IAM console, navigate to the Policies page and press the Create Policy button. Then select the JSON tab and insert the following policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EdfnMinPermissions",
"Effect": "Allow",
"Action": [
"ec2:DescribeRegions",
"ec2:DescribeInstances",
"ec2:DescribeFlowLogs",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeVpcs",
"ec2:DescribeNatGateways",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeTransitGatewayAttachments",
"ec2:SearchTransitGatewayRoutes",
"logs:DescribeSubscriptionFilters",
"logs:GetLogEvents",
"logs:DescribeLogStreams"
],
"Resource": "*"
},
{
"Sid": "S3Permissions",
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetBucketNotification",
"s3:GetObject",
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ChangeMessageVisibility*",
"sqs:DeleteMessage*",
"sqs:ReceiveMessage"
],
"Resource": "*"
},
{
"Sid": "KCLPermissions",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"dynamodb:CreateTable",
"dynamodb:DescribeTable",
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:DeleteItem",
"dynamodb:UpdateItem",
"dynamodb:Scan",
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:ListShards",
"kinesis:ListStreamConsumers",
"kinesis:SubscribeToShard",
"kinesis:DescribeStream*",
"kinesis:RegisterStreamConsumer"
],
"Resource": "*"
}
]
}

IAM Roles Configuration

When EDFN is installed on the EC2 instance, AWS access for IAM role configuration (the role can be created automatically when the CloudFormation template is used for S3/SQS configuration) can be configured using the following steps:

  1. Open the IAM console https://console.aws.amazon.com/iam/home.
  2. In the navigation pane, choose Roles, then Create New Role.
  3. On the Select Role Type page, under AWS Service Roles, choose Amazon EC2.
  4. On the Permissions page, choose the already created policy EdfnAgentPolicy from the policy list or create a new one with permissions from the previous section, then choose Next Step.
  5. Enter a name for the role, and then select Next Step.
  6. On the Review page, choose Create Role.
  7. Launch an Amazon EC2 instance with the IAM role that you’ve created previously:
    1. Open the Amazon EC2 console
    2. Go to the Instances page
    3. Select the instance, then Actions, Instance Settings, and Attach/Replace IAM Role.

NFO may collect flow logs from different accounts using the “Switch Role” action. For detailed information, visit Delegate Access Across AWS Accounts Using IAM Roles https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html. The EC2 Instance assumes the IAM role from another account and thus has access to resources in this account. The assumed role must have permissions from the EdfnAgentPolicy policy.

For example, if your NFO is installed in the account 111111111111 and the second account is 999999999999, access delegation may be configured using the following steps for account 999999999999:

  1. Open the IAM console: https://console.aws.amazon.com/iam/home
  2. In the navigation pane on the left, choose Roles and then choose Create role.
  3. Choose the Another AWS account role type.
  4. For Account ID, type first the account ID: 111111111111
  5. Optionally type External ID (must be unique across assumed accounts) or leave it blank.
  6. Choose Next: Permissions, to set the permissions that will be associated with the role.
  7. Select EdfnAgentPolicy.
  8. Then choose Next: Tags.
  9. Choose Next: Review and type the role name.
  10. After reviewing the role, choose Create role.
  11. Open the just created role and copy the role ARN.

After that, configure the EC2 Instance Role permission, for account 111111111111 by doing the following steps:

  1. Open the IAM console: https://console.aws.amazon.com/iam/home

  2. In the navigation pane on the left, choose Roles and then choose your EC2 Instance role.

  3. Choose the Permissions tab and click + Add inline policy.

  4. Insert the following JSON policy (don’t forget to replace role ARN):

  5. Choose Review policy and then type the policy name.

    {
    "Version": "2012-10-17",
    "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::999999999999:role/YourRoleName"
    }
    }
  6. After reviewing the policy, choose Create policy.

    After that, the EC2 instance role in account 111111111111 has access to resources in account 999999999999. Add the assumed role ARN, and optionally the external ID, to the EDFN agent's IAM Roles configuration list.

IAM User Configuration

Use this method if your AWS environment is accessed using an IAM user. Perform the following steps:

  1. Open AWS console
  2. Navigate to Identity and Access Management (IAM)
  3. Click Users in the left panel
  4. Click Create user
  5. Enter user name, for instance, edfn-agent
  6. Select Attach policies directly
  7. Find and select policy created in the section AWS Permissions Policy
  8. Click next to review and finally save the user

Then you need to open the user and generate secret access key for NFO API requests. Please perform the following steps:

  1. Open created user edfn-agent
  2. Click Security credentials tab
  3. On Access keys panel click Create access key
  4. Select Other use case and click next
  5. Press Create access key
  6. Copy access key and Secret access key, you need these keys for AWS input configuration (via EDFN Agent)

Configure EDFN Agent

When you click on Amazon VPC Flow Logs in NFO Input summary panel you will be presented with the following configuration screen.

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO AWS VPC Flow Logs processing includes data enrichment with fields such as EC2 instance names, VPC names, Regions, Services, etc. This information is updated on cron schedule set here.

Assume Role Session Duration

Set this parameter is you use IAM Role method to access your AWS environment. The EDFN agent uses temporary credentials for AWS access. By default, the AWS temporary session has a one hour duration. If all delegated roles have a longer or shorter session duration, you can set this parameter value from 15 minutes to 12 hours.

KCL Metrics Level

Set this parameter (Kinesis Client Library metrics level) to SUMMARY or DETAILED to investigate Kinesis Stream processing problems. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Logs Request Interval

By default, VPC Flow Logs are requested every 30 seconds. Set this parameter to a frequency you want. Valid values are from 1 second to 1 day.

S3 Concurrency

This parameter is used when VPC Flow Logs are ingested from S3. It may be increased (default = 2), when S3 objects are processed too slowly. This can be verified in your AWS on queue monitoring panel (SQS console – select queue – choose Monitoring tab).

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

IPFIX Records Rate Limit

This is NFO internal parameter – maximum IPFIX records per second. Default value 0 (means unlimited). If you see NFO server dropped messages (NetFlow Optimizer -> Status), this parameter could be set to another value, for example, 10000 records per second.

Set Access to your AWS Accounts

There are two alternative ways to access your AWS accounts where VPC Flow Logs are configured.

1. AWS Credentials Tab

Use this method if you have a list of independent AWS accounts. The EDFN Agent expects that each account has only one profile. On "AWS Credentials" tab add a row for each account:

Access key ID and Secret access key are mandatory.

These credentials are available here: AWS console > IAM > Users > select User name > Security credentials tab. "Profile name" and "Default region" are optional.

The EDFN agent monitors all available regions to gather information about your EC2 instances and VPCs. To retrieve the account ID and list of available regions, the agent makes API calls using “Default region”. If it isn't provided, "us-east-1" is used.

2. IAM Role Tab

Use this method if you create IAM Role to delegate access across AWS accounts. See https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html for details.

Create a list of your ARNs and optionally External IDs as follows:

arn:aws:iam::999999999999:role/Role1,
arn:aws:iam::999999999998:role/Role1,
arn:aws:iam::999999999997:role/Role2,externalID-2

and enter it here:

info

Do not enter ARN assigned to NFO instance!

Verify AWS Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of accounts and associated VPCs (IPFIX exporters section), Kinesis Streams, CloudWatch Log groups, and S3 buckets (with associated queues).

Open the IPFIX Exporters section to review and assign an exporter IP to each VPC. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Enable VPC Flow Logs Input

Depending on your AWS VPC Flow Logs collection, you can enable one or more of the following configurations.

note

Please note that to avoid duplicate VPC Flow logs ingestion, the EDFN Agent for each VPC will use Kinesis Stream, if configured, before attempting to get logs from the CloudWatch Log group.

Kinesis Streams

Open the Kinesis Streams section to enable VPC Flow logs ingestion using Kinesis Streams. Set Enhanced Fan-Out option, if necessary.

To investigate Kinesis Stream processing problems, change the parameter “KCL metrics level” – Kinesis Client Library metrics level: valid values are NONE, SUMMARY and DETAILED. For more information, visit Monitoring the Kinesis Client Library with Amazon CloudWatch (https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html).

CloudWatch Log Groups

Open the CloudWatch Log groups to enable VPC Flow logs ingestion using the CloudWatch API.

By default, VPC Flow Logs are requested every 30 seconds according to the “CloudWatchLogs request interval” parameter.

S3 and SQS

Open the “S3 and SQS” section to enable reading VPC Flow Logs from S3 using SQS messages notifications.

Network Conversations Module Output Fields (Module 10062)

The table below maps native AWS VPC Flow Logs fields to their corresponding output field names in the Network Conversations Monitor (Module 10062). For the complete Module 10062 field reference, see Network Conversations Monitor.

VPC Flow Logs Native FieldModule 10062 Output FieldDescription
nfc_idMessage type identifier (nfc_id=20062)
flow_typeType of flow (AWS)
exp_ipVPC virtual router IPv4 address, assigned per-VPC in the EDFN agent
exp_nameVPC flow exporter name
account-idaws_account_idAWS account ID
actionactionFlow action: A (Accept) or R (Reject)
bytesbytes_inBytes in the capture window (forward direction)
bytes_outBytes in the reverse direction (when bidirectional stitching is enabled)
packetspackets_inPackets in the capture window (forward direction)
packets_outPackets in the reverse direction (when bidirectional stitching is enabled)
srcaddrsrc_ipSource IPv4 or IPv6 address
srcportsrc_portSource port number
dstaddrdest_ipDestination IPv4 or IPv6 address
dstportdest_portDestination port number
protocolprotocolIANA protocol number
tcp-flagstcp_flagTCP flags (e.g. SYN, ACK, FIN)
startflow_start_timeStart time of the capture window
endflow_end_timeEnd time of the capture window
vpc-idaws_vpc_idVPC ID
aws_vpc_nameVPC name
interface-idaws_interface_idNetwork interface ID
instance-idaws_src_inst_idSource EC2 instance ID
aws_dest_inst_idDestination EC2 instance ID
src_vm_nameSource EC2 instance name
dest_vm_nameDestination EC2 instance name
src_vm_ip_pubSource EC2 instance public IPv4 address
dest_vm_ip_pubDestination EC2 instance public IPv4 address
src_cloud_regionCloud source region (e.g. us-east-1)
dest_cloud_regionCloud destination region
src_cloud_serviceCloud source service (e.g. EC2, S3)
dest_cloud_serviceCloud destination service
src_asnSource BGP Autonomous System Number
dest_asnDestination BGP Autonomous System Number
src_ccSource country code (GeoIP-derived, ISO 3166-1 alpha-2)
src_regionSource region (GeoIP-derived, city level)
src_citySource city (GeoIP-derived)
src_lonSource longitude (GeoIP-derived)
src_latSource latitude (GeoIP-derived)
dest_ccDestination country code (GeoIP-derived)
dest_regionDestination region (GeoIP-derived, city level)
dest_cityDestination city (GeoIP-derived)
dest_lonDestination longitude (GeoIP-derived)
dest_latDestination latitude (GeoIP-derived)
directionSession origin: inbound, outbound, internal, or unknown
stateConversation lifecycle state: B (Begin), C (Continuing), E (End)
durationConversation duration in seconds
threat_list_nameName of the matched cybersecurity threat list
reputationReputation score from the threat list
idpIdentity provider for the user
usernameUsername from identity enrichment (AD, Entra ID, Okta)
app_idApplication ID
app_nameApplication name
app_descApplication description

The following native VPC Flow Logs fields are not mapped by Module 10062:

VPC Flow Logs Native FieldNotes
subnet-idNot mapped by Module 10062
typeIPv4/IPv6/EFA traffic type — not mapped by Module 10062
pkt-srcaddrPacket-level origin source IP — not mapped
pkt-dstaddrPacket-level origin destination IP — not mapped
az-idAvailability Zone ID — not supported
log-statusNODATA and SKIPDATA entries are discarded by the EDFN agent
pkt-dst-aws-serviceDestination AWS service — not supported
pkt-src-aws-serviceSource AWS service — not supported
flow-directionAWS-native direction field — not supported (use direction from Module 10062 instead)
regionAWS region — not supported
sublocation-idSublocation ID — not supported
sublocation-typeSublocation type — not supported
traffic-pathNot supported
versionVPC Flow Log version. Version 2 uses fixed fields; version 3 and later use a configurable field list.