Oracle VCN Flow Logs Input Configuration

Prerequisites

Oracle Cloud Infrastructure (OCI) utilizes Virtual Cloud Network (VCN) flow logs to capture network traffic data.

VCN flow logs are enabled at the subnet level, ensuring that Virtual Network Interface Cards (VNICs) within the subnet contribute to the generation of VCN flow logs. The Logging service is responsible for the management of these logs.

Please note that NetFlow Optimizer (NFO) doesn't directly ingest logs from the Logging service. Instead, for NFO to process the VCN flow logs, they must be redirected to the Streaming service via the Service Connector Hub.

For more information about VCN flow logs, visit https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/vcn_flow_logs.htm

To access and ingest VCN flow logs you need one of the following:

1. Key-based Authentication parameters such as the user OCID, tenancy OCID, RSA key pair, etc.
2. Instance Principal Authentication, when NFO/EDFN is installed on an instance.

Configuration Steps

1. Enable VCN Flow Logs for subnets
2. Redirect VCN flow logs from the Logging service to a stream
3. Configure authentication
4. Configure EDFN Agent (NFO OCI flow logs input)
5. Enable and configure NFO module for VCN flow logs reporting

VCN Flow Logs Configuration

To configure VCN flow logs, ensure they are enabled for the specific subnet you intend to analyze. Subsequently, set up the transfer of these logs from the Logging service to the Streaming service through the Service Connector Hub.

Enable Flow Logs for the Subnet

To enable VCN flow logs, follow these steps:

1. Open the Oracle Cloud portal and navigate to the desired subnet
2. On the left side panel, select the Logs page
3. Click on "Enable Log" and, in the configuration window that appears, enter the necessary parameters, such as Compartment, Log group, Log name, and Log retention

For ease of configuration, it is recommended to use a single Log group specifically for VCN flow logs. Avoid writing other logs to this group to simplify the setup.

Create Stream

To create a new stream, follow these steps:

1. Open the Oracle Cloud portal and navigate to Analytics & AI > Messaging > Streaming
2. Click the "Create Stream" button
3. Enter the required parameters in the configuration window
4. Save the new stream

Create Service Connector

To transfer VCN flow logs from the Logging service to the stream, configure a Service Connector by following these steps in the Oracle Cloud portal:

1. Navigate to Logging > Service Connectors
2. Click the Create Service Connector button
3. In the configuration window, enter the required parameters:
   1. Connector name
   2. Source: Select the Logging service
   3. Target: Select the Streaming service
   4. Configure the source:
      1. Source connection: Select Compartment name and Log group. You may leave the "Logs" value unspecified to transfer all logs from the log group
      2. Log filter task: Select the where clause with the condition data.status = 'OK'
   5. Configuration task: Leave unspecified
   6. Configure the target: Select the Compartment and Stream where logs have to be transferred

Authentication Configuration

NFO requires access to Oracle Cloud Infrastructure API for VCN flow logs consuming, processing and enrichment. There are two choices how this access can be configured.

• Key-based Authentication – use this option if your EDFN is installed on premises, outside of Oracle Cloud or when you need to collect VCN flow logs from different tenancies. This method required user OCID, tenancy OCID, RSA key pair and region
• Instance Principal Authentication – use this option if your EDFN is installed on Oracle Compute Instance. This method is preferable and you do not need to specify access credentials

Key-Based Authentication

This authentication method is suitable when EDFN is installed outside Oracle Cloud or when you need to collect VCN flow logs from multiple tenancies. To set up authentication, follow these steps:

1. Create a domain group with specific access rights
2. Define a policy to specify user access rights
3. Create a user that EDFN will use for authentication
4. Generate an API key pair for user authentication

When EDFN is installed on an instance, you can utilize the Instance Principal Authentication method for collecting VCN flow logs from the installed tenancy. Additionally, you can add Key-Based Authentication credentials for consuming logs and enrichment from other tenancies.

Group

To create a Domain group, follow these steps:

1. Open the Oracle Cloud portal
2. Navigate to Identity > Domains
3. Select the domain to which the user and group will belong
4. Go to Groups and click the Create group button
5. Enter the name of the group and save it

Policy

A policy is a set of statements that control user permissions for specific resources or resource groups. To ensure that EDFN has the necessary access for building watch lists for enrichment and reading VCN flow logs, create the following policy:

Allow group <group-name> to inspect virtual-network-family in tenancy
Allow group <group-name> to read public-ips in tenancy
Allow group <group-name> to read ipv6s in tenancy
Allow group <group-name> to read instances in tenancy
Allow group <group-name> to inspect streams in tenancy
Allow group <group-name> to use streams in tenancy where target.stream.id = '<STREAM-OCID-HERE>'

For detail about policy syntax, visit https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/policysyntax.htm

User

To create a Domain User for EDFN authentication purposes, follow these steps:

1. Open the Oracle Cloud portal
2. Navigate to Identity > Domains
3. Select the domain
4. Go to Users and click the “Create user” button
5. Enter the required fields
6. Assign the user to the group created earlier

note

It is recommended to create a dedicated user specifically for EDFN authentication rather than using a regular user.

API key

The EDFN application uses an RSA key for authentication in place of a user password. For authentication, EDFN requires a private key, while the corresponding public key needs to be uploaded into Oracle Cloud Infrastructure (OCI) for user validation. Follow these steps:

1. Generate the encrypted key with a passphrase:

openssl genrsa -out oci_edfn_key.pem -aes128 2048

Store this key anywhere on the EDFN machine, e.g., in ~/.oci or ${edfn_home}/conf directory.

2. Change the key file permissions:

chmod 600 oci_edfn_key.pem

3. Generate the public key from the private key:

openssl rsa -pubout -in oci_edfn_key.pem -out oci_edfn_key_public.pem

4. Print and copy the public key content:

cat oci_edfn_key_public.pem

5. In the Oracle Cloud portal, open the previously created user details and navigate to API keys in the left panel
6. Click the Add API key button
7. Select the Paste a public key radio button and paste the public key content
8. Press the Add button to save the key
9. In the configuration file preview, you will find all required configurations for the EDFN Agent except the private key file path and passphrase. Use these properties for agent configuration. You can review these properties later by clicking on the API keys table row and selecting View configuration file

For key pair generation details, visit https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm

Instance Principal Authentication

When NFO/EDFN is installed on an instance, it can be authorized to make API calls to OCI services without the need for pre-installed RSA keys or other credentials. This option utilizes an access key/token retrieved using the Instance Metadata Service, available on any instance. To authorize an instance to call OCI services, set up the following resources:

1. Dynamic Group: All instances from the group will have permissions to make specific API calls. Provide a matching rule to specify which instances are members of the group
2. Policy: Specify rules to grant permissions to members of the dynamic group

For details, visit https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm

Dynamic Group

Dynamic groups function similarly to user groups, where all members of a group are granted permissions according to a policy specification. To create a dynamic group, follow these steps:

1. Open the Oracle Cloud portal
2. Navigate to Identity > Domains
3. Select a domain
4. Go to Dynamic groups and click the Create dynamic group button
5. Enter the group name and define matching rules For example, to match a single instance, you can use the following rule:

Any {instance.id = '<instance-ocid-here>'}

Attention

When the matching rule is changed, you have to wait for about one hour before the changes take effect, and a new instance is included or excluded from the dynamic group.

For more details about dynamic groups and rule syntax, visit https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/managingdynamicgroups.htm

Policy

A policy is a set of statements that control an instance's permissions for specific resources or resource groups. To grant EDFN access for building watch lists for enrichment and reading VCN flow logs, create the following policy:

Allow dynamic-group <group-name> to inspect virtual-network-family in tenancy
Allow dynamic-group <group-name> to read public-ips in tenancy
Allow dynamic-group <group-name> to read ipv6s in tenancy
Allow dynamic-group <group-name> to read instances in tenancy
Allow dynamic-group <group-name> to inspect streams in tenancy
Allow dynamic-group <group-name> to use streams in tenancy where target.stream.id = '<STREAM-OCID-HERE>'

For more details about policy syntax, visit https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/policysyntax.htm

Configure EDFN Agent

When you click on Oracle Cloud Infrastructure VCN Flow Logs on the NFO Inputs page you will be presented with the following configuration screen.

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO Oracle Cloud VCN Flow Logs processing includes data enrichment with fields such as Compute Instance names, Tenancy names, Compartment names, VCN names, Subnet names, Regions, etc. This information is updated on the cron schedule set here.

Streaming Concurrency

Adjust the number of parallel stream processors based on the comparison between producing and consuming messages. If the number of producing messages exceeds the number of consuming messages, consider increasing this parameter. You can monitor relevant statistics on the OCI Stream Metrics page. Additionally, the consuming rate can be optimized by adjusting the Stream messages/request parameter, as discussed below.

Concurrency

Adjust the number of parallel message processors based on the processing speed of messages. If messages are being processed too slowly, increase this value. A warning message on the Status page will indicate: “OCI VCN flow logs queue size is growing (N). Please increase the "Concurrency" parameter to solve this problem.” Additionally, this issue can be examined in the ${EDFN_HOME}/logs/oci.log file, where the internal queue size is monitored in consecutive statistics messages:

ipfix_msgs=15673912 recs=56734987 queue=1000000 queueSizeSlope=0.52

Decrease this value if the NFO Server is dropping some input flows.

Stream Messages/Request

This parameter defines the number of messages consumed by the EDFN agent per request. You can set any value up to 10,000.

Stream Group Cursor Name

EDFN utilizes a Consumer Group to consume messages. In this field, you can use any group name unless you have a single EDFN installation. However, when multiple EDFN instances consume messages from the same stream, this name becomes crucial.

1. If you need to process messages independently for each NFO, this parameter must be different for each EDFN
2. If each message should be processed only once, the name must be the same for all EDFNs

Max IPFIX Packet Size

This is an internal NFO parameter, representing the maximum IPFIX UDP message size. It is expected to be less than or equal to the MTU. When NFO and EDFN are installed on the same host, you can increase this parameter up to 3800 to enhance processing speed.

OCI Credentials Tab

The table on this tab may appear empty if NFO/EDFN is installed on an OCI Compute Instance and Instance Principal Authentication is used. However, if NFO/EDFN is installed outside your Oracle Cloud environment or you need to collect VCN flow logs from multiple tenancies, you must provide the following parameters for key-based authentication.

Some of these parameters can be copied from the User details page > API keys > Select the stored key and click View configuration file.

Profile Name

This is a short name for the profile. It appears in notification messages and EDFN logs. If left empty, the default name profile-<n> is used.

Tenancy OCID

The tenancy OCID to be connected. The table cannot contain duplicate tenancies.

User OCID

The OCID of the user who is connected to Oracle Cloud API. The user must have enough rights to inspect Virtual Network resources, Instances and Streams. Also this user has to have rights to consume messages from a stream with VCN flow logs.

Public Key MD5 Fingerprint

The MD5 fingerprint of the public key that was generated to the user.

Private Key PEM File Path

The absolute path to the private key file on the machine where EDFN is installed. EDFN service must have read access to this file.

Private Key Passphrase

This is the password used to encrypt the private key. We recommend generating private keys with passwords. This is because the file is stored unencrypted on the disk, while the passphrase is stored encrypted in the NFO Database.

Default Region

Home region. EDFN requests OCI resources on all available regions, but a default region is required for the first query.

Virtual Cloud Networks Tab

No configuration actions are required on this tab. The table is populated by the agent. You can use this table to review all available Virtual Cloud Networks in your tenancies.

VCN Subnets Tab

No configuration actions are required on this tab. The table is populated by the agent. You can use this table to review all available VCN Subnets and virtual router IPv4 addresses in your tenancies. The virtual router IPv4 address is used as the Syslog/JSON exp_ip field.

Streams Tab

This tab is also populated by the agent and contains a list of all streams. At least one stream with VCN flow logs must be selected for message consumption. If streams are not selected, the agent does not start and cannot consume VCN flow logs. The selected streams must exclusively contain VCN flow logs and no other logs. Otherwise, NFO will attempt to parse unexpected logs and ignore them.

Network Conversations Module Output Fields

The table below provides a side-by-side comparison of fields available in native VCN flow logs and NFO output fields across different versions.

VCN Flow Logs Native Fields	Enriched 20062 Output	Enrichment in Future Releases	Field Description
	nfc_id	nfc_id	Message type identifier
	flow_type	flow_type	Type of flow
	exp_ip	exp_ip	Subnet virtual router ipv4 address
data.action	action	action	Accept or reject
data.bytesOut	bytes_in	bytes_in	Number of bytes recorded in the capture window.
data.destinationAddress	dest_ip or dest_ip6	dest_ip or dest_ip6	Ipv4 or ipv6 of the destination.
data.destinationPort	dest_port	dest_port	Port number of the destination.
data.endTime	flow_end_time	flow_end_time	End time of the capture window in unix epoch seconds.
data.packets	packets_in	packets_in	Number of packets recorded in the capture window.
data.protocol	protocol	protocol	Iana protocol number.
data.sourceAddress	src_ip or src_ip6	src_ip or src_ip6	Ipv4 or ipv6 address of the source.
data.sourcePort	src_port	src_port	Port number of the source
data.startTime	flow_start_time	flow_start_time	Start time of the capture window in unix epoch seconds.
	exp_name	exp_name	Name of vcn flow exporter
	bytes_out	bytes_out	Number of bytes in reply when client-server flows are stitched
	packets_out	packets_out	Number of packets in reply when client-server flows are stitched
	src_vm_name	src_vm_name	Name of the source instance
	dest_vm_name	dest_vm_name	Name of the destination instance
	src_region	src_region	Source GEO region
	src_cloud_region	src_cloud_region	Cloud source region
	dest_region	dest_region	Destination GEO region
	dest_cloud_region	dest_cloud_region	Cloud destination region
	src_cloud_service	src_cloud_service	Cloud source service
	dest_cloud_service	dest_cloud_service	Cloud destination service
	direction	direction	Direction of originating flow
	threat_list_name	threat_list_name	Name of a cybersecurity threat list
	reputation	reputation	Reputation from the treat list
	idp	idp	User identity provider
	username	username	User name from login events (AD, Azure AD, Okta, etc)
	app_id	app_id	Application ID
	app_name	app_name	Application name
	app_desc	app_desc	Application description
	src_cc	src_cc	Source country code
	dest_cc	dest_cc	Destination country code
	oci_src_vcn_name	oci_src_vcn_name	Source vcn name
	oci_src_subnet_name	oci_src_subnet_name	Source subnet name
	oci_src_tenant_name	oci_src_tenant_name	Source tenancy name
	oci_src_ip_pub	oci_src_ip_pub	Source instance public ip address
	oci_dest_vcn_name	oci_dest_vcn_name	Destination vcn name
	oci_dest_subnet_name	oci_dest_subnet_name	Destination subnet name
	oci_dest_tenant_name	oci_dest_tenant_name	Destination tenancy name
	oci_dest_ip_pub	oci_dest_ip_pub	Destination instance public ip address
	oci_src_compartment_name	oci_src_compartment_name	Name of the source compartment of vnic
	oci_dest_compartment_name	oci_dest_compartment_name	Name of the destination compartment of vnic
		oci_src_vnic_name	Source vnic hostname
		oci_dest_vnic_name	Destination vnic hostname
		oci_src_vcn_dns_label	DNS label of the source vcn
		oci_src_subnet_dns_label	DNS label of the source subnet
		oci_src_vnic_fqdn	Source fqdn
		oci_dest_vcn_dns_label	DNS label of the destination vcn
		oci_dest_subnet_dns_label	DNS label of the destination subnet
		oci_dest_vnic_fqdn	Destination fqdn
oracle.compartmentid			OCID of the compartment the log group is in.
oracle.ingestedtime			Time the log was ingested by oci logging.
oracle.loggroupid			OCID of the log group.
oracle.logid			OCID of the log.
oracle.tenantid			OCID of the tenant.
oracle.vniccompartmentocid			OCID of the compartment to which the vnic belongs.
oracle.vnicocid			OCID of the vnic.
oracle.vnicsubnetocid			OCID of the subnet to which the vnic belongs.