Azure NSG and VNet Flow Logs Configuration

Prerequisites

Azure flow logs are generated by Virtual Networks (VNet) or Network Security Group (NSG). NSG can be assigned to a Virtual Network subnet or to an interface. Both VNet and NSG flow logs are stored within a storage account blob container: insights-logs-flowlogflowevent and insights-logs-networksecuritygroupflowevent.

warning

NSG flow logs will be retired on 30 September 2027, therefore we recommend using VNet flow logs. In addition, there are two versions of NSG flow logs. If you decide to utilize NSG flow logs, we recommend using flow logs version 2, because it contains bytes and packets. If you use version 1, bytes and packets information will be missing.

For more information on VNet and NSG Flow Logs, visit https://learn.microsoft.com/en-us/azure/network-watcher/vnet-flow-logs-overview and https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview.

Azure flow log requires an Azure Storage account to store the flow logs.

To access and ingest NSG flow logs you need one of the following:

1. Service principal identity
2. System-assigned managed identity

Configuration Steps

1. Configure VNet or NSG Flow Logs
2. Configure EDFN Agent for ingestion of Azure Flow Logs
3. Enable and Configure NFO Modules for Azure Flow Logs reporting

Azure Flow Logs Configuration

Network Watcher Creation

1. Open the Azure portal
2. Navigate to Network Watcher
3. Create Network Watcher for a subscription and regions to be monitored:
   1. Click “+ Create” button
   2. Select a subscription form the dropdown
   3. Select one or several regions
   4. Click “Add” to create new Network Watcher

Register Insights Provider

1. Open the Azure portal
2. Navigate to Subscriptions
3. From the list of subscriptions, select the subscription you want to enable the provider for
4. Select Resource providers, under the Settings group in the left panel
5. Confirm that the Status for the microsoft.insights provider is Registered. If the status is Unregistered, then select the provider and click on Register button

Storage Account Creation

Azure flow log data is written to an Azure Storage account. You must use StorageV2 (general purpose v2) account type.

1. Open the Azure portal
2. Navigate to Storage accounts
3. Click “+ Create” button
4. Select the subscription, the resource group, enter the storage account name and select the region. The storage account must be in the same region as the VNet or NSG
5. Select Performance: general-purpose v2 account)
6. Select a Reduncancy that matches your durability requirements
7. Press “Review and create” button and then “Create”

Enable Azure Flow Logs

1. Open the Azure portal
2. Navigate to Network Watcher
3. Select Flow logs under logs group in the left panel
4. Click “+ Create” button
5. Select the subscription you want to configure flow logs for
6. Seclect Flow log type: Virtual network (recommend) or Network security group
7. Select target resources: VNet, Subnet, network interface, or NSG
8. Select the storage account you’ve previously created
9. Set flow logs Retention (days) parameter
10. Review and create flow log

Storage Queue Configuration

In the previous section you’ve created the Storage Account and configure VNet or NSG flow logs. Flow logs can be found here: Azure portal -> Storage Accounts -> select the account name -> Containers, under the Blob service group in the left panel -> insights-logs-flowlogflowevent or insights-logs-networksecuritygroupflowevent container.

Flow logs are written into a Blob Block resource. For each VM interface a separate blob is created. Blobs are rotated every hour, but blocks with NSG flow logs are committed more often. NFO doesn’t monitor a new block uploaded directly. You have to configure a Storage Queue in the same Storage account to receive events about a new block committed.

1. Open the Azure portal
2. Navigate to Storage accounts
3. From the list of accounts, select the storage account you’ve created earlier for VNet or NSG Flow logs
4. Create Storage Queue. Important: Azure flow logs container and queue must be in the same storage account

1. Select Queues, under the Queue service group in the left panel
2. For queue creation select “+ Queue” and enter the queue name
3. Select “OK” to save the queue

5. Then create the storage account event subscription:

1. Select Events in the left panel and click on "+Event Subscription"
2. EVENT SUBSCRIPTION DETAILS section. Enter the subscription name
3. EVENT TYPES section. Filter to Event Types: Blob created
4. ENDPOINT DETAILS section. Endpoint Type: Storage Queues
5. ENDPOINT DETAILS section. Endpoint: Select an endpoint. Select the queue you’ve already created. The queue must be in the same storage account
6. (Optionally) If the storage account is used not only for flow logs (contains containers other than insights-logs-*), you have to configure filters
1. Navigate to the Filter section
2. Select “Enable subject filtering” checkbox
3. Enter Subject Begins With:
1. For VNet flow logs: /blobServices/default/containers/insights-logs-flowlogflowevent/
2. For NSG flow logs: /blobServices/default/containers/insights-logs-networksecuritygroupflowevent/
4. Enter Subject Ends With: .json
7. (Optionally) Navigate to “Additional Features” section and configure Retry Policies
8. (Optionally) Navigate to “Delivery Properties” and configure Storage queue message TTL
9. Choose “Create” to save the event subscription

After these configuration steps, when a new blob block with Azure flow logs is committed, a notification message is pushed into the Storage Queue. NFO receives this message, reads flow logs from the block and deletes the message.

Authentication and Access Control (IAM) Configuration

NFO requires access to the Azure Resource Manager and to the Storage Account for Flow Logs processing. There are two choices how this access can be configured.

• Using service principal identity - Use this option if your EDFN is installed on premises or outside of Azure. This method requires a secret access key, application (client) ID, subscription ID, and tenant ID
• Using system-assigned managed identity - Use this option if your EDFN is installed on an Azure Virtual Machine resource. This method does not require access credentials.

For more details, please visit https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

Service Principal Identity Authentication

An Azure service principal is an identity created for use with applications. The EDFN agent uses this service principal for Azure resources access. Do following steps for the service principal authentication.

1. Open the Azure portal
2. Navigate to Microsoft Entra ID and select App registrations under the Mange section in the left panel
3. Select “+ New registration“
4. Enter Name, for example, edfn-agent
5. Select “Register” to save the application. You will be navigated to the application overview page
6. Select “Certificates & secrets” in the left panel
7. Select “+ New client secret” and enter a name for the secret
8. Select “Save” and copy the secret key value into Microsoft Azure NSG Flow Logs input Secret key field.

You will need following properties form the application overview page:

• client: Application (client) ID
• tenant: Directory (tenant) ID
• key: client secret string value
• subscription: navigate to Subscriptions and copy value from the Subscription ID column. This will be the default subscription

If you have other subscriptions you want to monitor you have to give a read access to all of them it as follows:

1. Navigate to Subscriptions
2. From the list of subscriptions, select the subscription to be configured
3. Select Access control (IAM) in the left panel
4. Select “+ Add” and choose Add role assignment
5. In the “Role” tab, find and select Reader job function
6. In the "Members" tab, leave unchanged the “Assign access to” (User, group, or service principal)
7. Click “+ Select members”
8. Enter the service principal name in the Search field (for example, ednf-agent). When the service principal appears in the results, select it
9. Click “Review + assign”

Repeat these steps for each subscription to be monitored.

Storage Account Access for Service Principal Authentication

Even though EDFN may be authenticated with the “Storage account access key”, we recommend using Microsoft Entra ID authentication. The Virtual machine must have Storage Blob Data Reader and Storage Queue Data Message Processor roles to access storage account container and queue.

1. Navigate to your storage account
2. Select Access control (IAM) in the left panel
3. Select “+ Add” and choose Add role assignment
4. In the “Role” tab, find and select Storage Blob Data Reader job function role
5. In the "Members" tab, leave unchanged the “Assign access to” (User, group, or service principal)
6. Click “+ Select members”
7. Enter the service principal name in the Search field (for example, ednf-agent). When the service principal appears in the results, select it
8. Click “Review + assign”
9. Repeat steps 3-8, on the step 4 select role Storage Queue Data Message Processor

After these steps the storage container and the storage queue are processed without an access key.

System-assigned Managed Identity Authentication

For enabling system-assigned managed identity on the EDFN installation VM read the following instruction: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#enable-system-assigned-managed-identity-on-an-existing-vm

1. Open the Azure portal
2. Navigate to Virtual machines
3. From the list of VMs, select the VM, where EDFN is installed
4. Select Identity, under the Security group in the left panel
5. Under the System assigned tab, Status, select On and then click Save
6. Confirm enable the system assigned managed identity for the VM

After system-assigned managed identity is enabled, configure VM access to a subscription.

1. Navigate to Subscriptions
2. From the list of subscriptions, select the subscription to be configured
3. Select Access control (IAM) in the left panel
4. Select “+ Add” and choose Add role assignment
5. In the “Role” tab, find and select Reader job function role
6. In the “Members” tab, the “Assign access to” dropdown, choose Managed identity
7. Click “+ Select members”
8. In the “Subscription” dropdown, choose the subscription, where VM is placed
9. Search and select the VM
10. Click “SaveReview + assign”

Repeat previous steps for each subscription to be monitored.

Storage Account Access for System-assigned Managed Identity Authentication

Even though EDFN may be authenticated with the “Storage account access key”, we recommend using AD authentication. The Virtual machine must have Storage Blob Data Reader and Storage Queue Data Message Processor roles to access storage account container and queue. Instruction how to configure managed identity access is available here: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-linux-vm-access-storage#grant-your-vm-access-to-an-azure-storage-container.

1. Navigate to your storage account
2. Select Access control (IAM) in the left panel
3. Select “+ Add” and choose Add role assignment
4. In the “Role” dropdown, choose Storage Blob Data Reader
5. In the “Assign access to” dropdown, choose Virtual Machine
6. In the “Subscription” dropdown, choose the subscription, where VM is placed
7. Search and select the VM
8. Click “Save”
9. Repeat steps 3-8, on the step 4 select role Storage Queue Data Message Processor

After these steps the storage container and the storage queue are processed without an access key.

Configure EDFN Agent

When you click on Microsoft Azure Flow Logs in NFO Inputs page you will be presented with the the following configuration screen.

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO Azure Flow Logs processing includes data enrichment with fields such as VM names, Subscription names, NSG names, Subnet names, Regions, etc. This information is updated on cron schedule set here.

Subscription ID

This is the default subscription ID. Open Azure portal, navigate to Subscriptions and copy value from the Subscription ID column. There are two alternative ways to access your Azure directory where Flow Logs are configured: service principal or system-assigned managed identity. Set subscription ID, if you use service principal to access your Azure environments. If you use system-assigned managed identity, leave it blank.

Tenant ID

Directory (tenant) ID. If you use system-assigned managed identity, leave it blank. If you use service principal identity, open Azure portal > navigate to Microsoft Entra ID > select App Registrations > select application created before, e.g. edfn-agent > copy Directory (tenant) ID from the Overview page.

Application ID

Application (client) ID. If you use system-assigned managed identity, leave it blank. If you use service principal identity, open Azure portal > navigate to Microsoft Entra ID > select App Registrations > select application created before, e.g. edfn-agent > copy Application (client) ID from the Overview page.

Secret key

Application (client) secret key. Paste secret key created before. If you use system-assigned managed identity, leave it blank. If you haven't created a client secret yet, open Azure portal > navigate to Microsoft Entra ID > select App Registrations > select application created before, e.g. edfn-agent > select Certificates & secrets > click on Client secrets tab and then create a new one.

Concurrency

The number of parallel message processors. Increase this value when blobs are consumed too slowly. Decrease this value, when NFO Server drops some input flows.

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

note

Azure default URLs can be overwritten. Add $EDFN_HOME/conf/azure.properties file with the following content:

managementURI=https://management.core.windows.net/
baseURL=https://management.azure.com/
authURL=https://login.microsoftonline.com/
graphURL=https://graph.windows.net/
storageEndpointSuffix=.core.windows.net

Storage Accounts Tab

Open Storage accounts section to authorize access to Flow logs in Azure Storage.

Storage account name and Queue name are required. Access key is optional if you use AD authentication (service principal or system-assigned managed identity) for storage account access.

The storage account has two access keys, please use the first one. The second key can be used during first key regeneration. Access keys are available on Storage account page -> Access keys in the left panel.

Enter Storage account name, Access key, if needed, and Queue name.

Press "Save".

Verify Azure Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of subscriptions and associated virtual network names (IPFIX exporters section). This action will close the input configuration. It may take from several seconds to a few minutes to retrieve the list.

Open the IPFIX Exporters section to review and assign an exporter IP to each Virtual Network Name. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Network Conversations Module (nfc_id=20062) Output Fields

The table below provides a side-by-side comparison of fields available in native NSG flow logs and NFO enriched output fields from various Modules.

NSG Flow Logs V2 Native Fields	Enriched 20062 Output	Field Description
	nfc_id	Message type identifier
	flow_type	Type of flow
	exp_ip	Subnet virtual router ipv4 address
time		Time in UTC when the event was logged.
Time stamp		Time stamp of when the flow occurred in UNIX epoch format.
Source IP	src_ip	Source IP address.
Destination IP	dest_ip	Destination IP address.
Source port	src_port	Source port.
Destination port	dest_port	Destination port.
Protocol	protocol	Protocol of the flow. Valid values are T for TCP and U for UDP.
Traffic flow	direction	Direction of the traffic flow or origiinating flow
Traffic decision	action	Whether traffic was allowed or denied. Valid values are A for allowed and D for denied.
Flow State	state	State of the flow. Possible states are: B - Begin; C - Continuing for an ongoing flow; E - End when a flow is terminated.
Packets sent	packets_in	Total number of TCP packets sent from source to destination since the last update.
Bytes sent	bytes_in	Total number of TCP packet bytes sent from source to destination since the last update. Packet bytes include the packet header and payload.
Packets received	packets_out	Total number of TCP packets sent from destination to source since the last update.
Bytes received	bytes_out	Total number of TCP packet bytes sent from destination to source since the last update. Packet bytes include packet header and payload.
	src_host	Source host name
	dest_host	Destination host name
	azure_src_subs_id	Source subscription ID
	azure_src_subs_name	Source subscription name
	azure_src_vm_name	Source VM name
	azure_src_nsg_name	Source NSG name
	azure_src_vnet_name	Source virtual network name
	azure_src_subnet_name	Source subnet name
	azure_src_service	Source service
	src_region	Source GEO region
	src_cloud_region	Source Azure cloud region
	azure_src_res_grp_name	Source resource group name
	azure_dest_subs_id	Destination subscription ID
	azure_dest_subs_name	Destination subscription name
	azure_dest_vm_name	Destination VM name
	azure_dest_nsg_name	Destination NSG name
	azure_dest_vnet_name	Destination virtual network name
	azure_dest_subnet_name	Destination subnet name
	azure_dest_service	Destination service
	dest_cloud_region	Destination Azure cloud region
	azure_dest_res_grp_name	Destination resource group name
	flow_count	Number of consolidated flows
		Percent of total traffic volume within virtual router
	flow_start_time	Start time of the first consolidated flow
	flow_end_time	End of the last consolidated flow
	t_int	Observation time interval, msec
	exp_name	Name of NSG flow exporter
mac	src_mac	Source MAC address
mac	dest_mac	Destination MAC address
	threat_list_name	Name of a cybersecurity threat list
	reputation	Reputation from the treat list
	idp	User identity provider
	username	User name from login events (AD, Microsoft Entra ID, Okta, etc)
	app_id	Application ID
	app_name	Application name
	app_desc	Application description
	src_as	Source AS
	src_cc	Source country code
	src_region	Source GEO region
	src_city	Source city
	src_lon	Source longitude
	src_lat	Source latitude
	dest_as	Destination AS
	dest_cc	Destination country code
	dest_region	Destination GEO region
	dest_city	Destination city
	dest_lon	Destination longitude
	dest_lat	Destination latitude
systemId		System ID of the network security group.
category		Category of the event. The category is always NetworkSecurityGroupFlowEvent.
resourceid		Resource ID of the network security group.
operationName		Always NetworkSecurityGroupFlowEvents.
Version		Version number of the flow log's event schema.
rule		Rule for which the flows are listed.
mac		MAC address of the NIC for the VM where the flow was collected.