Microsoft Azure NSG Flow Logs Input Configuration

Prerequisites

Azure flow logs are generated by Network Security Group (NSG). NSG can be assigned to a Virtual Network subnet or to an interface. Flow logs are stored within a storage account blob container: insights-logs-networksecuritygroupflowevent.

There are two versions of NSG flow logs. We recommend using flow logs version 2, because it contains bytes and packets. If you use version 1, but bytes and packets information will be missing.

For more information on NSG Flow Logs, visit https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-overview.

NSG flow log requires an Azure Storage account to store the flow logs.

To access and ingest NSG flow logs you need one of the following:

1. Service principal identity
2. System-assigned managed identity

Configuration Steps

1. Configure NSG Flow Logs
2. Configure EDFN Agent for ingestion of Azure NSG Flow Logs
3. Enable and Configure NFO Modules for Azure NSG Flow Logs reporting

NSG Flow Logs Configuration

For detailed instruction about NSG flow logs configuration visit following link: https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-nsg-flow-logging-portal.

Enable Network Watcher

1. Open the Azure portal
2. In the search box, enter Network Watcher. When Network Watcher appears in the results, select it
3. From the list of subscriptions, select the subscription you want to enable the Network Watcher for. Then expand regions
4. Select a region and click on the “…” to the right of the region
5. Select Enable Network Watcher

Register Insights Provider

1. Open the Azure portal
2. In the search box, enter Subscriptions. When Subscriptions appears in the results, select it
3. From the list of subscriptions, select the subscription you want to enable the provider for
4. Select Resource providers, under the Settings group in the left panel
5. Confirm that the Status for the microsoft.insights provider is Registered. If the status is Unregistered, then select the provider and click on Register button

Storage Account Creation

NSG flow log data is written to an Azure Storage account. You must use StorageV2 (general purpose v2) account type.

1. Open the Azure portal
2. Select “+ Create a resource” at the top left corner of the portal
3. Select Storage in the left panel and then Storage account
4. Select the subscription, the resource group, enter the storage account name and select the region. The storage account must be in the same region as the NSG
5. Select Account kind: StorageV2 (general purpose v2)
6. Select a replication strategy that matches your durability requirements
7. Press “Review and create” button and then “Create”

Enable NSG Flow Logs

1. Open the Azure portal
2. Navigate to Network Watcher
3. Select NSG flow logs under logs group in the left panel
4. From the list of NSGs, select the NSG name you want to configure flow logs for
5. Switch flow logs status from Off to On
6. Select flow logs version 2
7. Select the storage account you’ve previously created
8. Optionally you can set flow logs Retention (days) parameter
9. Save settings

Storage Queue Configuration

In the previous section you’ve created the Storage Account and configure NSG flow logs. Flow logs can be found here: Azure portal -> Storage Accounts -> select the account name -> Containers, under the Blob service group in the left panel -> insights-logs-networksecuritygroupflowevent container.

Flow logs are written into a Blob Block resource. For each VM interface a separate blob is created. Blobs are rotated every hour, but blocks with NSG flow logs are committed more often. NFO doesn’t monitor a new block uploaded directly. You have to configure a Storage Queue in the same Storage account to receive events about a new block committed.

1. Open the Azure portal
2. Navigate to Storage accounts
3. From the list of accounts, select the storage account you’ve created earlier for NSG Flow logs
4. Create Storage Queue. Important: NSG flow logs container and queue must be in the same storage account
   1. Select Queues, under the Queue service group in the left panel
   2. For queue creation select “+ Queue” and enter the queue name
   3. Select “OK” to save the queue
5. Then create the storage account event subscription:
   1. Select Events in the left panel and click on "+Event Subscription"
   2. EVENT SUBSCRIPTION DETAILS section. Enter the subscription name
   3. EVENT TYPES section. Filter to Event Types: Blob created
   4. ENDPOINT DETAILS section. Endpoint Type: Storage Queues
   5. ENDPOINT DETAILS section. Endpoint: Select an endpoint. Select the queue you’ve already created. The queue must be in the same storage account
   6. (Optionally) If the storage account is used not only for flow logs (contains containers other than insights-logs-networksecuritygroupflowevent), you have to configure filters
   1. Navigate to the Filter section
   2. Select “Enable subject filtering” checkbox
   3. Enter Subject Begins With: /blobServices/default/containers/insights-logs-networksecuritygroupflowevent/
   4. Enter Subject Ends With: .json
   7. Choose “Create” to save the event subscription
   After these configuration steps, when a new blob block with NSG flow logs is committed, a notification message is pushed into the Storage Queue. NFO receives this message, reads flow logs from the block and deletes the message.

Authentication and Access Control (IAM) Configuration

NFO requires access to the Azure Resource Manager and to the Storage Account for NSG Flow Logs processing. There are two choices how this access can be configured.

• Using service principal identity - Use this option if your EDFN is installed on premises or outside of Azure. This method requires a secret access key, application (client) ID, subscription ID, and tenant ID
• Using system-assigned managed identity - Use this option if your EDFN is installed on an Azure Virtual Machine resource. This method does not require access credentials.

For more details please visit https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

Service Principal Identity Authentication

An Azure service principal is an identity created for use with applications. The EDFN agent uses this service principal for Azure resources access. Do following steps for the service principal authentication.

1. Open the Azure portal
2. Navigate to Azure Active Directory and select App registrations under the Mange section in the left panel
3. Select “+ New registration“
4. Enter Name, for example, edfn-agent
5. Select “Register” to save the application. You will be navigated to the application overview page
6. Select “Certificates & secrets” in the left panel
7. Select “+ New client secret” and enter a name for the secret
8. Select “Save” and copy the secret key value into Microsoft Azure NSG Flow Logs input Secret key field.

You will need following properties form the application overview page:

• client: Application (client) ID
• tenant: Directory (tenant) ID
• key: client secret string value
• subscription: navigate to Subscriptions and copy value from the Subscription ID column. This will be the default subscription

If you have other subscriptions you want to monitor you have to give a read access to all of them it as follows:

1. Navigate to Subscriptions
2. From the list of subscriptions, select the subscription to be configured
3. Select Access control (IAM) in the left panel
4. Select “+ Add” and choose Add role assignment
5. In the “Role” dropdown, choose Reader
6. Leave unchanged the “Assign access to” dropdown (Azure AD user, group, or service principal)
7. Enter the service principal name in the Select field (for example, ednf-agent). When the service principal appears in the results, select it
8. Click “Save”

Repeat these steps for each subscription to be monitored.

Storage Account Access for Service Principal Authentication

Even though EDFN may be authenticated with the “Storage account access key”, we recommend using AD authentication. The Virtual machine must have Storage Blob Data Reader and Storage Queue Data Message Processor roles to access storage account container and queue.

1. Navigate to your storage account
2. Select Access control (IAM) in the left panel
3. Select “+ Add” and choose Add role assignment
4. In the “Role” dropdown, choose Storage Blob Data Reader
5. Leave unchanged the “Assign access to” dropdown (Azure AD user, group, or service principal)
6. Enter the service principal name in the Select field (for example, ednf-agent). When the service principal appears in the results, select it
7. Click “Save”
8. Repeat steps 3-7, on the step 4 select role Storage Queue Data Message Processor

After these steps the storage container and the storage queue are processed without an access key.

System-assigned Managed Identity Authentication

For enabling system-assigned managed identity on the EDFN installation VM read the following instruction: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#enable-system-assigned-managed-identity-on-an-existing-vm

1. Open the Azure portal
2. Navigate to Virtual machines
3. From the list of VMs, select the VM, where EDFN is installed
4. Select Identity, under the Settings group in the left panel
5. Under the System assigned tab, Status, select On and then click Save
6. Confirm enable the system assigned managed identity for the VM

After system-assigned managed identity is enabled, configure VM access to a subscription.

1. Navigate to Subscriptions
2. From the list of subscriptions, select the subscription to be configured
3. Select Access control (IAM) in the left panel
4. Select “+ Add” and choose Add role assignment
5. In the “Role” dropdown, choose Reader
6. In the “Assign access to” dropdown, choose Virtual Machine
7. In the “Subscription” dropdown, choose the subscription, where VM is placed
8. Search and select the VM
9. Click “Save”

Repeat previous steps for each subscription to be monitored.

Storage Account Access for System-assigned Managed Identity Authentication

Even though EDFN may be authenticated with the “Storage account access key”, we recommend using AD authentication. The Virtual machine must have Storage Blob Data Reader and Storage Queue Data Message Processor roles to access storage account container and queue. Instruction how to configure managed identity access is available here: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-linux-vm-access-storage#grant-your-vm-access-to-an-azure-storage-container.

1. Navigate to your storage account
2. Select Access control (IAM) in the left panel
3. Select “+ Add” and choose Add role assignment
4. In the “Role” dropdown, choose Storage Blob Data Reader
5. In the “Assign access to” dropdown, choose Virtual Machine
6. In the “Subscription” dropdown, choose the subscription, where VM is placed
7. Search and select the VM
8. Click “Save”
9. Repeat steps 3-8, on the step 4 select role Storage Queue Data Message Processor

After these steps the storage container and the storage queue are processed without an access key.

Configure EDFN Agent

When you click on Microsoft Azure NSG Flow Logs in NFO Inputs page you will be presented with the the following configuration screen.

On this screen you can configure the following parameters:

Settings Tab

Cron Schedule

NFO Azure NSG Flow Logs processing includes data enrichment with fields such as VM names, Subscription names, NSG names, Subnet names, Regions, etc. This information is updated on cron schedule set here.

Subscription ID

This is the default subscription ID. Open Azure portal, navigate to Subscriptions and copy value from the Subscription ID column. There are two alternative ways to access your Azure directory where NSG Flow Logs are configured: service principal or system-assigned managed identity. Set subscription ID, if you use service principal to access your Azure environments. If you use system-assigned managed identity, leave it blank.

Tenant ID

Directory (tenant) ID. If you use system-assigned managed identity, leave it blank. If you use service principal identity, open Azure portal > navigate to Azure Active Directory > select App Registrations > select application created before, e.g. edfn-agent > copy Directory (tenant) ID from the Overview page.

Application ID

Application (client) ID. If you use system-assigned managed identity, leave it blank. If you use service principal identity, open Azure portal > navigate to Azure Active Directory > select App Registrations > select application created before, e.g. edfn-agent > copy Application (client) ID from the Overview page.

Secret key

Application (client) secret key. Paste secret key created before. If you use system-assigned managed identity, leave it blank. If you haven't created a client secret yet, open Azure portal > navigate to Azure Active Directory > select App Registrations > select application created before, e.g. edfn-agent > select Certificates & secrets > click on Client secrets tab and then create a new one.

Concurrency

The number of parallel message processors. Increase this value when blobs are consumed too slowly. Decrease this value, when NFO Server drops some input flows.

Max IPFIX Packet Size

This is NFO internal parameter – maximum IPFIX UDP message size. It is expected to be less or equals to MTU. When NFO and EDFN are installed on the same host, the parameter may be increased up to 3900 to increase processing speed.

note

Azure default URLs can be overwritten. Add $EDFN_HOME/conf/azure.properties file with the following content:

managementURI=https://management.core.windows.net/
baseURL=https://management.azure.com/
authURL=https://login.microsoftonline.com/
graphURL=https://graph.windows.net/
storageEndpointSuffix=.core.windows.net

Storage Accounts Tab

Open Storage accounts section to authorize access to NSG Flow logs in Azure Storage.

Storage account name and Queue name are required. Access key is optional if you use AD authentication (service principal or system-assigned managed identity) for storage account access.

The storage account has two access keys, please use the first one. The second key can be used during first key regeneration. Access keys are available on Storage account page -> Access keys in the left panel.

Enter Storage account name, Access key, if needed, and Queue name.

Press "Save".

Verify Azure Access and Set IPFIX Exporters

Press the “Run now” button to retrieve the list of subscriptions and associated virtual network names (IPFIX exporters section). This action will close the input configuration. It may take from several seconds to a few minutes to retrieve the list.

Open the IPFIX Exporters section to review and assign an exporter IP to each Virtual Network Name. This IP will be reported as exp_ip= field in syslogs. It is used for compatibility with physical network device *flow reporting in visualizations and alerting.

Azure Modules and Network Conversations Module Output Fields

The table below provides a side-by-side comparison of fields available in native NSG flow logs and NFO enriched output fields from various Modules.

NSG Flow Logs V2 Native Fields	Enriched 20401 Output	Enriched 20467 Output	Enriched 20062 Output	Field Description
	nfc_id	nfc_id	nfc_id	Message type identifier
			flow_type	Type of flow
	exp_ip	exp_ip	exp_ip	Subnet virtual router ipv4 address
time	time	time		Time in UTC when the event was logged.
Time stamp	flow_time	flow_time		Time stamp of when the flow occurred in UNIX epoch format.
Source IP	src_ip	src_ip	src_ip	Source IP address.
Destination IP	dest_ip	dest_ip	dest_ip	Destination IP address.
Source port	src_port	src_port	src_port	Source port.
Destination port	dest_port	dest_port	dest_port	Destination port.
Protocol	protocol	protocol	protocol	Protocol of the flow. Valid values are T for TCP and U for UDP.
Traffic flow	direction	direction	direction	Direction of the traffic flow or origiinating flow
Traffic decision	decision	decision	action	Whether traffic was allowed or denied. Valid values are A for allowed and D for denied.
Flow State	state	state	state	State of the flow. Possible states are: B - Begin; C - Continuing for an ongoing flow; E - End when a flow is terminated.
Packets sent	packets_in	packets_in	packets_in	Total number of TCP packets sent from source to destination since the last update.
Bytes sent	bytes_in	bytes_in	bytes_in	Total number of TCP packet bytes sent from source to destination since the last update. Packet bytes include the packet header and payload.
Packets received	packets_out	packets_out	packets_out	Total number of TCP packets sent from destination to source since the last update.
Bytes received	bytes_out	bytes_out	bytes_out	Total number of TCP packet bytes sent from destination to source since the last update. Packet bytes include packet header and payload.
	src_host	src_host	src_host	Source host name
	dest_host	dest_host	dest_host	Destination host name
		src_subs_id	azure_src_subs_id	Source subscription ID
		src_subs_name	azure_src_subs_name	Source subscription name
		src_vm_name	azure_src_vm_name	Source VM name
		src_nsg_name	azure_src_nsg_name	Source NSG name
		src_vnet_name	azure_src_vnet_name	Source virtual network name
		src_subnetwork_name	azure_src_subnet_name	Source subnet name
		src_service	azure_src_service	Source service
		src_region	src_region	Source GEO region
			src_cloud_region	Source Azure cloud region
		src_res_grp_name	azure_src_res_grp_name	Source resource group name
		dest_subs_id	azure_dest_subs_id	Destination subscription ID
		dest_subs_name	azure_dest_subs_name	Destination subscription name
		dest_vm_name	azure_dest_vm_name	Destination VM name
		dest_nsg_name	azure_dest_nsg_name	Destination NSG name
		dest_vnet_name	azure_dest_vnet_name	Destination virtual network name
		dest_subnetwork_name	azure_dest_subnet_name	Destination subnet name
		dest_service	azure_dest_service	Destination service
		dest_region	dest_cloud_region	Destination Azure cloud region
		dest_res_grp_name	azure_dest_res_grp_name	Destination resource group name
		flow_count	flow_count	Number of consolidated flows
		percent_of_total		Percent of total traffic volume within virtual router
		flow_start_time	flow_start_time	Start time of the first consolidated flow
		flow_end_time	flow_end_time	End of the last consolidated flow
		t_int	t_int	Observation time interval, msec
			exp_name	Name of NSG flow exporter
mac			src_mac	Source MAC address
mac			dest_mac	Destination MAC address
			threat_list_name	Name of a cybersecurity threat list
			reputation	Reputation from the treat list
			idp	User identity provider
			username	User name from login events (AD, Azure AD, Okta, etc)
			app_id	Application ID
			app_name	Application name
			app_desc	Application description
			src_as	Source AS
			src_cc	Source country code
			src_region	Source GEO region
			src_city	Source city
			src_lon	Source longitude
			src_lat	Source latitude
			dest_as	Destination AS
			dest_cc	Destination country code
			dest_region	Destination GEO region
			dest_city	Destination city
			dest_lon	Destination longitude
			dest_lat	Destination latitude
systemId				System ID of the network security group.
category				Category of the event. The category is always NetworkSecurityGroupFlowEvent.
resourceid				Resource ID of the network security group.
operationName				Always NetworkSecurityGroupFlowEvents.
Version				Version number of the flow log's event schema.
rule				Rule for which the flows are listed.
mac				MAC address of the NIC for the VM where the flow was collected.