Azure Blob Storage Syslog or JSON

Use this output type to send NFO data to Azure Blob Storage account container. Logs may be in Syslog or JSON format. Logs are separated by a new line character. Blob may be uncompressed or GZIP compressed.

NFO supports two types of blobs: Append blob and Block blob.

For Azure Blob Storage output configuration you need Storage account name or Connection string, Access key or SAS (Shared access signature) token, Container name. When NFO is installed in the Azure environment, NFO may authenticate without access key. When NFO in installed on VM in the Azure, System-assigned managed identity authentication may be used.

Access key is available here: Open Azure portal > Storage accounts > select storage account name > on the left panel navigate to “Access keys” from “Security + networking” items group > show and copy key1 value.

We recommend to use Storage account name, and access key or SAS token instead of Connection string. Connection string may be used when custom endpoints are required or Azure Storage Emulator is used for testing purposes.

info

Connection string is stored unencrypted and value is printed in the logs.

Azure Blob Storage output has the following parameters.

Parameter	Description
Storage Account	Azure storage account name or Connection string, e.g. DefaultEndpointsProtocol=https or AccountName=storagename or EndpointSuffix=core.windows.net. Account name is preferred, but if you have to use Connection string, for security reasons, we recommend to enter Connection string without AccountKey or SharedAccessSignature properties. Use Access key parameter below.
Access key	Access key or SAS (Shared access signature). If not specified, managed identity authentication is used.
Container name	Azure data storage container name.
Blob name	Blob resource name, this may be a constant string or a pattern like nfo_${nfc_id}_${exp_ip}, where nfc_id and exp_ip are substituted from the syslog/JSON message. The full resource name will be ${HOSTNAME}/${NFO_UUID}/yyyy/MM/dd/HH/${blob_name}.yyyyMMddHHmmssSSS.log[.gz].
Block put interval (sec)	Block flush interval, default 300 seconds (5 minutes). Block is saved when one of these conditions are met: block hasn’t been saved during the interval, or block size is reached the limit (4MiB for Append Blobs and 8MiB for Block Blobs), or blob has to be rotated (end of the hour). Blobs are rotated at the beginning of each hour.
Compression	gzip or no compression. If gzip is selected (default), each block is compressed independently: each block has gzip header and trailer, so it can be read without previous blocks.
Blocks	Multiple blocks (up to 50,000) or single block. Due to Append Blob block limit, a single block Append blob has 4MiB size limit. For Block Blob NFO use 8MiB blocks, but in case of single block, up to 100MiB block is used.
Blob type	Block Blob or Append Blob type. For details, visit https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs. NFO uses single append operation for added a new Append blob block. When append blob is rotated, it is sealed, making it read only. NFO uses two operations: stageBlock and commitBlockList for adding a new blob block.
Max blob size MiB	Maximum blob size in MiB. Default value is 100MiB, minimum 1MiB, maximum 200,000MiB.
Report threads	Output threads count (default is 2). This is the number of threads allocated to process messages produced by NFO and sent to Azure Blob Storage.
Report interval (sec)	Time interval in seconds between report threads executions (default is 10)
nfc_id filter	Comma separated list of NFO Modules’ nfc_ids to be send to Azure. This is an optional parameter, if not set, all messages are sent.

System-assigned Managed Identity Authentication

note

This section applies to NFO installation on Azure VM. For more information, visit https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/qs-configure-portal-windows-vm#enable-system-assigned-managed-identity-on-an-existing-vm

1. Open the Azure portal
2. Navigate to Virtual machines
3. From the list of VMs, select the VM, where NFO is installed
4. Select Identity, under the Settings group in the left panel
5. Under the System assigned tab, Status, select On and then click Save
6. Confirm enable the system assigned managed identity for the VM

Even though NFO may be authenticated with the “Access key”, we recommend using AD authentication. The Virtual machine must have Storage Blob Data Owner role to access storage account container. Instruction how to configure managed identity access is available here: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-linux-vm-access-storage#grant-your-vm-access-to-an-azure-storage-container.

1. Navigate to your storage account or storage account container
2. Select Access control (IAM) in the left panel
3. Select + Add and choose Add role assignment
4. On the Role tab, choose Storage Blob Data Owner
5. On the Members tab select Managed Identity radio button, and click + Select members link
6. On the right panel select subscription, Managed identity: Virtual Machine, then choose Virtual Machine
7. Click Select
8. Click Review + assign to save assignment

After these steps NFO writes to the storage container without an access key.

Instead of Storage Blob Data Owner role, custom role may be used. For details, visit https://learn.microsoft.com/en-us/azure/role-based-access-control/custom-roles-portal.

1. Navigate to the Storage account or Storage account container
2. Open Access Control (IAM), Roles tab
3. Find Storage Blob Data Owner, select … and click Clone
4. Enter Custom role name, for example: NFO Storage Blob Data Writer
5. Open JSON tab and modify permissions section as follows:

"permissions": [
  {
    "actions": [
      "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action",
      "Microsoft.Storage/storageAccounts/blobServices/containers/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/write"
    ],
    "notActions": [],
    "dataActions": [
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write",
      "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action"
    ],
    "notDataActions": []
  }
]

6. Click Review + update, then Update.