SNMP and NetFlow Data Monitoring Content Pack for ITSI

Introduction

In IT environments, many critical issues stem from factors beyond the application stack, often originating within network devices, among other components. This document, focused on Network Devices, offers comprehensive insights and methods for mapping these dependencies within Splunk IT Service Intelligence (ITSI).

The image below displays all the services contained in the content pack, showcased within the Service Analyzer:

The following section provides detailed instructions for the installation and configuration of the 'SNMP and NetFlow Data Monitoring Content Pack for ITSI.' This Pack relies on SNMP polling and optionaly NetFlow data sent to Splunk by NetFlow Optimizer (NFO).

This content pack includes:

• Entity Types
• KPI base searches
• Service templates
• Services

Entity Types

There are two Entity Types:

• Network Device: his type is intended for grouping entities that serve as network devices, such as routers, switches, firewalls, and more
• Network Interface: This type is used to categorize entities that represent interfaces on network devices

KPI Base Searches

The content pack includes searches for data in both the events index and the metrics index.

Data in Events Index

• Network Device CPU Utilization: This metric, used by Network Device-related services, displays the device's CPU utilization as a percentage. The data is obtained through SNMP queries (nfc_id=20103).
• Network Device Memory Utilization: Utilized by Network Device-related services, this metric indicates the percentage of memory utilization on the device. The data is gathered via SNMP queries (nfc_id=20103).
• Interface Errors and Discards: Employed by the Network Interface Health service, this metric showcases the percentage of errors and discards in relation to all transmitted and received packets on the interface. The data is collected through SNMP queries (nfc_id=20103).
• Interface Relative Load - Netflow based: Utilized by the Network Interface Health service, this metric demonstrates the percentage of current traffic on the interface compared to its speed. Interface speed is determined through SNMP queries, while current traffic is based on NetFlow data (nfc_id=20067).
• Interface Relative Load – SNMP: Utilized by the Network Interface Health service, this metric demonstrates the percentage of current traffic on the interface compared to its speed. Interface speed and current traffic is determined through SNMP queries (nfc_id=20103).

Data in Metrics Index

• Network Device CPU Utilization Metrics: This metric, used by Network Device-related services, displays the device's CPU utilization as a percentage. The data is acquired through SNMP queries (nfc_id=20103).
• Network Device Memory Utilization Metrics: Utilized by Network Device-related services, this metric indicates the percentage of memory utilization on the device. The data is gathered via SNMP queries (nfc_id=20103).
• Interface Errors and Discards Metrics: Employed by the Network Interface Health service, this metric showcases the percentage of errors and discards in relation to all transmitted and received packets on the interface. The data is collected through SNMP queries (nfc_id=20103).
• Interface Relative Load – SNMP polling based Metrics: Utilized by the Network Interface Health service, this metric demonstrates the percentage of current traffic on the interface compared to its speed. Interface speed and current traffic is determined through SNMP queries (nfc_id=20103).
• Interface Relative Load - Netflow based Metrics: Used by the Network Interface Health service, this metric illustrates the percentage of the interface's current traffic in comparison to its speed. Interface speed is determined through SNMP queries, while current traffic is based on NetFlow data (nfc_id=20067).

Service Templates

There is one service template: Network Device. This template is intended for the following four services: Network Firewall, Network LoadBalancer, Network Router, and Network Switch. It enables you to define specific KPIs for each device type.

Services

The following services are defined in the Content Pack:

• Network Firewall
• Network LoadBalancer
• Network Router
• Network Switch
• Network Interface Health

The first four services are based on Network Device service template and include the following four KPIs:

• Network Device CPU Utilisation
• Network Device CPU Utilisation Metrics
• Network Device Memory Utilisation
• Network Device Memory Utilisation Metrics

The Network Interface Health includes the following six KPIs:

• Interface Errors and Discards
• Interface Errors and Discards Metrics
• Interface Relative Load - NetFlow based
• Interface Relative Load - NetFlow based Metrics
• Interface Relative Load - SNMP
• Interface Relative Load - SNMP Metrics

Prerequisites

Please be sure to have the following before you begin the installation of the Content Pack:

1. NetFlow Optimizer (NFO) is installed in your environment
2. Top Traffic Monitor Module is enabled (for optional NetFlow processing)
3. SNMP Information Monitor is enabled (for data about interfaces, such as ifName, ifSpeed, etc.)
4. SNMP Custom OID Sets Monitor Module is enabled (for SNMP polling of CPU and Memory utilization)
5. NFO output is configured to send data to Splunk event index or metrics index

The Content Pack relies on the following:

• The data from SNMP polling, and optionally NetFlow data, is ingested into Splunk, either in the event index as defined by the nfo_netflow_index macro or in the metrics index as defined by the nfo_netflow_index_metrics macro. Both macros are provided within the Content Pack.

• The following fields are expected in SNMP polling events (nfc_id=20103):

  • sysName
  • exp_ip
  • nfo_hostname
  • mgmt_ip
  • cpu_load_percent
  • mem_used_percent
  • mem_used
  • mem_free
  • mem_total
  • ifName
  • ifIndex
  • ifInErrors
  • ifOutErrors
  • ifInDiscards
  • ifOutDiscards
  • ifInUcastPkts
  • ifOutUcastPkts
  • ifInNUcastPkts
  • ifOutNUcastPkts
  • ifHCInOctets
  • ifHCOutOctets
  • ifHighSpeed

note

For memory utilization you may need either mem_used_percent or any two of the following OIDs: mem_used, mem_free, or mem_total

• The following fields are expected in NetFlow events (nfc_id=20067) (if NetFlow data is used):
  • exp_ip
  • input_snmp
  • output_snmp
  • nfo_hostname
  • flow_smpl_id

Installation and Configuration Steps

In this section, you'll find the step-by-step installation and configuration instructions.

Download the Content Pack by clicking on the following link: https://splunk-netflow.s3.us-west-2.amazonaws.com/itsi-cp/DA-ITSI-CP-CUST-64e79cc8cadcb104d97b7703_1_0_6.tar.gz

1. Install the Content Pack into Splunk as an App

2. Import the Content Pack

In ITSI app go to Configuration>Data Integrations>Content Library and import the new Content Pack

3. Configure Index

1. If you are using events based index: In Settings->Advanced search->Search macros find nfo_netflow_index macro and change it to point to your events index
2. If you are using metrics based index: In Settings->Advanced search->Search macros find nfo_netflow_index_metrics macro and change it to point to your metrics index

4. Import Network Devices as Entities

Import Network Devices as Entities using ad hoc search in case of events based index:

`nfo_netflow_index` nfc_id=20103
    | stats count by nfo_hostname mgmt_ip exp_ip sysName
    | eval device = nfo_hostname." - ".mgmt_ip." / ".exp_ip." (".sysName.")"  
    | eval entity_type="Network Device"
    | table device exp_ip mgmt_ip nfo_hostname sysName entity_type

Import Network Devices as Entities using ad hoc search in case of metrics based index:

| mstats count(t_int)
    WHERE  `nfo_netflow_index_metrics` nfc_id=20103
    BY  mgmt_ip exp_ip nfo_hostname sysName      
    | eval device = nfo_hostname." - ".mgmt_ip." / ".exp_ip." (".sysName.")"  
    | eval entity_type="Network Device"
    | table device exp_ip mgmt_ip nfo_hostname sysName entity_type

Go to: Configuration>Entity Management>Create Entity>Import from Search:

Import:

• device as Entity Title
• exp_ip as Entity Information Field
• mgmt_ip as Entity Information Field
• nfo_hostname as Entity Information Field
• sysName as Entity Information Field
• entity_type as Entity Type Field

5. Enable Services

Enable one or more Network services and assign entities to them: Configuration>Services>Network Router>Entities.

Select a few based on Entity Title.

Enable the service when saving it.

6. Import interface entities

There are several options to import interface entities:

1. From NetFlow data in events index
2. From SNMP data in events index
3. From SNMP data in metrics index

Use the following search to import all interfaces from NetFlow data in events index reported by Top Traffic Minitor Module (nfc_id=20067):

`nfo_netflow_index` nfc_id=20067
    | stats count by nfo_hostname exp_ip input_snmp output_snmp
    | eval snmps=input_snmp+","+output_snmp
    | eval snmps=split(snmps,",")
    | search snmps = "*"
    | `nfo_get_iface_name(snmp_name, snmps)`
    | `nfo_get_exporter_name(exp_ip_name)`
    | strcat exp_ip_name "/" snmp_name interface
    | dedup interface
    | eval interface_name=nfo_hostname."-".interface
    | lookup interfaces_20003_lookup nfo_hostname exp_ip OUTPUT SNMPDeviceName
    | eval sysName =mvindex(SNMPDeviceName,0)
    | fillnull value="unknown" sysName
    | eval entity_type = "Network Interface"
    | table interface_name interface exp_ip nfo_hostname sysName entity_type

The recommendation is to limit the number of interface entities, as there could be too many for ITSI to handle efficiently. The following search applies the critical_interfaces_lookup, where you can specify the list of interfaces to monitor:

`nfo_netflow_index` nfc_id=20067 [ | inputlookup critical_interfaces_lookup| rename management_ip as exp_ip  | fields exp_ip nfo_hostname | format ]
    | stats count by nfo_hostname exp_ip input_snmp output_snmp
    | eval snmps=input_snmp+","+output_snmp
    | eval snmps=split(snmps,",")
    | mvexpand snmps
    | search [ | inputlookup critical_interfaces_lookup | rename management_ip as exp_ip | rename snmp_index as snmps   | fields exp_ip nfo_hostname snmps | format ]
    | `nfo_get_iface_name(snmp_name, snmps)`
    | `nfo_get_exporter_name(exp_ip_name)`
    | strcat exp_ip_name "/" snmp_name interface
    | dedup interface
    | eval interface_name=nfo_hostname."-".interface
    | lookup interfaces_20003_lookup nfo_hostname exp_ip OUTPUT SNMPDeviceName
    | eval sysName =mvindex(SNMPDeviceName,0)
    | fillnull value="unknown" sysName
    | eval entity_type = "Network Interface"
    | table interface_name interface exp_ip nfo_hostname sysName entity_type

Here is an example of critical_interfaces.csv lookup file:

nfo_hostname,"management_ip","snmp_index","comment"
localhost","10.0.3.2",1,"Important interface"
localhost",10.0.5.22,116,"Uplink"

Where:

nfo_hostname - is the name of NFO host

management_ip - is the SNMP polling IP address of the device

snmp_index - is the index of the interface (received from SNMP polling)

comment - internal comment

Use the following search to import all interfaces from SNMP data in events reported by SNMP Custom OID Sets Monitor (nfc_id=20103):

`nfo_netflow_index` nfc_id=20103
    | stats count by nfo_hostname exp_ip sysName ifIndex ifName mgmt_ip
    | eval device_sysname = (((exp_ip . " (") . sysName) . ")")
    | eval interface = if(ifIndex==ifName,ifIndex,(((ifIndex . " ( ") . ifName) . " )"))
    | eval interface_name =nfo_hostname."-".device_sysname."/".interface
    | rename mgmt_ip as device_ip
    | eval entity_type = "Network Interface"      
    | table interface_name interface exp_ip device_ip nfo_hostname sysName entity_type

The recommendation is to limit the number of interface entities, as there could be too many for ITSI to handle efficiently. The following search applies the critical_interfaces_lookup, where you can specify the list of interfaces to monitor:

`nfo_netflow_index` nfc_id=20103 [ | inputlookup critical_interfaces_lookup | rename management_ip as mgmt_ip | rename snmp_index as ifIndex   | fields mgmt_ip nfo_hostname ifIndex | format ]
    | stats count by nfo_hostname exp_ip sysName ifIndex ifName mgmt_ip
    | eval device_sysname = (((exp_ip . " (") . sysName) . ")")
    | eval interface = if(ifIndex==ifName,ifIndex,(((ifIndex . " ( ") . ifName) . " )"))
    | eval interface_name =nfo_hostname."-".device_sysname."/".interface
    | rename mgmt_ip as device_ip
    | eval entity_type = "Network Interface"      
    | table interface_name interface exp_ip device_ip nfo_hostname sysName entity_type

Use the following search to import all interfaces from SNMP data in metrics index reported by SNMP Custom OID Sets Monitor (nfc_id=20103):

| mstats count(t_int)
    WHERE  `nfo_netflow_index_metrics` nfc_id=20103
    BY  nfo_hostname exp_ip sysName ifIndex ifName mgmt_ip
    | eval device_sysname = (((exp_ip . " (") . sysName) . ")")
    | eval interface = if(ifIndex==ifName,ifIndex,(((ifIndex . " ( ") . ifName) . " )"))
    | eval interface_name =nfo_hostname."-".device_sysname."/".interface
    | rename mgmt_ip as device_ip
    | eval entity_type = "Network Interface"      
    | table interface_name interface exp_ip device_ip nfo_hostname sysName entity_type

The recommendation is to limit the number of interface entities, as there could be too many for ITSI to handle efficiently. The following search applies the critical_interfaces_lookup, where you can specify the list of interfaces to monitor:

| mstats count(t_int)
    WHERE  `nfo_netflow_index_metrics` nfc_id=20103
    BY  nfo_hostname exp_ip sysName ifIndex ifName mgmt_ip
    | search [ | inputlookup critical_interfaces_lookup | rename management_ip as mgmt_ip | rename snmp_index as ifIndex   | fields mgmt_ip nfo_hostname ifIndex | format ]
    | eval device_sysname = (((exp_ip . " (") . sysName) . ")")
    | eval interface = if(ifIndex==ifName,ifIndex,(((ifIndex . " ( ") . ifName) . " )"))
    | eval interface_name =nfo_hostname."-".device_sysname."/".interface
    | rename mgmt_ip as device_ip
    | eval entity_type = "Network Interface"      
    | table interface_name interface exp_ip device_ip nfo_hostname sysName entity_type

Import:

• interface_name as Entity Title

Optionally:

• exp_ip as Entity Information Field
• interface as Entity Information Field
• device_ip as Entity Information Field
• nfo_hostname as Entity Information Field
• device_sysname as Entity Information Field
• entity_type as Entity Type Field

7. Enable Network Interface Health Service

Enable Network Interface Health service and assign entities to them:

Configuration>Services>Network Interface Health>Entities  

Receiving SNMP Traps as Universal Alerting in the Content Pack for ITSI Monitoring and Alerting

For more information on Universal Alerting in the Content Pack for ITSI Monitoring and Alerting, visit https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/UniversalAlerting.

1. Make sure that you installed TA-netflow v4.5.17 or later

   This version has a fix for src field. The link to this TA is: https://splunkbase.splunk.com/app/1838

2. Install the Content Pack for ITSI Monitoring and Alerting

   For details, visit https://docs.splunk.com/Documentation/CPITSIMonitorAlert/latest/CP/Install

3. Disable and enable the relevant ITSI components

   • Disable all existing ITSI correlation searches, especially if custom-built searches exist for the alert sources that you will normalize. If the searches remain enabled, the resulting Notable Events will be duplicated or otherwise confusing
   • Disable all Notable Event Aggregation Policies (NEAPs), especially custom-built policies for the alert sources that you are normalizing. If they remain enabled, the resulting episodes will be duplicated or otherwise confusing
   • Later, thoughtfully enable the correlation searches and NEAPs, if they don't overlap with the new Universal components
   • Enable the ITSI correlation searches:
     • Universal Correlation Search
     • Episode Monitoring - Set Episode to Highest Alarm Severity
   • Enable the Notable Event Aggregation Policies:
     • Episodes by Alarm
     • Episodes by Alert Group
     • Episodes by ITSI service
     • Episodes by src

4. Send SNMP traps to the NFO

Installing and Configuring SNMP Agent and Utilities

The installation and configuration of SNMP Agent and Utilities are optional components that provide additional capabilities to your NetFlow Optimizer setup.

The SNMP Agent is essential for SNMP polling of the host, enabling you to collect valuable data for monitoring CPU and memory utilization. Additionally, SNMP utilities offer useful tools for troubleshooting purposes.

SNMP Agent Installation

1. Execute the following command:

RHEL

yum install -y net-snmp

Ubuntu

sudo apt-get install snmpd

2. Set the SNMP Read-Only Community String as 'public' by adding the line below to the configuration file (/etc/snmp/snmpd.conf):

rocommunity public

3. Start the snmpd service:

systemctl enable snmpd && systemctl start snmpd

Allowing SNMP Ports in Firewall

Execute the following commands:

RHEL

firewall-cmd --zone=public --add-port=161/udp --permanent
firewall-cmd --zone=public --add-port=162/udp --permanent
firewall-cmd --reload

Ubuntu

sudo ufw allow 161/udp
sudo ufw allow 162/udp

SNMP Utilities Installation

You may want to install SNMP Utilities to test your SNMP polling, e.g. snmpwalk command.

Execute the following command:

RHEL

yum install net-snmp-utils

Ubuntu

sudo apt-get install snmp

snmpwalk Examples

$ snmpwalk -v 2c -c public -O e 127.0.0.1  1.3.6.1.2.1.31.1.1.1.1
IF-MIB::ifName.1 = STRING: lo
IF-MIB::ifName.2 = STRING: eth0
$ snmpwalk -v 2c -c public -O e 127.0.0.1  1.3.6.1.2.1.31.1.1.1.15
IF-MIB::ifHighSpeed.1 = Gauge32: 10
IF-MIB::ifHighSpeed.2 = Gauge32: 0
$ snmpwalk -v 2c -c public -O e 127.0.0.1  1.3.6.1.2.1.31.1.1.1.6
IF-MIB::ifHCInOctets.1 = Counter64: 218081865900
IF-MIB::ifHCInOctets.2 = Counter64: 308231409749
$ snmpwalk -v 2c -c public -O e 127.0.0.1  1.3.6.1.2.1.31.1.1.1.10
IF-MIB::ifHCOutOctets.1 = Counter64: 218081872093
IF-MIB::ifHCOutOctets.2 = Counter64: 410216133901