Skip to main content
Version: Next

Data Ingestion Overview

Data Ingestion is the entry point for all telemetry processed by NetFlow Optimizer (NFO). The platform is designed to handle diverse data formats, ranging from traditional network flows to modern cloud logs and streaming telemetry, normalizing them into a unified format for analysis and enrichment.

Ingestion Architecture​

NFO acts as a high-performance collector that accepts push-based telemetry (Flows, MDT) and performs pull-based ingestion (Cloud, SNMP) via the External Data Feeder (EDFN). Once ingested, all data is processed through the NFO engine's in-memory pipeline for real-time normalization.

Supported Ingestion Types​

Network Flow Ingestion​

The core of NFO's capabilities. Support for all major industry-standard flow protocols.

  • Protocols: NetFlow v5/v9, IPFIX, sFlow, and J-Flow.
  • Use Case: Comprehensive visibility into on-premises network traffic and bandwidth utilization.

Cloud Flow Logs​

Native integration with public cloud infrastructure to provide a unified view of hybrid environments.

  • Platforms: AWS VPC Flow Logs, Azure NSG Flow Logs, Google Cloud (GCP), and Oracle Cloud (OCI).
  • Mechanism: Automated ingestion of log files from cloud storage buckets or event hubs.

Streaming Telemetry (MDT)​

Support for modern Model-Driven Telemetry (MDT) used in high-density carrier-grade equipment.

  • Connectivity: Dial-out (push) model using gRPC or UDP.
  • Structure: Ingests structured data defined by YANG models for sub-second visibility into device performance.

SNMP & Device Discovery​

Infrastructure-level visibility that complements flow data.

  • Discovery: Automatically identifies network devices and maps interface indexes (ifIndex) to human-readable names.
  • Polling: Collects interface statistics, CPU/Memory health, and system metadata.

Data Enrichment​

The "Value-Add" layer that transforms raw telemetry into actionable intelligence.

  • Metadata: Adds GeoIP, Threat Intelligence, User Identity (Active Directory), and VM metadata (VMware vCenter) to ingested records.

Ingestion Workflow​

  1. Configure Source: Set your network devices or cloud providers to send data to the NFO IP/Port or storage location.
  2. Verify Ingestion: Use the NFO Status dashboard to monitor the Input Rate (Packets per second) for each source.
  3. Apply Logic: Enable NFO Modules to filter, aggregate, or enrich the incoming data.
  4. Route to Output: Send the processed results to your preferred analytics platform.

Next Step​

To begin your configuration, start with Network Flow Ingestion to set up your primary network telemetry streams.