v12 dashboard: NetFlow

[kryonsx]

Part of: #2696 · Needs first: #2697 (click-through to the Log Explorer and the value/count table)
Related: #1736 (parsers and rules for this integration, owned by the detection team)

Goal

Ship a built-in NetFlow dashboard that shows, at a glance, how much NetFlow is sending, whether it is still sending, and what kinds of events they are. The heart of it is the list of NetFlow logs grouped by destinations (widget W7): click one and the Log Explorer opens on exactly those logs.

Where the data comes from

Integration (catalog name)	NETFLOW
Data type	netflow
How the logs arrive	Routers, switches and firewalls export NetFlow v1/v5/v6/v7/v9 or IPFIX packets to the forwarder on UDP 2055, which decodes them and sends one key="value" text line per flow record.
Parser	definitions/filters/netflow/netflow.yaml
What dataSource holds	The exporter's UDP address as 'IP:port' (for example '10.0.0.1:52345'), not just the IP. Proof: collectors/forwarder/collector/netflow/netflow.go line 314 passes addr.String() to processMessage, and parser.go lines 59-63 set DataSource: remote. The exporter IP alone is also stored as log.exporter (goflow.go / tehmaze.go set NFSender from net.SplitHostPort).
Grouped by	target.ip (destinations)

Why target.ip: A flow has no event name. The natural grouping would be the destination port (it names the service), with protocol next, but the parser breaks both today: destination ports are missing for NetFlow v5/v1/v6/v7 and wrong for v9/IPFIX ports of 256 and above (443 is stored as 1187), and protocol is set only for v9/IPFIX (see parserIssues). The destination IP is the least-bad choice: it is present and correct on every flow from every NetFlow version, it answers 'where is this traffic going', and clicking one opens every flow to that host. Cardinality is moderate to high, so it is shown as a top-25 list.

Typical values: 8.8.8.8, 10.10.1.20, 142.250.72.14

Widgets

Standard layout from the parent issue; rows W2, W3 and W9 onward are specific to this integration.

#	Title	Shown as	Query	Why
W1	Total flows	number	logs: count	Number of flow records received; each record is one log.
W2	Flows to internet addresses	number	logs: count; filter target.geolocation.country exists	Flows whose destination is a public IP (geolocation is only added to public IPs).
W3	Flows from internet addresses	number	logs: count; filter origin.geolocation.country exists	Flows whose source is a public IP, including replies to outbound connections.
W4	Alerts	number	alerts: count	Alerts raised by the 12 shipped NetFlow rules (dataType netflow).
W5	Flow volume over time	area chart	logs: count over time	Flow count per time bucket; shows exporter outages and traffic spikes (counts flows, not bytes).
W6	Flows by exporter	bar chart	logs: top 10 values of log.exporter; filter log.exporter exists	Which devices send the most flows. Uses log.exporter (the exporter's IP address alone) rather than dataSource, which holds 'IP:port' and would split one device into several bars whenever its source port changes.
W7	Top destinations	value and count table	logs: top 25 values of target.ip; filter target.ip exists	The main list: the hosts receiving the most flows; click one to see every flow to it.
W8	Destinations over time (top 5)	line chart	logs: count over time, one line per value of target.ip (top 5); filter target.ip exists	Shows when traffic to the busiest destinations starts or spikes.
W9	Top talkers	bar chart	logs: top 10 values of origin.ip; filter origin.ip exists	Source IPs that open the most flows (scanners and infected hosts stand out).
W10	Destination countries	bar chart	logs: top 10 values of target.geolocation.country; filter target.geolocation.country exists	Where outbound traffic goes; unexpected countries are worth a look.
W11	Flows by protocol	bar chart	logs: top 10 values of protocol; filter protocol ≠	TCP, UDP, ICMP and others; only v9/IPFIX flows carry a protocol, so empty values are hidden.
W12a	Source countries	bar chart	logs: top 10 values of origin.geolocation.country; filter origin.geolocation.country exists	Countries that send traffic into the network.
W12b	Flows by NetFlow version	pie chart	logs: top 6 values of log.version; filter log.version exists	Six possible values; explains why protocol (W11) covers only part of the flows (v9 and IPFIX only).
W13	Alerts by rule	bar chart	alerts: top 10 values of name	Which NetFlow rules fire most.
W14	Alerts by severity	bar chart	alerts: top 10 values of severity	Split of NetFlow alerts into low, medium and high.
W15	Latest flows	table of latest logs	logs: latest 20 records; columns @timestamp, dataSource, target.ip, origin.ip, protocol, target.geolocation.country, log.version	The newest flow records; ports are left out on purpose because the stored values are wrong.

Fields used and where they come from

• target.ip: Destination IP of the flow. Examples: 8.8.8.8, 10.10.1.20. Source: kv step lines 18-20 writes log.dstIp; rename lines 149-152 to target.ip; quotes trimmed lines 163-193. Collector sets dstIp for every version (goflow.go lines 40-42 for v5, 155-160 and 175-180 for v9/IPFIX; tehmaze.go for v1/v6/v7).
• origin.ip: Source IP of the flow (the 'talker'). Examples: 192.168.1.34. Source: kv lines 18-20 writes log.srcIp; rename lines 145-148 to origin.ip; quotes trimmed lines 163-193.
• target.geolocation.country: Country name of a public destination IP. Private and local ranges get no geolocation. Examples: United States, Germany. Source: dynamic step lines 268-273 (com.utmstack.geolocation, destination target.geolocation); plugins/geolocation/geolocate.go IsLocal skips 10/8, 172.16/12, 192.168/16, 127/8, 169.254/16, 224.0.0.0/24; names from locations-en.csv (bases.go line 28); JSON key 'country' from go-sdk plugins.pb.go Geolocation.
• origin.geolocation.country: Country name of a public source IP. Examples: China, Netherlands. Source: dynamic step lines 260-265
• protocol: IP protocol name. Set only for NetFlow v9 and IPFIX flows; empty for v5, v1, v6, v7. Examples: TCP, UDP, ICMP, GRE, ESP. Source: grok lines 80-86 extracts log.protocol only from proto="/[n]" (the v9/IPFIX byte-list format); add steps lines 276-1133 map the IANA number to the name; log.protocol itself is deleted at line 1150.
• log.version: NetFlow version of the record. Examples: Netflow-V5, Netflow-V9, IPFIX, Netflow-V1, Netflow-V6. Source: kv lines 18-20, quotes trimmed lines 163-193; values set by the collector (goflow.go lines 20, 71, 96; tehmaze.go lines 20, 45, 74).
• log.exporter: IP of the exporting device, without the port. Examples: 10.0.0.1. Source: kv lines 18-20, quotes trimmed lines 163-193; collector NFSender from net.SplitHostPort (goflow.go line 15).
• target.port: Destination port. Not used in any widget: missing for v5/v1/v6/v7 and wrong for v9/IPFIX ports of 256 and above. Examples: 53, 1187 (really 443). Source: grok lines 96-102 (only matches dstPort="[a b]"), trims lines 195-230, cast lines 233-239, rename lines 250-257

Watch out for

• dataSource is 'IP:port' (the exporter's address plus its UDP source port), so W6 groups by log.exporter, which holds the IP alone. Use log.exporter for any per-device filter too.
• The dashboard counts flow records, not bytes or packets: a tiny flow and a 10 GB flow count the same. Byte and packet totals are not stored as numbers anyway (see parserIssues).
• Flow records are one-way, so one TCP connection usually gives two flows (request and reply). 'Flows from internet addresses' therefore includes replies to outbound connections.
• protocol exists only for NetFlow v9 and IPFIX flows; W11 hides the empty value and W12b shows how many flows come from each version.
• Geolocation is added only to public IPs (plugins/geolocation/geolocate.go IsLocal). 100.64.0.0/10 and private IPv6 ranges are not treated as local and may get an empty or odd result.
• No destination-port widget: it was the best candidate for the main list, but the stored ports are missing or wrong today (see parserIssues). Add 'Top destination ports' (target.port) once the parser is fixed.
• log.bytesIn, log.bytesOut, log.packetsIn and log.packetsOut are always '0' because the collector hardcodes them (goflow.go line 19).
• Not verified: that the kv step tolerates the v9/IPFIX values that contain spaces (for example bytes="[0 0 5 220]"). If it fails on them, v9/IPFIX flows would lose origin.ip and target.ip too. The grok and kv engines are not in the snapshot.

Parser problems found while designing this

These are not dashboard work, but they limit what the dashboard can show. They belong to #1736; raise them there rather than working around them in the dashboard.

• Ports for NetFlow v5, v1, v6 and v7: the grok steps (lines 88-102) only match bracketed values like dstPort="[0 53]". For these versions the v12 collector writes dstPort="443", which never matches, and the kv copies log.srcPort and log.dstPort are deleted at the end (lines 1142 and 1148). These flows have no origin.port or target.port.
• Ports for NetFlow v9 and IPFIX are wrong from 256 up: goflow2 returns the raw bytes and the collector formats them with %v (collectors/forwarder/collector/netflow/goflow.go lines 191-194), giving dstPort="[1 187]" for port 443. The parser removes a leading '0' and all spaces (lines 214-230) and casts to int, so 443 becomes 1187, 3389 becomes 1361, 8080 becomes 31144. Only ports below 256 come out right. Best fix: convert byte slices to integers in the collector's extractFieldValue.
• protocol is empty for NetFlow v5, v1, v6 and v7: the add steps (lines 276-1133) compare log.protocol to IANA numbers, but log.protocol only comes from the v9/IPFIX format proto="/[6]" (grok lines 80-86). For the other versions the collector already sends the name (proto="TCP", proto.go ProtoToName), but the parser deletes log.proto (line 1149) instead of copying it to protocol.
• Byte, packet, interface, mask, AS and TCP-flag values are never stored as usable fields: the kv copies are deleted (lines 1136-1150) and the grok copies exist only for v9/IPFIX and hold byte lists such as '0 0 5 220' (log.totalBytes).
• log.direction is not in the quote-trim lists (lines 163-193) and, for v9/IPFIX, holds '[0]' or '[1]' instead of Ingress/Egress because the collector compares the formatted byte slice to "0"/"1" (goflow.go lines 217-225).
• Most shipped NetFlow rules key on target.port, protocol, log.bytes or origin.bytesSent (definitions/rules/netflow), which are missing or wrong, so W4, W13 and W14 will stay low until the parser is fixed.
• Collector: dataSource includes the exporter's source port because the NetFlow listener passes addr.String() (collectors/forwarder/collector/netflow/netflow.go and parser.go), unlike the syslog listener, which strips it. Stripping it would make each device one data source.
• Collector: flow data that arrives before its NetFlow v9 or IPFIX template is dropped without any log line ("template not found" is filtered out in netflow.go). Logging it once per exporter would explain empty dashboards.

How to build it

Before you start: parser field names are changing while the parsers are updated for the engine's new underscore handling (see "Field names are about to move" in #2696). Check every field in this issue against the v12 parser at that moment and against real logs, and build against what you find.

1. Add definitions/dashboards/integration-netflow.yaml. Keep it in the top folder: the test that checks shipped dashboards (TestEveryShippedDashboardDefinitionIsValid) only reads the top folder.
2. Start from the file below; it follows the table above and passes the same checks as the backend (domain.Spec.Validate).
3. W7 is a value/count table: a category query shown with chart type table. It already renders; v12 dashboards: groundwork for integration dashboards (click-through to the Log Explorer, value/count table, Agents dashboard fix) #2697 makes its rows clickable and its headers readable.
4. Load real logs from this technology on a v12 test server (or replay samples) and check every widget before opening the pull request.

Starting dashboard file

# Dashboard version v1.0.0
#
# System-owned default dashboard for the NetFlow integration, seeded by
# backend/modules/dashboards/repository/dashboard_bootstrap.go.
# Field names come from definitions/filters/netflow/netflow.yaml.
# Verify every widget against real logs before shipping.
name: "NetFlow"
description: "What NetFlow is sending: volume, event types, and the activity worth a look."
widgets:
  - layout: { x: 0, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Total flows"

  - layout: { x: 3, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "target.geolocation.country"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Flows to internet addresses"

  - layout: { x: 6, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "origin.geolocation.country"
          op: exists
    config:
      __builder:
        chartType: metric
        title: "Flows from internet addresses"

  - layout: { x: 9, y: 0, w: 3, h: 2 }
    spec:
      dataset: alerts
      dataType: "netflow"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Alerts"

  - layout: { x: 0, y: 2, w: 8, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: time
      metric:
        agg: count
    config:
      __builder:
        chartType: area
        title: "Flow volume over time"

  - layout: { x: 8, y: 2, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "log.exporter"
      filters:
        - field: "log.exporter"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Flows by exporter"

  - layout: { x: 0, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "target.ip"
      filters:
        - field: "target.ip"
          op: exists
      limit: 25
    config:
      __builder:
        chartType: table
        title: "Top destinations"

  - layout: { x: 6, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: time
      metric:
        agg: count
      dimension: "target.ip"
      filters:
        - field: "target.ip"
          op: exists
      limit: 5
    config:
      __builder:
        chartType: line
        title: "Destinations over time (top 5)"

  - layout: { x: 0, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "origin.ip"
      filters:
        - field: "origin.ip"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top talkers"

  - layout: { x: 4, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "target.geolocation.country"
      filters:
        - field: "target.geolocation.country"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Destination countries"

  - layout: { x: 8, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "protocol"
      filters:
        - field: "protocol"
          op: not_eq
          value: ""
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Flows by protocol"

  - layout: { x: 0, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "origin.geolocation.country"
      filters:
        - field: "origin.geolocation.country"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Source countries"

  - layout: { x: 6, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "log.version"
      filters:
        - field: "log.version"
          op: exists
      limit: 6
    config:
      __builder:
        chartType: pie
        title: "Flows by NetFlow version"

  - layout: { x: 0, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "name"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by rule"

  - layout: { x: 6, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "netflow"
      chart: category
      metric:
        agg: count
      dimension: "severity"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by severity"

  - layout: { x: 0, y: 24, w: 12, h: 6 }
    spec:
      dataset: logs
      dataType: "netflow"
      chart: table
      metric:
        agg: count
      limit: 20
      columns: ["@timestamp", "dataSource", "target.ip", "origin.ip", "protocol", "target.geolocation.country", "log.version"]
    config:
      __builder:
        chartType: table
        title: "Latest flows"

Done when

• definitions/dashboards/integration-netflow.yaml is merged to release/v12.0.0 and go test ./modules/dashboards/... passes in backend/.
• With NetFlow logs flowing on a v12 test server, every widget shows data. A widget that stays empty while logs arrive means a wrong field or value: fix it, don't ship it.
• Clicking a row, bar or line point opens the Log Explorer on the same logs, and the Log Explorer count matches the widget.
• A time range with no logs shows empty states, not errors.
• A screenshot of the finished dashboard is attached to this issue.