v12 dashboard: FortiGate

[kryonsx]

Part of: #2696 · Needs first: #2697 (click-through to the Log Explorer and the value/count table)
Related: #1975 (parsers and rules for this integration, owned by the detection team)

Goal

Ship a built-in FortiGate dashboard that shows, at a glance, how much FortiGate is sending, whether it is still sending, and what kinds of events they are. The heart of it is the list of FortiGate logs grouped by event types (log ID) (widget W7): click one and the Log Explorer opens on exactly those logs.

Where the data comes from

Integration (catalog name)	FORTIGATE
Data type	firewall-fortigate-traffic
How the logs arrive	The FortiGate sends syslog over UDP (or TCP) port 7005 to the UTMStack forwarder, which passes each line unchanged; the setup guide configures 'config log syslogd setting' with 'set format default', so each line is followed by key=value pairs (date=... devname="..." logid="..." type="..." subtype="..." ...).
Parser	definitions/filters/fortinet/fortinet.yaml
What dataSource holds	The IP address the syslog message came from: the FortiGate itself, or a relay in between such as FortiAnalyzer (the forwarder's own host name when the sender is 127.0.0.1). Proof: collectors/forwarder/collector/syslog/handler.go resolveRemoteAddr (lines 21-33) and readLoop (line 79, TCP), listener.go lines 264-266 (UDP); handleMessage (lines 178-182) sends DataType: inst.DataType and DataSource: msgDS.DataSource; log-input/ingest/server.go applyDefaults (lines 158-160) only writes 'unknown' when it is empty; fortinet.yaml never writes dataSource. So W6 is 'Logs by firewall IP'. The FortiGate's own host name is in log.devname and its serial number in log.devid (default format, quotes removed by the trims at fortinet.yaml lines 351-436).
Grouped by	log.logid (event types (log ID))

Why log.logid: Every FortiOS log carries a 10-digit log ID (logid="0000000013") that names exactly one kind of message: 2 digits for the log type, 2 for the subtype and 6 for the message ID (FortiOS 7.2.7 Log Reference, 'Log ID definitions'). The key=value step (fortinet.yaml lines 24-27) writes it to log.logid on every line and the quote trims (lines 351-436) remove the quotes, so values are plain text such as '0000000013' (leading zeros kept, no cast). In CEF format the same value comes from FTNTFGTlogid (rename at lines 82-84). It is on 100% of FortiGate logs and a firewall usually produces a few dozen distinct IDs, so it groups well. It is a code, and the parser keeps no readable companion: FortiOS sends one (logdesc="Admin login failed"), but the key=value step splits on spaces and log.logdesc keeps only '"Admin' (see parser issues). The front end should show a fixed ID-to-name lookup next to the number (list below; meanings from the FortiOS 7.2.7 Log Reference, full ID = type code + subtype code + message ID). Traffic logs, usually the bulk, all share 0000000013 (forward) or 0001000014 (local), so W9-W12 split traffic further. The raw key 'logid' has no underscore, so this name does not depend on the underscore change.

Typical values: 0000000013 Forward traffic (session end), 0001000014 Local traffic (to or from the FortiGate), 0000000020 Forward traffic statistics (long session), 0100032001 Admin login successful, 0100032002 Admin login failed, 0100032003 Admin logout successful, 0100044546 Attribute configured (configuration change), 0100044547 Object attribute configured (configuration change), 0101039424 SSL VPN tunnel up, 0101039426 SSL VPN login fail, 0101037129 Progress IPsec phase 2, 0101037138 IPsec connection status changed, 0102043008 Authentication success, 0102043009 Authentication failed, 0211008192 Infected file detected and blocked, 0316013056 URL belongs to a blocked category, 0317013312 URL belongs to an allowed category, 0419016384 Attack detected by TCP/UDP signature (IPS), 0720018432 Attack detected by TCP/UDP anomaly (DoS policy), 1059028704 Application control: pass

Widgets

Standard layout from the parent issue; rows W2, W3 and W9 onward are specific to this integration.

#	Title	Shown as	Query	Why
W1	Total logs	number	logs: count	All FortiGate logs in the selected time range.
W2	Denied traffic	number	logs: count; filter log.type is one of [traffic]; action is one of [deny]	Traffic sessions a firewall policy denied (action 'deny'). actionResult is not used because the parser never sets it for this log format.
W3	Intrusion, virus and anomaly detections	number	logs: count; filter log.subtype is one of [ips, virus, anomaly]	Security-profile detections: intrusion signatures, viruses and DoS anomalies (these subtypes only exist on UTM logs).
W4	Alerts	number	alerts: count	Alerts raised from FortiGate logs.
W5	Log volume over time	area chart	logs: count over time	Shows gaps and spikes in what the firewalls send.
W6	Logs by firewall IP	bar chart	logs: top 10 values of dataSource	dataSource is the sending firewall's IP address, so this shows which firewall is busiest or silent.
W7	Top event types	value and count table	logs: top 25 values of log.logid; filter log.logid exists	The main list: FortiOS log IDs by count, shown with the name from the lookup; click one to open those logs.
W8	Event types over time (top 5)	line chart	logs: count over time, one line per value of log.logid (top 5); filter log.logid exists	When each of the five most common log IDs happened.
W9	Top firewall policies (ID)	bar chart	logs: top 10 values of log.policyid; filter log.type is one of [traffic]; log.policyid exists	Which policies match the most traffic; 0 is the implicit deny. The ID is used because policy names are cut at the first space.
W10	Top applications	bar chart	logs: top 10 values of log.app; filter log.type is one of [traffic]; log.app exists	Sessions per application as identified by the FortiGate.
W11	Denied traffic by destination port	bar chart	logs: top 10 values of target.port; filter log.type is one of [traffic]; action is one of [deny]; target.port exists	Which services blocked connections were aiming at; scans show up here (445, 3389, 23).
W12a	Denied traffic by source country	bar chart	logs: top 10 values of origin.geolocation.country; filter log.type is one of [traffic]; action is one of [deny]; origin.geolocation.country exists	Where denied connections from public IPs come from (private sources get no country).
W12b	Failed logins by user	bar chart	logs: top 10 values of log.user; filter log.logid is one of [0100032002, 0101039426, 0102043009]; log.user exists	Accounts with failed admin, SSL VPN and user-authentication logins; a spike on one name suggests password guessing.
W13	Alerts by rule	bar chart	alerts: top 10 values of name	Which FortiGate detection rules fire most.
W14	Alerts by severity	bar chart	alerts: top 50 values of severity	Split of FortiGate alerts into low, medium and high.
W15	Latest logs	table of latest logs	logs: latest 20 records; columns @timestamp, dataSource, log.logid, log.subtype, action, origin.ip, target.ip, target.port	The newest raw records with their log ID, subtype, action and addresses.

Fields used and where they come from

• log.logid: FortiOS log ID, text with leading zeros. Examples: 0000000013, 0100032002, 0419016384. Source: kv step lines 24-27 (key 'logid'); quote trims lines 357 and 400; CEF: rename log.FTNTFGTlogid -> log.logid lines 82-84
• log.type: Log type. Examples: traffic, utm, event. Source: kv step lines 24-27; quote trims lines 358 and 401; CEF: grok on log.cat ('traffic:forward') lines 174-182
• log.subtype: Log subtype: forward, local (traffic); ips, virus, webfilter, app-ctrl, anomaly, dns, ssl (utm); system, vpn, user (event). Examples: forward, ips, vpn. Source: kv step lines 24-27; quote trims lines 359 and 402; CEF: rename log.FTNTFGTsubtype -> log.subtype lines 91-93
• action: Action: accept, close, timeout, deny, server-rst, client-rst, ip-conn (traffic); detected, dropped, blocked, pass (utm); login, ssl-login-fail, edit (event). Examples: close, deny, blocked. Source: rename log.action / log.act -> action lines 227-231; quote trims lines 373 and 416
• log.policyid: Firewall policy ID, text. 0 is the implicit deny policy. Examples: 1, 0. Source: kv step lines 24-27 (FortiOS sends it without quotes); CEF: rename log.FTNTFGTpolicyid -> log.policyid lines 112-114
• log.app: Application name from application control. Examples: HTTPS.BROWSER, DNS, Microsoft.Teams. Source: kv step lines 24-27; quote trims lines 375 and 418; CEF: rename log.FTNTFGTapp -> log.app lines 130-132
• origin.ip: Source IP address. Examples: 10.1.100.11, 203.0.113.9. Source: rename log.srcip / log.src -> origin.ip lines 242-246
• target.ip: Destination IP address. Examples: 172.16.200.55. Source: rename log.dstip / log.dst -> target.ip lines 232-236
• target.port: Destination port (number). Examples: 443, 3389. Source: rename log.dstport / log.dpt -> target.port lines 237-241; cast to int lines 328-338
• origin.geolocation.country: Country name of a public source IP (UTMStack's own lookup). Examples: Germany, United States. Source: dynamic com.utmstack.geolocation, source origin.ip, destination origin.geolocation, lines 439-444; JSON key 'country' from the go-sdk Geolocation message
• log.user: User name, still wrapped in quotes and cut at the first space. Examples: "admin1", "bob". Source: kv step lines 24-27 (key 'user'); not listed in the quote trims at lines 351-436
• log.devname: FortiGate host name (default format only). Examples: FGT-HQ. Source: kv step lines 24-27; quote trims lines 355 and 398
• actionResult: Set only for CEF lines ('accept', 'denied', 'blocked'); never set in the default format the setup guide uses (see parser issues). Examples: denied. Source: add steps lines 308-325, which run before the quote trims at lines 341-436

Watch out for

• The setup guide sets 'set format default' (key=value pairs). The fields used here also exist for CEF lines (checked with FortiOS CEF samples), except log.devname (CEF puts the host name in the syslog header) and log.user (CEF user keys were not checked). Other FortiOS syslog formats (csv, rfc5424, json) were not checked.
• Every field depends on the line starting with '' (grok at lines 16-21). If a relay strips the priority, log.kvMessage is never written, the key=value step fails and the log keeps only raw; only W1, W5, W6 and W15 would show it.
• actionResult is not used. In the default format the parser adds it before it removes the quotes from action (adds at lines 308-325, trims at lines 341-436), so the check equals("action", "deny") sees '"deny"' and never matches. Running the parser's step code on action="deny" gives action 'deny' and no actionResult. W2, W11 and W12a filter on action instead.
• Traffic logs are usually most of the volume and nearly all share log ID 0000000013 (forward) or 0001000014 (local), so W7 is dominated by them. W9 to W12a break traffic down by policy, application, port and country.
• Values that contain spaces are cut at the first space, because the key=value step splits on spaces and only msg is repaired. This hits log.logdesc ('"Admin'), log.policyname, log.user and log.group ('"John' for "John Smith") and the FortiGate country fields ('United' for United States). That is why W12a uses origin.geolocation.country (UTMStack's own lookup) and W9 uses log.policyid rather than log.policyname.
• log.user keeps its quote marks (it is not in the trim lists), so W12b labels read "admin1" with the quotes. Clicking still works because the Log Explorer filter uses the stored value.
• log.logid is text with leading zeros ('0000000013'); keep it as text in the lookup and in filters. The 'in' filters used here are safe because the store turns JSON fields into text before comparing (go-sdk store/clickhouse filter.go).
• W12b counts three log IDs: 0100032002 admin login failed, 0101039426 SSL VPN login fail and 0102043009 user authentication failed (FortiOS 7.2.7 Log Reference). IPsec negotiation errors and 0102043010 (authentication lockout) are not included.
• origin.geolocation exists only for public source IPs (plugins/geolocation/geolocate.go IsLocal skips private ranges). Outbound traffic mostly has private sources, so W12a mainly shows inbound connections from the internet.
• If logs reach UTMStack through FortiAnalyzer or another relay, dataSource is the relay's IP for every log; the firewall's name is then only in log.devname.
• The alert widgets may stay nearly empty. Of the 7 FortiGate rules only the antivirus rule (type utm, subtype virus, action blocked) clearly matches what the parser produces. The others need log.logdesc to equal full text such as 'Admin login successful' (the parser keeps '"Admin'), log.action (renamed to action by the parser), a 'sandbox' subtype, or specific text such as 'type=utm' inside log.msg, which the parser also drops when msg is the last key.
• Field names: none of the fields used here depend on the underscore change. The raw keys logid, type, subtype, action, policyid, app and user contain no underscore, and the other fields are set by renames or the geolocation step (also checked by re-running the samples with underscores kept).
• Parser behavior was checked by running the EventProcessor step code (grok, kv, trim, rename, add, cast, delete) with go-sdk v1.1.26 regular expressions, CEL conditions and event conversion over FortiOS sample lines (log reference samples plus a few built from the FortiOS 7.2.7 field lists). The geolocation lookup was simulated. Not tested on a live UTMStack, and the engine version in the v12 image comes from a CI variable, so it could not be confirmed.

Parser problems found while designing this

These are not dashboard work, but they limit what the dashboard can show. They belong to #1975; raise them there rather than working around them in the dashboard.

• actionResult is never set for the default format: the add steps (lines 308-325) run before the quote trims (lines 341-436), so action is still '"deny"' when equals("action", "deny") is tested. Fix: move the three add steps after the trims. Also, client-rst, server-rst and ip-conn are ordinary session resets or connection errors, not firewall blocks, so mapping them to 'blocked' would overcount once the order is fixed.
• Values with spaces are cut at the first space and keep the opening quote: the key=value step (lines 24-27) splits on spaces and only msg is repaired. log.logdesc (the readable event name) becomes '"Admin', log.policyname '"Implicit', log.user '"John', srccountry and dstcountry 'United'. Fix: a quote-aware key=value split, or rescue groks like the one for msg for logdesc, policyname, user, group and the country fields.
• Quotes stay on keys outside the trim lists (lines 351-436): log.user, log.group, log.status, log.reason, log.attack, log.virus, log.severity, log.crlevel, log.policyname, log.logdesc, log.msg. Fix: trim quotes on all values.
• log.msg is lost when msg is the last key: 'delete log.msg' (lines 30-32) runs first and the rescue grok (lines 207-215) needs another key=value after msg. Event logs often end with msg (the admin-login-failed and SSL-VPN-login-fail samples), so log.msg is missing there, which also breaks the VPN brute-force rule.
• The CEF header grok (lines 36-74) never matches: a pattern that is only '{{.data}}' matches an empty string, so the grok stops at the vendor field and log.cefName, log.cefSeverity and the other CEF header fields never exist. For CEF lines whose event name contains a space, the first extension key is glued to the header ('close|3|deviceExternalId'); only the server-rst and client-rst variants are recovered (lines 77-79), so log.devid is lost on the rest.
• Does not affect this dashboard: the renames of log.dest_ip, log.dest_port, log.src_ip and log.src_port (lines 272-287) never match with the current engine, which strips underscores from key names.

How to build it

Before you start: parser field names are changing while the parsers are updated for the engine's new underscore handling (see "Field names are about to move" in #2696). Check every field in this issue against the v12 parser at that moment and against real logs, and build against what you find.

1. Add definitions/dashboards/integration-fortigate.yaml. Keep it in the top folder: the test that checks shipped dashboards (TestEveryShippedDashboardDefinitionIsValid) only reads the top folder.
2. Start from the file below; it follows the table above and passes the same checks as the backend (domain.Spec.Validate).
3. W7 is a value/count table: a category query shown with chart type table. It already renders; v12 dashboards: groundwork for integration dashboards (click-through to the Log Explorer, value/count table, Agents dashboard fix) #2697 makes its rows clickable and its headers readable.
4. Load real logs from this technology on a v12 test server (or replay samples) and check every widget before opening the pull request.

Starting dashboard file

# Dashboard version v1.0.0
#
# System-owned default dashboard for the FortiGate integration, seeded by
# backend/modules/dashboards/repository/dashboard_bootstrap.go.
# Field names come from definitions/filters/fortinet/fortinet.yaml.
# Verify every widget against real logs before shipping.
name: "FortiGate"
description: "What FortiGate is sending: volume, event types, and the activity worth a look."
widgets:
  - layout: { x: 0, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Total logs"

  - layout: { x: 3, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "log.type"
          op: in
          value: ["traffic"]
        - field: "action"
          op: in
          value: ["deny"]
    config:
      __builder:
        chartType: metric
        title: "Denied traffic"

  - layout: { x: 6, y: 0, w: 3, h: 2 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: metric
      metric:
        agg: count
      filters:
        - field: "log.subtype"
          op: in
          value: ["ips", "virus", "anomaly"]
    config:
      __builder:
        chartType: metric
        title: "Intrusion, virus and anomaly detections"

  - layout: { x: 9, y: 0, w: 3, h: 2 }
    spec:
      dataset: alerts
      dataType: "firewall-fortigate-traffic"
      chart: metric
      metric:
        agg: count
    config:
      __builder:
        chartType: metric
        title: "Alerts"

  - layout: { x: 0, y: 2, w: 8, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: time
      metric:
        agg: count
    config:
      __builder:
        chartType: area
        title: "Log volume over time"

  - layout: { x: 8, y: 2, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "dataSource"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Logs by firewall IP"

  - layout: { x: 0, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "log.logid"
      filters:
        - field: "log.logid"
          op: exists
      limit: 25
    config:
      __builder:
        chartType: table
        title: "Top event types"

  - layout: { x: 6, y: 6, w: 6, h: 6 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: time
      metric:
        agg: count
      dimension: "log.logid"
      filters:
        - field: "log.logid"
          op: exists
      limit: 5
    config:
      __builder:
        chartType: line
        title: "Event types over time (top 5)"

  - layout: { x: 0, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "log.policyid"
      filters:
        - field: "log.type"
          op: in
          value: ["traffic"]
        - field: "log.policyid"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top firewall policies (ID)"

  - layout: { x: 4, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "log.app"
      filters:
        - field: "log.type"
          op: in
          value: ["traffic"]
        - field: "log.app"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Top applications"

  - layout: { x: 8, y: 12, w: 4, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "target.port"
      filters:
        - field: "log.type"
          op: in
          value: ["traffic"]
        - field: "action"
          op: in
          value: ["deny"]
        - field: "target.port"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Denied traffic by destination port"

  - layout: { x: 0, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "origin.geolocation.country"
      filters:
        - field: "log.type"
          op: in
          value: ["traffic"]
        - field: "action"
          op: in
          value: ["deny"]
        - field: "origin.geolocation.country"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Denied traffic by source country"

  - layout: { x: 6, y: 16, w: 6, h: 4 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "log.user"
      filters:
        - field: "log.logid"
          op: in
          value: ["0100032002", "0101039426", "0102043009"]
        - field: "log.user"
          op: exists
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Failed logins by user"

  - layout: { x: 0, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "name"
      limit: 10
    config:
      __builder:
        chartType: bar
        title: "Alerts by rule"

  - layout: { x: 6, y: 20, w: 6, h: 4 }
    spec:
      dataset: alerts
      dataType: "firewall-fortigate-traffic"
      chart: category
      metric:
        agg: count
      dimension: "severity"
    config:
      __builder:
        chartType: bar
        title: "Alerts by severity"

  - layout: { x: 0, y: 24, w: 12, h: 6 }
    spec:
      dataset: logs
      dataType: "firewall-fortigate-traffic"
      chart: table
      metric:
        agg: count
      limit: 20
      columns: ["@timestamp", "dataSource", "log.logid", "log.subtype", "action", "origin.ip", "target.ip", "target.port"]
    config:
      __builder:
        chartType: table
        title: "Latest logs"

Done when

• definitions/dashboards/integration-fortigate.yaml is merged to release/v12.0.0 and go test ./modules/dashboards/... passes in backend/.
• With FortiGate logs flowing on a v12 test server, every widget shows data. A widget that stays empty while logs arrive means a wrong field or value: fix it, don't ship it.
• Clicking a row, bar or line point opens the Log Explorer on the same logs, and the Log Explorer count matches the widget.
• A time range with no logs shows empty states, not errors.
• A screenshot of the finished dashboard is attached to this issue.