KNS | ˈkɛvɪn netsec

Musings about networks and security

Tag: ha

  • A real look at FortiGate FGCP HA BGP failover behaviour

    A FortiGate Cluster Protocol (FGCP) HA deployment is nothing new in today’s world, and with more and more of these clusters functioning as BGP routers, especially with the proliferation of SD-WAN and ADVPN, having a high BGP service uptime is becoming critical. In order to achieve this, there are a few things to keep in mind. These things, dear reader, are what this post is about.

    The setup

    • 2x FortiGate 70G on 7.6.7
    • FortiGate 60F on 7.6.5
    • 2x Windows 11 clients on 25H2
    BGP HA failover topology

    A BGP peering in the AS 65001 over the 192.0.2.0/24 subnet has been established between the 70G cluster and the standalone 60F.

    Both FortiGate deployments have a client behind them; they are announcing the client subnets, and the 60F is also announcing subnets ranging from 169.254.1.0/24 to 169.254.250.0/24 as a very small stress test.

    Note: When I use the word “downstream”, I mean any BGP peer of the HA cluster. This doesn’t mean the peer is actually downstream of the traffic flow. Writing “BGP peer of the HA cluster” every time would be bad writing.

    The most important information

    You should get the reason for reading this post as quickly as possible, so here it is, if you start with default values:

    • Enable Graceful Restart either at the BGP global level or per neighbor on both systems
    • The route-ttl setting of the HA cluster needs to be high enough to allow for full route convergence to happen
    • Keeping the preceding point in mind, the downstream device needs to have a BGP advertisement-interval at least lower than the route-ttl value of the HA cluster
    • The downstream device must not tear down the existing BGP session before the new BGP session has been established on the new primary FortiGate, meaning keep keepalive and holdtime in mind

    Basics about FGCP BGP behaviour

    Before I get into the meat and potatoes, I want to put out some information on how BGP behaves in an FGCP cluster.

    • The BGP routing process and thus the BGP neighborships only exist on the primary unit.
    • Once a failover happens, the new primary has to establish the BGP neighborship again.
    • Routes are copied from the primary to the secondary at the route-hold interval (default 10 seconds).
    • route-wait determines how long to wait after the primary recognizes a routing table update until it copies it to the secondary (default 0 seconds, meaning on every routing table update).
    • Routes are kept in a cluster for the route-ttl duration (default 10 seconds).
    • The copied routes are visible in the kernel table on the secondary using get router info kernel command. You do not see them with the get router info routing-table database command.
    • Importantly, once the route-ttl timer has run out and the new primary has not received the kernel routes previously acquired from the BGP peer, again, these kernel routes will be discarded. This behaviour is most likely consistent across all routing protocols, not just BGP.

    Here is the regular routing table and the kernel routing table output of the secondary:

    70G-BGP1(Secondary) # get router info routing-table database

70G-BGP1(Secondary) #

70G-BGP1(Secondary) # get router info kernel | grep port2
tab=65535 vf=0 vrf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.0/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65535 vf=0 vrf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.1/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65535 vf=0 vrf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.255/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->10.10.10.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.1.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.2.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.3.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.4.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.5.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.6.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
...

    With this information in mind, the failover process for BGP looks as follows, with default values:

    1. The cluster performs a failover
    2. The new primary forms a new BGP neighborship
    3. The downstream peer announces its routes
      • If this happens after the route-ttl timer, connectivity is impacted.
    4. After all routes have been received and installed, normal operations continue

    With default values, you will lose connectivity if traffic relies on BGP routing despite the copied routes because of the route-ttl timer, but this will be solved.

    More information on route-ttl, route-hold, and route-wait is in the official documentation.

    With default values

    Assuming we did nothing to our BGP configuration, and we only did the minimum to establish a session, we will observe that upon a failover, the new primary will lose its BGP kernel routes after 10 seconds, the route-ttl timer.

    Here is the BGP configuration of the HA cluster and the neighbor output of the downstream peer:

    HA cluster BGP configuration and neighbor output of downstream

    70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
    config redistribute "connected"
    end
    config redistribute "rip"
    end
    config redistribute "ospf"
    end
    config redistribute "static"
    end
    config redistribute "isis"
    end
    config redistribute6 "connected"
    end
    config redistribute6 "rip"
    end
    config redistribute6 "ospf"
    end
    config redistribute6 "static"
    end
    config redistribute6 "isis"
    end
end

60F-DOWNSTREAM # get router info bgp neighbors 192.0.2.1
VRF 0 neighbor table:
BGP neighbor is 192.0.2.1, remote AS 65001, local AS 65001, internal link
  BGP version 4, remote router ID 192.0.2.1
  BGP state = Established, up for 00:04:31
  Last read 00:00:54, hold time is 180, keepalive interval is 60 seconds
  Configured hold time is 180, keepalive interval is 60 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (old and new)
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised and received
    Address family VPNv6 Unicast: advertised and received
    Address family L2VPN EVPN: advertised and received
  Received 32 messages, 1 notifications, 0 in queue
  Sent 35 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  NLRI treated as withdraw: 0
  Minimum time between advertisement runs is 30 seconds
  Update source is SW-BGP

 For address family: IPv4 Unicast
  BGP table version 13, neighbor version 13
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  1 accepted prefixes, 1 prefixes in rib
  1 announced prefixes

    It takes about 3 minutes and 30 seconds for the new primary to get all routes. This duration comes from the keepalive timer of the downstream peer, with a default of 180 seconds and an advertisement interval of 30 seconds.

    Looking at the debugs from the downstream peer, we see this behaviour (I have tried to align the failover with the last keepalive being received):

    BGP debug with default values

    2026-07-19 14:30:29 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:30:29 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:30:34 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 14
2026-07-19 14:30:35 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 58 KAlive msg(s) sent
2026-07-19 14:30:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:30:43 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:30:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:10 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:31:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:31:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:31:28 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 59 KAlive msg(s) sent
2026-07-19 14:31:36 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:31:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:59 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:12 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 14
2026-07-19 14:32:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:23 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 60 KAlive msg(s) sent
2026-07-19 14:32:25 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:32:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:32:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:50 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:19 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 61 KAlive msg(s) sent
2026-07-19 14:33:20 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:33:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:33:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:33:29 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:33:29 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:33:29 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:33:29 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:33:29 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:33:29 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Connect Event: 17
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 62 KAlive msg(s) sent
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:33:35 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:33:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:34:01 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:34:03 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
    • At 14:30:29, the last keepalive was received
    • At 14:33:29, exactly 3 minutes/180 seconds later, the hold timer expired
    • At 14:33:35, the HA peer comes up
    • At 14:34:01, the prefix is received

    This is a worst-case scenario. The convergence time can be lower because the hold timer expiration depends on when the last keepalive was successfully received, so you can subtract up to 59 seconds here.

    Tweaking timers

    Keepalive and hold timer values can be easily changed, and we can do this at both the global and neighbor level on a FortiGate. I have set both timers to their minimum at the global level on both devices. Technically, it’s enough to only do it on the downstream peer. Note that changing these values requires a new BGP session.

    config router bgp
    set as 65001
    set router-id 192.0.2.2
    set keepalive-timer 1
    set holdtime-timer 3
end

    Looking at the debugs on the downstream again, we see that we get our prefix much quicker. It takes around 33 seconds now.

    BGP debug with minimum keepalive and holdtimer values

    2026-07-19 14:41:12 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 260 KAlive msg(s) sent
2026-07-19 14:41:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:41:13 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:41:13 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 3
2026-07-19 14:41:13 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:41:13 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:41:13 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:41:13 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:41:13 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
...
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Cap: RR Cap(new) for all address-families
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 6
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Cap: Cap Code 65, Cap Len 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 261 KAlive msg(s) sent
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:16 [root] BGP: bgp_keepalive_proc: notif_rcv 4-4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:41:16 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 262 KAlive msg(s) sent
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
...
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 290 KAlive msg(s) sent
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 2, length 48
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: Starting UPDATE decoding... Bytes To Read (29), msg_size (29)
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:41:45 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:41:45 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 291 KAlive msg(s) sent
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
    • At 14:41:12, the last keepalive is being sent
    • At 14:41:13, the hold timer expires
    • At 14:41:16, the HA peer comes up
    • At 14:41:45, the prefix is received

    These numbers make sense because we have a hold timer of 3 seconds and an advertisement interval of 30 seconds.

    Faster advertisements

    It’s obvious that if we advertise routes more often, we get our routes faster.

    Changing the advertisement interval on the downstream peer to 1 means we get to route convergence in about 7 seconds.

    config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 1
        next
    end
end

    BGP debugs with changed advertisement interval

    2026-07-19 14:45:00 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 484 KAlive msg(s) sent
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 3
2026-07-19 14:45:01 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:45:01 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:45:01 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:45:01 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:45:01 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:45:05 [root] BGP: [NETWORK] Accept Thread: Incoming conn from host 192.0.2.1 (FD=28 VRF=0)
2026-07-19 14:45:05 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 14
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Connect Event: 17
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 1
...
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 485 KAlive msg(s) sent
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:45:07 [root] BGP: bgp_keepalive_proc: notif_rcv 4-4
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:45:07 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 2, length 48
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: Starting UPDATE decoding... Bytes To Read (29), msg_size (29)
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:45:08 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 2
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Attr IP-Unicast: Tot-attr-len 21
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Update: Msg #15 Size 48
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 486 KAlive msg(s)
    • At 14:45:00, the last keepalive is being sent
    • At 14:45:01, the hold timer expires
    • At 14:45:07, the HA peer comes up
    • At 14:45:08, the prefix is received

    That’s okay, but we still lose traffic.

    Some more timers

    The following timers don’t actually change anything for this scenario, but they are still helpful in regular operations, and I want to highlight them. These are the scan and connect timers.

    The connect-timer value determines how long the FortiGate waits before attempting a new BGP connection attempt after the previous one has failed.

    The scan-time determines the interval at which the FortiGate scans for next-hop reachability, and if the next hop isn’t reachable, the route gets dropped.

    config router bgp
    set as 65001
    set router-id 192.0.2.2
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set connect-timer 1
        next
    end
end

    You can read more about these timers and lots of other ones in the official documentation.

    Not tearing down the session with Graceful Restart

    So we are down to around 8 seconds, and this is with a torn-down session on the downstream side. At this point it’s pointless to try and optimize this approach. We need the session to not get torn down on the downstream peer and keep the routes available on both sides.

    If you know BGP, you already know the answer to this: Graceful Restart (GR)

    Graceful Restart is designed to keep routes in the Routing Information Base (RIB) even if the peer is down. There are timers around this, but the defaults will be enough for us. Keep in mind that both sides need to support GR for this to work. Every half-decent router should be able to, but I just want to mention it.

    GR can be enabled at the global and neighbor level on a FortiGate, and doing so clears your BGP sessions, either globally or only for a neighbor. I enable it on the neighbor.

    70G-BGP1(Primary) # show router bgp
onfig router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set capability-graceful-restart enable
        next
    end
end

    I have also unset all other previous settings, so here is the full BGP config (excluding network statements and redistribute sections) of both devices at this point:

    60F-DOWNSTREAM # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "SW-BGP"
            set remote-as 65001
            set update-source "SW-BGP"
        next
    end
end

70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
end

    For this exercise, it’s technically enough to enable GR only for the HA peer, but you might as well do it for both peers.

    You can verify the GR capability using the get router info bgp neighbors <NEIGHBOR_IP> command.

    BGP neighbor output after graceful restart

    60F-DOWNSTREAM # get router info bgp neighbors 192.0.2.1
VRF 0 neighbor table:
BGP neighbor is 192.0.2.1, remote AS 65001, local AS 65001, internal link
  BGP version 4, remote router ID 192.0.2.1
  BGP state = Established, up for 00:00:23
  Last read 00:00:01, hold time is 3, keepalive interval is 1 seconds
  Configured hold time is 3, keepalive interval is 1 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (old and new)
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised and received
    Address family VPNv6 Unicast: advertised and received
    Address family L2VPN EVPN: advertised and received
  Received 1509 messages, 7 notifications, 0 in queue
  Sent 1550 messages, 16 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  NLRI treated as withdraw: 0
  Minimum time between advertisement runs is 1 seconds
  Update source is SW-BGP

 For address family: IPv4 Unicast
  BGP table version 19, neighbor version 18
  Index 1, Offset 0, Mask 0x2
  AF-dependant capabilities:
    Graceful restart: advertised, received, negotiated
      Forwarding states are being preserved
  Community attribute sent to this neighbor (both)
  1 accepted prefixes, 1 prefixes in rib
  1 announced prefixes

 For address family: VPNv4 Unicast
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 For address family: VPNv6 Unicast
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 For address family: L2VPN EVPN
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 Connections established 24; dropped 23
 Graceful-restart Status:
  Remote restart-time is 120 sec

    We see Graceful restart: advertised, received, negotiated for the IPv4 Unicast address family, and we also see some additional information at the bottom regarding the remote restart time.

    With GR enabled, our BGP routing behaviour is much better. It’s not interesting to show any debugs, but on a failover, traffic forwarding works fully up until the route-ttl timer, default of 10 seconds, is over (this behaviour was explained further above). At that point, connectivity is lost until the routes get announced again.

    We can work and fix this with our acquired knowledge.

    The last tweaks

    Now we need to make sure two things happen:

    1. Routes are announced more often
    2. Routes are kept in the cluster’s kernel routing table for longer, or at least until we get the new routes

    We know the answer to both, and the answers are advertisement-interval and route-ttl.

    If we set the advertisement interval on the downstream at least lower than the route TTL of the cluster, everything should be fine. So an interval of 3 seconds with a TTL of 10 should be enough. This is theoretically correct, but it also depends on your environment. If you expect to receive a lot of routes, it might take longer to ingest them all, so consider upping the TTL past the default, or reducing the advertisement interval, or doing both.

    My downstream peer is only announcing 251 routes, and with an interval of 3 and a TTL of 10, I see zero traffic loss.

    The final configuration

    This doesn’t differ much from the configuration posted in the GR section, except that the advertisement interval is now set to something other than the default. The route TTL is at its default of 10, but it’s still shown.

    60F-DOWNSTREAM # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set activate6 disable
            set capability-graceful-restart enable
            set interface "SW-BGP"
            set remote-as 65001
            set update-source "SW-BGP"
        next
    end
end

70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
end
config system ha
    set route-ttl 10
end

    With this configuration, I was able to perform an HA failover without losing any pings or interruption of a file transfer between the downstream client and the HA client.

    You can adapt this configuration with the scan and connect timers, and of course whatever else you need, but be careful about the hold and keepalive timers, lest you tear down the BGP session.

    I heard about BFD

    No. Bidirectional Forwarding Detection (BFD) is a great feature that helps with detecting failures in routing sessions, but in this case, it does the opposite of what we want. If BFD detects that its session is down, it will also tear down the BGP session, which would lead to traffic loss.

    When it comes to combining BFD with GR, you have to consult the vendor documentation because some vendors might support this combination. Start with the assumption that this is not supported; however, Fortinet explicitly does not recommend it, as can be read here and here.

    Use BFD with caution. Combining BFD with graceful restart is not recommended by Fortinet. Other vendors also explicitly mention not to combine them in configuration, as it might cause suboptimal routing performances when graceful restart and BFD are both configured.

    BGP graceful restart or OSPF graceful restart doesn’t work with BFD, and it is not recommended.

    Wrapping up

    The fact that graceful restart was the solution to this problem wasn’t a surprise, because that’s what it is designed for, but the behaviour with route-ttl was new to me, and I am glad that I spent the time creating this post, because I have definitely learned something new. I hope that you, dear reader, have also learned something.

  • Connecting to an HA FortiClient EMS cluster without an external load balancer

    Do you want redundancy? Don’t answer that question; the answer is “Yes!” This means you want your FortiClient EMS deployment to be redundant, and this gives you the problem of how to handle the FortiClients and FortiGate connection to your HA EMS nodes if you don’t have an external load balancer.

    Well, dear reader, I’ve got just the solution for you, so let’s see how it’s done.

    The setup

    • FortiGate 70G on 7.6.7
    • 2x FortiClient EMS on 7.4.7
    • Windows 11 client on 25H2 with FortiClient 7.4.7

    As far as layer 3 is concerned, I have a CLIENT and DMZ VLAN, where, respectively, the client and the EMS instances are located, as well as the UNUSED VLAN, which gets used later.

    Topology

    The configuration is very light this time, but it is still on the Fortinet resources GitHub repository.

    FortiClient EMS HA basics

    First things first. There are several HA deployment options, and I specifically cover the one using only VM appliances. As the documentation states, this requires FortiClient EMS 7.4.5, so keep this in mind.

    Setting the cluster up is not hard; it took me about 5 minutes, and there are no additional licenses necessary; you just need the resources for the VM.

    A few points here:

    • In order to change the priority of the nodes, you use the ha standby command on the current primary (use ha get nodes to see the cluster state). There are two roles in the cluster: the database and EMS
    • If you want to demote the current primary for both roles, you use ha standby --type=”db” and ha standby --type=”ems”. The documentation linked above only mentions it but doesn’t give an example of how to do it in the document. You need the CLI reference for this.
      • In the following example, EMS2 is the primary for both roles at first and gets demoted
      • I have removed the “Preferred DCs” column so it fits nicely in here, and because it’s empty anyway
    ems@EMS2 $> ha get nodes
EMS Node(s):
 Name                                             | Role    | Status  | Last Seen
--------------------------------------------------+---------+---------+----------------------------
 EMS2 (*)                                         | primary | online  | 2026-06-18 18:32:57.841091
 EMS1                                             | standby | online  | 2026-06-18 18:33:00.042757

DB Node(s):
 Host                            | Port    | Role    | Status   | Latency (ms)
--------------------------------------------------+---------+---------+------------
 192.168.1.177                   | 5432    | standby | online   | 28
 127.0.0.1                       | 5432    | primary | online   | 27
ems@EMS2 $> ha standby --type="db"
Node demoted successfully!
ems@EMS2 $> ha standby --type="ems"
Node demoted successfully!
ems@EMS2 $> ha get nodes
EMS Node(s):
 Name                                             | Role    | Status  | Last Seen
--------------------------------------------------+---------+---------+----------------------------
 EMS1                                             | primary | online  | 2026-06-18 18:35:10.04539
 EMS2 (*)                                         | standby | online  | 2026-06-18 18:35:01.322546

DB Node(s):
 Host                            | Port    | Role    | Status   | Latency (ms)
--------------------------------------------------+---------+---------+------------
 192.168.1.177                   | 5432    | primary | online   | 27
 127.0.0.1                       | 5432    | standby | online   | 26
    • The telemetry service, TCP/8013, is only active on the current primary, and the HTTPS service is active on both nodes
      • This is an important point for later
    • EMS does not have its own virtual IP or load balancer service, like haproxy, so any outside connections to it have to get load balanced via something. If you have an external load balancer, this is simple, but that’s a luxury not everyone has
    • The failover time is calculated using the following formula: High Availability Keep Alive Interval * 2 + 60
      • The High Availability Keep Alive Interval is configured in the EMS Settings menu
      • With default settings, this means it takes 80 seconds for a failover to occur

      With the basics covered, let’s connect things to EMS.

      The FortiClient to EMS connection

      This one is easy. You can use the Load Balance feature on the FortiGate to accomplish this (it has to be enabled in Feature Visibility first), and then you can configure everything in the Virtual Servers menu under Policy & Objects.

      For FortiClients, we need, at the minimum, TCP/8013 for the telemetry, so create your virtual server using that as the port and also the health check (either in-line or using the Health Check menu). If you need the installer port, default TCP/11443, you can do that too.

      FortiClient EMS HA Virtual Server
      config firewall vip
    edit "EMS-HA-FORTICLIENT"
        set type server-load-balance
        set server-type tcp
        set extip 192.168.1.190
        set extintf "CLIENTS"
        set monitor "tcp8013"
        set extport 8013
        config realservers
            edit 1
                set ip 192.168.1.177
                set port 8013
            next
            edit 2
                set ip 192.168.1.178
                set port 8013
            next
        end
    next
end

      With the virtual server created, you can create your policy accordingly. Note that if you use the type TCP in your virtual server, the policy can be created using the flow-based inspection mode, which means hardware offloading. You only need proxy-based for non-TCP/UDP/IP types. Attach security profiles to the policy as necessary.

      FortiClient EMS HA Policy

      That’s everything you need for FortiClient to connect to EMS. Use the FQDN or an invitation code in the FortiClient GUI and hit Connect. Simple, right?

      The FortiGate to EMS connection

      This is the one that required some time to get working because the FortiGate needs to load-balance its own traffic, which I didn’t think was actually possible, but apparently it is.

      We, again, start with the virtual server, and we need TCP/443 here, but crucially, the health check must not be TCP/443. As mentioned above, the HTTPS service is active on both nodes, so this would create immediate issues depending on your load-balancing method and, in general, create issues.

      What service is only active on the primary node? The telemetry service, so we reuse the health check for TCP/8013, we already used for the FortiClient connection. The real servers still use TCP/443, however.

      I am binding the virtual server to the UNUSED VLAN because I only need to put the virtual server in a policy to activate it. This is done more so to show that this is possible. In reality, you would probably bind it to a management interface, so if you go to the virtual server IP, you always land on the active EMS node’s GUI.

      FortiGate EMS HA Virtual Server
      config firewall vip
    edit "EMS-HA-FORTIGATE"
        set type server-load-balance
        set server-type tcp
        set extip 192.168.1.190
        set extintf "LOOPBACK"
        set monitor "tcp8013"
        set extport 443
        config realservers
            edit 1
                set ip 192.168.1.177
                set port 443
            next
            edit 2
                set ip 192.168.1.178
                set port 443
            next
        end
    next
end

      With this in mind, the policy we need is nothing special. The virtual server just needs to be active, and again, we can use a flow-based policy. Attach security profiles to the policy as necessary.

      FortiGate EMS HA Policy

      In the Fabric Connector for EMS, we use the FQDN, which resolves to the virtual server IP, and at least in my configuration, nothing happens because the traffic gets sourced incorrectly.

      FortiGate EMS connector

      To solve this, you go to the CLI and set a source IP for this connection. I am using the IP that the DMZ interface has, so that will be the one I see on EMS.

      Since we’re already in the CLI, we can use the command execute fctems verify <ID> (insert your ID) to start the verification, which should give us a certificate to accept.

      FortiGate EMS fabric CLI configuration

      70G-EMSHA # config endpoint-control fctems
70G-EMSHA (fctems) # edit 1
70G-EMSHA (1) # set source-ip 192.168.1.202
70G-EMSHA (1) # end

The configuration will not be effective unless server certificate is verified.
You can get and verify server certificate by the following command:
"execute fctems verify 1" (ems table id)

70G-EMSHA # execute fctems verify 1

        Subject:     CN = ems-ha.ad.labdomain.com
        Issuer:      DC = com, DC = labdomain, DC = ad, CN = WIN-CA
        Valid from:  2026-06-18 08:10:35  GMT
        Valid to:    2028-06-17 08:10:35  GMT
        Fingerprint: 6D:6B:80:BC:10:9B:31:F7:15:7A:CC:03:71:01:98:9C:97:A0:4D:C5:21:12:C8:10:15:31:F3:43:E8:9B:2E:A7
        Root CA:     No
        Version:     3
        Serial Num:
                32:00:00:00:44:88:6a:ed:ca:e6:cb:5b:39:00:02:00:00:00:44
        Extensions:
                Name:     X509v3 Subject Key Identifier
                Critical: no
                Content:
                BC:33:E8:D9:39:2F:29:36:21:E2:B7:88:9A:70:8D:79:63:F7:C6:55

                Name:     X509v3 Key Usage
                Critical: yes
                Content:
                Digital Signature, Key Encipherment

                Name:     X509v3 Subject Alternative Name
                Critical: no
                Content:
                DNS:ems-ha.ad.labdomain.com, DNS:ems1.ad.labdomain.com, DNS:ems2.ad.labdomain.com

                Name:     X509v3 Authority Key Identifier
                Critical: no
                Content:
                F3:CD:D6:6C:D4:C1:35:68:D7:EE:AA:07:7A:A8:5A:73:B8:48:6C:D5

                Name:     X509v3 CRL Distribution Points
                Critical: no
                Content:
                Full Name:
  URI:ldap:///CN=WIN-CA(2),CN=WIN-AD,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=ad,DC=labdomain,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint
  URI:http://WIN-AD.ad.labdomain.com/CertEnroll/WIN-CA(2).crl


                Name:     Authority Information Access
                Critical: no
                Content:
                CA Issuers - URI:ldap:///CN=WIN-CA,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=ad,DC=labdomain,DC=com?cACertificate?base?objectClass=certificationAuthority
CA Issuers - URI:http://WIN-AD.ad.labdomain.com/CertEnroll/WIN-AD.ad.labdomain.com_WIN-CA(2).crt

                Name:     Microsoft certificate template
                Critical: no
                Content:
                0-.%+.....7.....n...z...%...w........G..f..d...

                Name:     X509v3 Extended Key Usage
                Critical: no
                Content:
                TLS Web Server Authentication

                Name:     Microsoft Application Policies Extension
                Critical: no
                Content:
                0.0
..+.......

EMS configuration needs user to confirm server certificate.
Do you wish to add the above certificate to trusted remote certificates? (y/n)y

Certificate successfully configured and verified.

      Once we have entered the good old y, we can go to EMS and authorize the FortiGate that appeared.

      EMS fabric device

      Wrapping up

      And that’s that. You can do your failover tests (mine worked), and hopefully, dear reader, you can now connect your FortiClients and your FortiGate to your EMS HA deployment. Additional FortiGates should be much easier to handle; it’s just the first one that requires a bit of special attention.

      And if you don’t have enough of FortiClient EMS yet, you can read about how you can use its certificate management capabilities to help with VPN authentication, full SSL/TLS inspection and 802.1X.