bgp – KNS | ˈkɛvɪn netsec

A FortiGate Cluster Protocol (FGCP) HA deployment is nothing new in today’s world, and with more and more of these clusters functioning as BGP routers, especially with the proliferation of SD-WAN and ADVPN, having a high BGP service uptime is becoming critical. In order to achieve this, there are a few things to keep in mind. These things, dear reader, are what this post is about.

The setup

2x FortiGate 70G on 7.6.7
FortiGate 60F on 7.6.5
2x Windows 11 clients on 25H2

A BGP peering in the AS 65001 over the 192.0.2.0/24 subnet has been established between the 70G cluster and the standalone 60F.

Both FortiGate deployments have a client behind them; they are announcing the client subnets, and the 60F is also announcing subnets ranging from 169.254.1.0/24 to 169.254.250.0/24 as a very small stress test.

Note: When I use the word “downstream”, I mean any BGP peer of the HA cluster. This doesn’t mean the peer is actually downstream of the traffic flow. Writing “BGP peer of the HA cluster” every time would be bad writing.

The most important information

You should get the reason for reading this post as quickly as possible, so here it is, if you start with default values:

Enable Graceful Restart either at the BGP global level or per neighbor on both systems
The route-ttl setting of the HA cluster needs to be high enough to allow for full route convergence to happen
Keeping the preceding point in mind, the downstream device needs to have a BGP advertisement-interval at least lower than the route-ttl value of the HA cluster
The downstream device must not tear down the existing BGP session before the new BGP session has been established on the new primary FortiGate, meaning keep keepalive and holdtime in mind

Basics about FGCP BGP behaviour

Before I get into the meat and potatoes, I want to put out some information on how BGP behaves in an FGCP cluster.

The BGP routing process and thus the BGP neighborships only exist on the primary unit.
Once a failover happens, the new primary has to establish the BGP neighborship again.
Routes are copied from the primary to the secondary at the route-hold interval (default 10 seconds).
route-wait determines how long to wait after the primary recognizes a routing table update until it copies it to the secondary (default 0 seconds, meaning on every routing table update).
Routes are kept in a cluster for the route-ttl duration (default 10 seconds).
The copied routes are visible in the kernel table on the secondary using get router info kernel command. You do not see them with the get router info routing-table database command.
Importantly, once the route-ttl timer has run out and the new primary has not received the kernel routes previously acquired from the BGP peer, again, these kernel routes will be discarded. This behaviour is most likely consistent across all routing protocols, not just BGP.

Here is the regular routing table and the kernel routing table output of the secondary:

70G-BGP1(Secondary) # get router info routing-table database

70G-BGP1(Secondary) #

70G-BGP1(Secondary) # get router info kernel | grep port2
tab=65535 vf=0 vrf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.0/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65535 vf=0 vrf=0 scope=254 type=2 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.1/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65535 vf=0 vrf=0 scope=253 type=3 proto=2 prio=0 0.0.0.0/0.0.0.0/0->192.0.2.255/32 pref=192.0.2.1 gwy=0.0.0.0 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->10.10.10.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.1.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.2.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.3.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.4.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.5.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
tab=65534 vf=0 vrf=0 scope=0 type=1 proto=19 prio=2147483649 0.0.0.0/0.0.0.0/0->169.254.6.0/24 pref=0.0.0.0 gwy=192.0.2.2 dev=6(port2)
...

With this information in mind, the failover process for BGP looks as follows, with default values:

The cluster performs a failover
The new primary forms a new BGP neighborship
The downstream peer announces its routes
- If this happens after the route-ttl timer, connectivity is impacted.
After all routes have been received and installed, normal operations continue

With default values, you will lose connectivity if traffic relies on BGP routing despite the copied routes because of the route-ttl timer, but this will be solved.

More information on route-ttl, route-hold, and route-wait is in the official documentation.

With default values

Assuming we did nothing to our BGP configuration, and we only did the minimum to establish a session, we will observe that upon a failover, the new primary will lose its BGP kernel routes after 10 seconds, the route-ttl timer.

Here is the BGP configuration of the HA cluster and the neighbor output of the downstream peer:

HA cluster BGP configuration and neighbor output of downstream

70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
    config redistribute "connected"
    end
    config redistribute "rip"
    end
    config redistribute "ospf"
    end
    config redistribute "static"
    end
    config redistribute "isis"
    end
    config redistribute6 "connected"
    end
    config redistribute6 "rip"
    end
    config redistribute6 "ospf"
    end
    config redistribute6 "static"
    end
    config redistribute6 "isis"
    end
end

60F-DOWNSTREAM # get router info bgp neighbors 192.0.2.1
VRF 0 neighbor table:
BGP neighbor is 192.0.2.1, remote AS 65001, local AS 65001, internal link
  BGP version 4, remote router ID 192.0.2.1
  BGP state = Established, up for 00:04:31
  Last read 00:00:54, hold time is 180, keepalive interval is 60 seconds
  Configured hold time is 180, keepalive interval is 60 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (old and new)
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised and received
    Address family VPNv6 Unicast: advertised and received
    Address family L2VPN EVPN: advertised and received
  Received 32 messages, 1 notifications, 0 in queue
  Sent 35 messages, 0 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  NLRI treated as withdraw: 0
  Minimum time between advertisement runs is 30 seconds
  Update source is SW-BGP

 For address family: IPv4 Unicast
  BGP table version 13, neighbor version 13
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  1 accepted prefixes, 1 prefixes in rib
  1 announced prefixes

It takes about 3 minutes and 30 seconds for the new primary to get all routes. This duration comes from the keepalive timer of the downstream peer, with a default of 180 seconds and an advertisement interval of 30 seconds.

Looking at the debugs from the downstream peer, we see this behaviour (I have tried to align the failover with the last keepalive being received):

BGP debug with default values

2026-07-19 14:30:29 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:30:29 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:30:34 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 14
2026-07-19 14:30:35 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 58 KAlive msg(s) sent
2026-07-19 14:30:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:30:43 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:30:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:10 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:31:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:31:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:31:28 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 59 KAlive msg(s) sent
2026-07-19 14:31:36 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:31:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:31:59 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:12 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 14
2026-07-19 14:32:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:23 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 60 KAlive msg(s) sent
2026-07-19 14:32:25 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:32:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:32:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:32:50 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:32:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:19 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 61 KAlive msg(s) sent
2026-07-19 14:33:20 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:33:28 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:28 [root] BGP: [RIB] Scanning BGP RIB for VRF 0...
2026-07-19 14:33:28 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:33:29 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:33:29 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:33:29 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:33:29 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:33:29 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:33:29 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Connect Event: 17
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 62 KAlive msg(s) sent
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:33:35 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:33:35 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:33:43 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:33:58 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:34:01 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:34:01 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:34:03 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34

At 14:30:29, the last keepalive was received
At 14:33:29, exactly 3 minutes/180 seconds later, the hold timer expired
At 14:33:35, the HA peer comes up
At 14:34:01, the prefix is received

This is a worst-case scenario. The convergence time can be lower because the hold timer expiration depends on when the last keepalive was successfully received, so you can subtract up to 59 seconds here.

Tweaking timers

Keepalive and hold timer values can be easily changed, and we can do this at both the global and neighbor level on a FortiGate. I have set both timers to their minimum at the global level on both devices. Technically, it’s enough to only do it on the downstream peer. Note that changing these values requires a new BGP session.

config router bgp
    set as 65001
    set router-id 192.0.2.2
    set keepalive-timer 1
    set holdtime-timer 3
end

Looking at the debugs on the downstream again, we see that we get our prefix much quicker. It takes around 33 seconds now.

BGP debug with minimum keepalive and holdtimer values

2026-07-19 14:41:12 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 260 KAlive msg(s) sent
2026-07-19 14:41:13 [root] BGP: [RIB] Scanning BGP Network Routes for VRF 0...
2026-07-19 14:41:13 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:41:13 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 3
2026-07-19 14:41:13 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:41:13 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:41:13 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:41:13 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:41:13 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
...
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Cap: RR Cap(new) for all address-families
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 6
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Open Cap: Cap Code 65, Cap Len 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 261 KAlive msg(s) sent
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:16 [root] BGP: bgp_keepalive_proc: notif_rcv 4-4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:41:16 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:16 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 262 KAlive msg(s) sent
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:41:17 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
...
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:44 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 290 KAlive msg(s) sent
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 2, length 48
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: Starting UPDATE decoding... Bytes To Read (29), msg_size (29)
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:41:45 [root] BGP: [NSM] Verified NH 192.0.2.1 with NSM
2026-07-19 14:41:45 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 291 KAlive msg(s) sent
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:41:45 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!

At 14:41:12, the last keepalive is being sent
At 14:41:13, the hold timer expires
At 14:41:16, the HA peer comes up
At 14:41:45, the prefix is received

These numbers make sense because we have a hold timer of 3 seconds and an advertisement interval of 30 seconds.

Faster advertisements

It’s obvious that if we advertise routes more often, we get our routes faster.

Changing the advertisement interval on the downstream peer to 1 means we get to route convergence in about 7 seconds.

config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 1
        next
    end
end

BGP debugs with changed advertisement interval

2026-07-19 14:45:00 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 484 KAlive msg(s) sent
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 10
2026-07-19 14:45:01 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 3
2026-07-19 14:45:01 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down Hold Timer Expired"
2026-07-19 14:45:01 [root] BGP: %BGP-3-NOTIFICATION: sending to 192.0.2.1 4/0 (Hold Timer Expired/Unspecified Error Subcode) 0 data-bytes []
2026-07-19 14:45:01 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Down BGP Notification FSM-ERR"
2026-07-19 14:45:01 [root] BGP: VRF 0 NSM withdraw: 192.168.1.0/24
2026-07-19 14:45:01 [root] BGP: [GRST] Timer Announce Defer: Check VRF=0
2026-07-19 14:45:05 [root] BGP: [NETWORK] Accept Thread: Incoming conn from host 192.0.2.1 (FD=28 VRF=0)
2026-07-19 14:45:05 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 14
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Idle Event: 3
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [NETWORK] FD=28, Sock Status: 0-Success
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Connect Event: 17
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 1
...
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenSent Event: 19
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 485 KAlive msg(s) sent
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:45:07 [root] BGP: bgp_keepalive_proc: notif_rcv 4-4
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:45:07 [root] BGP: 192.0.2.1-Outgoing [FSM] State: OpenConfirm Event: 26
2026-07-19 14:45:07 id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 192.0.2.1 Up "
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 2, length 48
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: Starting UPDATE decoding... Bytes To Read (29), msg_size (29)
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Update: NLRI Len(4)
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 27
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [RIB] Update: Received Prefix 192.168.1.0/24 path_id 0
2026-07-19 14:45:08 [root] BGP: VRF 0 NSM announce: 192.168.1.0/24
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] Msg-Hdr: type 4, length 19
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [DECODE] KAlive: Received!
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 26
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [FSM] State: Established Event: 34
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 2
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Attr IP-Unicast: Tot-attr-len 21
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Update: Msg #15 Size 48
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Msg-Hdr: Type 4
2026-07-19 14:45:08 [root] BGP: 192.0.2.1-Outgoing [ENCODE] Keepalive: 486 KAlive msg(s)

At 14:45:00, the last keepalive is being sent
At 14:45:01, the hold timer expires
At 14:45:07, the HA peer comes up
At 14:45:08, the prefix is received

That’s okay, but we still lose traffic.

Some more timers

The following timers don’t actually change anything for this scenario, but they are still helpful in regular operations, and I want to highlight them. These are the scan and connect timers.

The connect-timer value determines how long the FortiGate waits before attempting a new BGP connection attempt after the previous one has failed.

The scan-time determines the interval at which the FortiGate scans for next-hop reachability, and if the next hop isn’t reachable, the route gets dropped.

config router bgp
    set as 65001
    set router-id 192.0.2.2
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set connect-timer 1
        next
    end
end

You can read more about these timers and lots of other ones in the official documentation.

Not tearing down the session with Graceful Restart

So we are down to around 8 seconds, and this is with a torn-down session on the downstream side. At this point it’s pointless to try and optimize this approach. We need the session to not get torn down on the downstream peer and keep the routes available on both sides.

If you know BGP, you already know the answer to this: Graceful Restart (GR)

Graceful Restart is designed to keep routes in the Routing Information Base (RIB) even if the peer is down. There are timers around this, but the defaults will be enough for us. Keep in mind that both sides need to support GR for this to work. Every half-decent router should be able to, but I just want to mention it.

GR can be enabled at the global and neighbor level on a FortiGate, and doing so clears your BGP sessions, either globally or only for a neighbor. I enable it on the neighbor.

70G-BGP1(Primary) # show router bgp
onfig router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set capability-graceful-restart enable
        next
    end
end

I have also unset all other previous settings, so here is the full BGP config (excluding network statements and redistribute sections) of both devices at this point:

60F-DOWNSTREAM # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "SW-BGP"
            set remote-as 65001
            set update-source "SW-BGP"
        next
    end
end

70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
end

For this exercise, it’s technically enough to enable GR only for the HA peer, but you might as well do it for both peers.

You can verify the GR capability using the get router info bgp neighbors <NEIGHBOR_IP> command.

BGP neighbor output after graceful restart

60F-DOWNSTREAM # get router info bgp neighbors 192.0.2.1
VRF 0 neighbor table:
BGP neighbor is 192.0.2.1, remote AS 65001, local AS 65001, internal link
  BGP version 4, remote router ID 192.0.2.1
  BGP state = Established, up for 00:00:23
  Last read 00:00:01, hold time is 3, keepalive interval is 1 seconds
  Configured hold time is 3, keepalive interval is 1 seconds
  Neighbor capabilities:
    Route refresh: advertised and received (old and new)
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised and received
    Address family VPNv6 Unicast: advertised and received
    Address family L2VPN EVPN: advertised and received
  Received 1509 messages, 7 notifications, 0 in queue
  Sent 1550 messages, 16 notifications, 0 in queue
  Route refresh request: received 0, sent 0
  NLRI treated as withdraw: 0
  Minimum time between advertisement runs is 1 seconds
  Update source is SW-BGP

 For address family: IPv4 Unicast
  BGP table version 19, neighbor version 18
  Index 1, Offset 0, Mask 0x2
  AF-dependant capabilities:
    Graceful restart: advertised, received, negotiated
      Forwarding states are being preserved
  Community attribute sent to this neighbor (both)
  1 accepted prefixes, 1 prefixes in rib
  1 announced prefixes

 For address family: VPNv4 Unicast
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 For address family: VPNv6 Unicast
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 For address family: L2VPN EVPN
  BGP table version 1, neighbor version 1
  Index 1, Offset 0, Mask 0x2
  Community attribute sent to this neighbor (both)
  0 accepted prefixes, 0 prefixes in rib
  0 announced prefixes

 Connections established 24; dropped 23
 Graceful-restart Status:
  Remote restart-time is 120 sec

We see Graceful restart: advertised, received, negotiated for the IPv4 Unicast address family, and we also see some additional information at the bottom regarding the remote restart time.

With GR enabled, our BGP routing behaviour is much better. It’s not interesting to show any debugs, but on a failover, traffic forwarding works fully up until the route-ttl timer, default of 10 seconds, is over (this behaviour was explained further above). At that point, connectivity is lost until the routes get announced again.

We can work and fix this with our acquired knowledge.

The last tweaks

Now we need to make sure two things happen:

Routes are announced more often
Routes are kept in the cluster’s kernel routing table for longer, or at least until we get the new routes

We know the answer to both, and the answers are advertisement-interval and route-ttl.

If we set the advertisement interval on the downstream at least lower than the route TTL of the cluster, everything should be fine. So an interval of 3 seconds with a TTL of 10 should be enough. This is theoretically correct, but it also depends on your environment. If you expect to receive a lot of routes, it might take longer to ingest them all, so consider upping the TTL past the default, or reducing the advertisement interval, or doing both.

My downstream peer is only announcing 251 routes, and with an interval of 3 and a TTL of 10, I see zero traffic loss.

The final configuration

This doesn’t differ much from the configuration posted in the GR section, except that the advertisement interval is now set to something other than the default. The route TTL is at its default of 10, but it’s still shown.

60F-DOWNSTREAM # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.2
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set activate6 disable
            set capability-graceful-restart enable
            set interface "SW-BGP"
            set remote-as 65001
            set update-source "SW-BGP"
        next
    end
end

70G-BGP1(Primary) # show router bgp
config router bgp
    set as 65001
    set router-id 192.0.2.1
    config neighbor
        edit "192.0.2.2"
            set activate6 disable
            set capability-graceful-restart enable
            set interface "port2"
            set remote-as 65001
            set update-source "port2"
        next
    end
end
config system ha
    set route-ttl 10
end

With this configuration, I was able to perform an HA failover without losing any pings or interruption of a file transfer between the downstream client and the HA client.

You can adapt this configuration with the scan and connect timers, and of course whatever else you need, but be careful about the hold and keepalive timers, lest you tear down the BGP session.

I heard about BFD

No. Bidirectional Forwarding Detection (BFD) is a great feature that helps with detecting failures in routing sessions, but in this case, it does the opposite of what we want. If BFD detects that its session is down, it will also tear down the BGP session, which would lead to traffic loss.

When it comes to combining BFD with GR, you have to consult the vendor documentation because some vendors might support this combination. Start with the assumption that this is not supported; however, Fortinet explicitly does not recommend it, as can be read here and here.

Use BFD with caution. Combining BFD with graceful restart is not recommended by Fortinet. Other vendors also explicitly mention not to combine them in configuration, as it might cause suboptimal routing performances when graceful restart and BFD are both configured.

BGP graceful restart or OSPF graceful restart doesn’t work with BFD, and it is not recommended.

Wrapping up

The fact that graceful restart was the solution to this problem wasn’t a surprise, because that’s what it is designed for, but the behaviour with route-ttl was new to me, and I am glad that I spent the time creating this post, because I have definitely learned something new. I hope that you, dear reader, have also learned something.

Tag: bgp

A real look at FortiGate FGCP HA BGP failover behaviour