KNS | ˈkɛvɪn netsec

Musings about networks and security

Multi-homed FortiGate FGCP HA BGP failover behaviour

Written by

Kevin Guenay

in

After publishing my previous post on a simple FGCP HA BGP deployment and how to reduce failover times with only a cluster and a single BGP peer, I got a comment on how this looks in a multi-homed environment, because I did not mention this scenario at all, and the way Graceful Restart (GR) works creates problems, as the comment explains. This was a totally valid comment, and after getting my hands on a fourth FortiGate and hours of testing, I can share my findings with you, dear reader.

Disclaimer

I truly did try to be diligent in my testing, but with all the configurations, options, and behaviours, something might have slipped by me. I hope that all the numbers I provide are correct, and I am truly sorry if I made a mistake anywhere.

If you end up using anything I write here, I urge you to test, test, and test. My lab is not your production environment, and you are probably not working with only FortiGates. Not every vendor implements BGP and its various mechanisms the same way, so the validation will ultimately be up to you.

If you are only interested in the conclusion, jump to the “Conclusion and case recommendations” section.

With that out of the way, let’s get to… 

The setup

  • 2x FortiGate 70G on 7.6.7
  • FortiGate 60F on 7.6.7
  • FortiGate 40F on 7.6.7
  • 2x Windows 11 clients on 25H2

The 70G cluster, using AS 65001, is peering with the 60F (ISP1), using AS 65003, and the 40F (ISP2), using AS 65004. Direct links, links connected to a switch or loopbacks are used (this depends on the cases being examined).

The 70G cluster and the ISP FortiGates have a client behind them. The ISP FortiGates serve as the gateway for the client using VRRP, but this isn’t mentioned in the topologies, won’t be a thing in the real world and is purely there to make it easier for me in my lab environment.

The 70G cluster is announcing 192.168.1.0/24, and the ISP FortiGate are each announcing a default route.

What am I trying to achieve?

The two main points I want to achieve are:

  • An HA failover should not lead to any or minimal traffic loss
    • HA failovers are triggered using diagnose sys ha reset-uptime
  • An ISP failover should not lead to any or minimal traffic loss
    • ISP failovers are triggered by shutting down the physical interface

I also have a few points that are important for this article:

  • I recommend reading my previous post first, because lots of things, like route-ttl, are important here too.
  • I always take the worst result I have experienced in my testing.
    • If one test resulted in 1 lost ping and another in 3 lost pings, I will only mention the test with 3 lost pings.
  • I test both with and without SNAT.
    • snat-route-change is enabled even if it doesn’t matter for my tests; it might matter to you.
  • All ping tests are performed by using a Windows ping with the -t and –w 100 options, so continuous pings with a timeout of 100 milliseconds. This is important, because when you read “3 pings lost with SNAT”, you have to keep in mind that these aren’t standard pings with a 4-second timeout.
    • Using a low timeout value is necessary because once we get to BFD, sub-second failovers are the name of the game.
  • I am using a scan-time of 5, an advertisement-interval of 3, and a connect-timer of 1. This is done both because it helps me with my testing and because we want to put our best foot forward.
    • The advertisement-interval can be taken down to 1, but I simply opted not to do that, because if this becomes a problem, the configuration is bad.
  • I am splitting this up into cases, where each case represents a specific configuration; the results and, if necessary, explanations.
  • I am keeping the BGP keepalive and holdtime-timer at the default values of 60 and 180, respectively. I will mention if they can make a difference.
  • BFD timers are left at the defaults.

Now, let’s get to testing.

Case 1: GR enabled, BGP on interface, no BFD, direct links

Multi-homed BGP on direct links

This is similar to the topology of my previous post except that a second ISP joined. Graceful Restart is enabled, BFD is not configured, and the cluster has direct connections to both ISPs.

Results:

  • HA failover: No pings lost
  • ISP failover: 4 pings lost with and without SNAT

This is a special case because direct links to two ISPs is probably not something you will encounter in the real world; however, BGP’s default behaviour of tearing down a peer if the interface is down helps here.

We can observe this behaviour by turning on BGP debugs.

diagnose debug console timestamp enable
diagnose ip router bgp all enable
diagnose ip router bgp level info
diagnose debug enable
[root] BGP: NSM Message Header
[root] BGP:  VR ID: 1
[root] BGP:  VRF ID: 0
[root] BGP:  Message type: Link Down (30)
[root] BGP:  Message length: 52
[root] BGP:  Message ID: 0x00000000
[root] BGP: NSM Interface
[root] BGP:  Interface index: 7
[root] BGP:  Name: port3
[root] BGP:  Flags: 4354

We can clearly see the Link Down message type, and that signals that the attached BGP peer should be torn down, and at that point traffic only has one exit, the other ISP.

For completeness sake, here is an example of a Link Up message:

[root] BGP: NSM Message Header
[root] BGP:  VR ID: 1
[root] BGP:  VRF ID: 0
[root] BGP:  Message type: Link Up (29)
[root] BGP:  Message length: 60
[root] BGP:  Message ID: 0x00000000
[root] BGP: NSM Interface
[root] BGP:  Interface index: 8
[root] BGP:  Name: port4
[root] BGP:  Flags: 4419
[root] BGP:  Bandwidth: 0.000000

And the relevant configuration of the cluster and the ISPs:

Case 1 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system ha
    set route-ttl 30
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit 172.31.255.10
            set advertisement-interval 3
            set capability-graceful-restart enable
            set soft-reconfiguration enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit 172.31.254.10
            set advertisement-interval 3
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set soft-reconfiguration enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!

config router bgp
    set as 65004
    set router-id 192.0.2.4
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

Case 2: GR enabled, BGP on interface, no BFD, switched links

Multi-homed BGP on switched links

The only thing that changed is that now the FortiGates are all connected to a switch, and each ISP gets a VLAN so the peers can find each other. This is probably the most common topology, because getting two links from an ISP can be a challenge in itself.

Results:

  • HA failover: No pings lost
  • ISP failover: Complete traffic loss, because of GR

This is expected, because the Link Down message is never generated, since the link to the switch stays up on the cluster’s side and traffic is blackholed. At this point, you are at the mercy of the BGP hold timer.

The configuration is the same as case 1.

Case 3: GR enabled, BGP on interface, BFD enabled in BGP, switched links

Refer to case 2 for the topology.

The topology stays the same as with case 2, but BFD capabilities are now enabled in the settings and for BGP.

Results:

  • HA Failover:
    • 1 ping lost with SNAT
    • 6 pings lost without SNAT
  • ISP failover:
    • 3 pings lost with and without SNAT

The HA failover scenario without SNAT is surprising, but that’s how it is. I even tested it again while writing this sentence.

The loss of pings on an HA failover is attributed to the fact that BFD is not synchronized in HA and the BFD session needs to re-establish on the new primary, which takes a bit over 5 seconds. Once the BFD session is up, forwarding works for non-SNAT sessions. 

After BFD, the BGP session needs to re-establish, so the synchronized HA routes, thanks to route-ttl, are being used in the meantime.

Enabling BFD debugs and a debug flow shows the behaviour for a non-SNAT scenario:

diagnose debug console timestamp enable
diagnose debug application bfdd -1
diagnose ip router bfd all enable
diagnose debug enable

HA failover with SNAT

2026-08-23 15:13:55 bfd_ha_event: Got HA MESSAGE, msg->nvdoms=1
2026-08-23 15:13:55 bfd_ha_event: HA state vfid=0 old=78336 new=8704
2026-08-23 15:13:55 bfd_ha_event: vd=root is master
2026-08-23 15:13:55 bfd_cmdb_vd_callback: VD=root no bfd neighbours or multihops.
2026-08-23 15:13:56 id=65308 trace_id=512 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1002."
2026-08-23 15:13:56 id=65308 trace_id=512 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-0000077f, original direction"
2026-08-23 15:13:56 id=65308 trace_id=512 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.255.10 via port3"
2026-08-23 15:13:56 id=65308 trace_id=512 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-23 15:13:56 id=65308 trace_id=512 func=get_new_addr line=1331 msg="find SNAT: IP-172.31.255.1(from IPPOOL), port-5132"
2026-08-23 15:13:56 id=65308 trace_id=512 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-23 15:13:57 id=65308 trace_id=513 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1003."
2026-08-23 15:13:57 id=65308 trace_id=513 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-0000077f, original direction"
2026-08-23 15:13:57 id=65308 trace_id=513 func=npu_handle_session44 line=1460 msg="Trying to offloading session from wan1 to port3, skb.npu_flag=00000400 ses.state=00010204 ses.npu_state=0x00000000"
2026-08-23 15:13:57 id=65308 trace_id=513 func=ip_session_install_npu_session line=445 msg="npu session installation succeeded"
2026-08-23 15:13:57 id=65308 trace_id=513 func=fw_forward_dirty_handler line=445 msg="state=00010204, state2=00000401, npu_state=00000400"
2026-08-23 15:13:57 id=65308 trace_id=513 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-23 15:13:58 id=65308 trace_id=514 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1004."
2026-08-23 15:13:58 id=65308 trace_id=514 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-0000077f, original direction"
2026-08-23 15:13:58 id=65308 trace_id=514 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-23 15:13:58 id=65308 trace_id=514 func=get_new_addr line=1331 msg="find SNAT: IP-172.31.255.1(from IPPOOL), port-5133"
2026-08-23 15:13:58 id=65308 trace_id=514 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-23 15:13:59 id=65308 trace_id=515 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1005."
2026-08-23 15:13:59 id=65308 trace_id=515 func=init_ip_session_common line=6607 msg="allocate a new session-00000588"
2026-08-23 15:13:59 id=65308 trace_id=515 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.255.10 via port3"
2026-08-23 15:13:59 id=65308 trace_id=515 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-23 15:13:59 id=65308 trace_id=515 func=get_new_addr line=1331 msg="find SNAT: IP-172.31.255.1(from IPPOOL), port-5118"
2026-08-23 15:13:59 id=65308 trace_id=515 func=fw_forward_handler line=1014 msg="Allowed by Policy-1: SNAT"
2026-08-23 15:13:59 id=65308 trace_id=515 func=ip_session_confirm_final line=3357 msg="npu_state=0x0, hook=4"
2026-08-23 15:13:59 id=65308 trace_id=515 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-23 15:14:00 id=65308 trace_id=516 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1006."
2026-08-23 15:14:00 id=65308 trace_id=516 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000588, original direction"
2026-08-23 15:14:00 id=65308 trace_id=516 func=npu_handle_session44 line=1460 msg="Trying to offloading session from wan1 to port3, skb.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000000"
2026-08-23 15:14:00 id=65308 trace_id=516 func=ip_session_install_npu_session line=445 msg="npu session installation succeeded"
2026-08-23 15:14:00 id=65308 trace_id=516 func=fw_forward_dirty_handler line=445 msg="state=04010204, state2=00000001, npu_state=00000400"
2026-08-23 15:14:00 id=65308 trace_id=516 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-23 15:14:01 bfd_nl_event: processing nl event...
2026-08-23 15:14:01 bfd_nl_event: processing nl event type=1 pid=1628
2026-08-23 15:14:01 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49162,port4,B]: bfd_session_send_packet_now
2026-08-23 15:14:01 bfd_nl_process_register: reg 172.31.254.1->172.31.254.10 ifindex=8
2026-08-23 15:14:01 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49162,port4,B]: state DOWN -> INIT local_diag=0x00
2026-08-23 15:14:01 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49162,port4,B]: bfd_session_send_packet_now
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:01 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49162,port4,B]: state INIT -> UP local_diag=0x00
2026-08-23 15:14:01 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49162,port4,B]: bfd_session_send_packet_now
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:01 bfd_write:Session[172.31.254.1->172.31.254.10,49162,port4,B]: Sending BFD packet
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:01 bfd_write:Session[172.31.254.1->172.31.254.10,49162,port4,B]: Sending BFD packet
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:01 bfd_write:Session[172.31.254.1->172.31.254.10,49162,port4,B]: Sending BFD packet
2026-08-23 15:14:01 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49162,port4,B]: received BFD packet
2026-08-23 15:14:02 bfd_write:Session[172.31.254.1->172.31.254.10,49162,port4,B]: Sending BFD packet
2026-08-23 15:14:02 bfd_nl_event: processing nl event...
2026-08-23 15:14:02 bfd_nl_event: processing nl event type=1 pid=1628
2026-08-23 15:14:02 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49163,port3,C]: bfd_session_send_packet_now
2026-08-23 15:14:02 bfd_nl_process_register: reg 172.31.255.1->172.31.255.10 ifindex=7
2026-08-23 15:14:02 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49163,port3,C]: state DOWN -> INIT local_diag=0x00
2026-08-23 15:14:02 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49163,port3,C]: bfd_session_send_packet_now
2026-08-23 15:14:02 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49163,port3,C]: received BFD packet
2026-08-23 15:14:02 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49163,port3,C]: state INIT -> UP local_diag=0x00
2026-08-23 15:14:02 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49163,port3,C]: bfd_session_send_packet_now

In the debugs, we can see that SNAT, policy lookup and forwarding happen before BFD is even established on the new primary.

Looking at the same case, an HA failover, but without SNAT, it doesn’t look much different, except that real forwarding only happens after BFD has been established, which you can’t even see in the debugs.

HA failover without SNAT

2026-08-23 15:22:00 bfd_ha_event: Got HA MESSAGE, msg->nvdoms=1
2026-08-23 15:22:00 bfd_ha_event: HA state vfid=0 old=78336 new=8704
2026-08-23 15:22:00 bfd_ha_event: vd=root is master
2026-08-23 15:22:00 bfd_cmdb_vd_callback: VD=root no bfd neighbours or multihops.
2026-08-23 15:22:00 id=65308 trace_id=527 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1419."
2026-08-23 15:22:00 id=65308 trace_id=527 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000914, original direction"
2026-08-23 15:22:00 id=65308 trace_id=527 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.254.10 via port4"
2026-08-23 15:22:00 id=65308 trace_id=527 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=32, len=4"
2026-08-23 15:22:01 id=65308 trace_id=528 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1420."
2026-08-23 15:22:01 id=65308 trace_id=528 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000914, original direction"
2026-08-23 15:22:01 id=65308 trace_id=528 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.254.10 via port4"
2026-08-23 15:22:01 id=65308 trace_id=528 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=32, len=4"
2026-08-23 15:22:03 id=65308 trace_id=529 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1421."
2026-08-23 15:22:03 id=65308 trace_id=529 func=init_ip_session_common line=6607 msg="allocate a new session-00000791"
2026-08-23 15:22:03 id=65308 trace_id=529 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.255.10 via port3"
2026-08-23 15:22:03 id=65308 trace_id=529 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-23 15:22:03 id=65308 trace_id=529 func=fw_forward_handler line=1014 msg="Allowed by Policy-1:"
2026-08-23 15:22:03 id=65308 trace_id=529 func=ip_session_confirm_final line=3357 msg="npu_state=0x1, hook=4"
2026-08-23 15:22:04 id=65308 trace_id=530 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1422."
2026-08-23 15:22:04 id=65308 trace_id=530 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000791, original direction"
2026-08-23 15:22:04 id=65308 trace_id=530 func=npu_handle_session44 line=1460 msg="Trying to offloading session from wan1 to port3, skb.npu_flag=00000400 ses.state=04000204 ses.npu_state=0x00000001"
2026-08-23 15:22:04 id=65308 trace_id=530 func=fw_forward_dirty_handler line=445 msg="state=04000204, state2=00000001, npu_state=00000001"
2026-08-23 15:22:06 bfd_nl_event: processing nl event...
2026-08-23 15:22:06 bfd_nl_event: processing nl event type=1 pid=1628
2026-08-23 15:22:06 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49166,port3,F]: bfd_session_send_packet_now
2026-08-23 15:22:06 bfd_nl_process_register: reg 172.31.255.1->172.31.255.10 ifindex=7
2026-08-23 15:22:06 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49166,port3,F]: state DOWN -> UP local_diag=0x00
2026-08-23 15:22:06 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49166,port3,F]: bfd_session_send_packet_now
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 id=65308 trace_id=531 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1423."
2026-08-23 15:22:06 id=65308 trace_id=531 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000791, original direction"
2026-08-23 15:22:06 id=65308 trace_id=531 func=npu_handle_session44 line=1460 msg="Trying to offloading session from wan1 to port3, skb.npu_flag=00000400 ses.state=04000204 ses.npu_state=0x00000001"
2026-08-23 15:22:06 id=65308 trace_id=531 func=fw_forward_dirty_handler line=445 msg="state=04000204, state2=00000001, npu_state=00000001"
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:06 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:06 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:07 bfd_nl_event: processing nl event...
2026-08-23 15:22:07 bfd_nl_event: processing nl event type=1 pid=1628
2026-08-23 15:22:07 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49167,port4,10]: bfd_session_send_packet_now
2026-08-23 15:22:07 bfd_nl_process_register: reg 172.31.254.1->172.31.254.10 ifindex=8
2026-08-23 15:22:07 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49167,port4,10]: state DOWN -> INIT local_diag=0x00
2026-08-23 15:22:07 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49167,port4,10]: bfd_session_send_packet_now
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49167,port4,10]: state INIT -> UP local_diag=0x00
2026-08-23 15:22:07 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49167,port4,10]: bfd_session_send_packet_now
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.254.1->172.31.254.10,49167,port4,10]: Sending BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.254.1->172.31.254.10,49167,port4,10]: Sending BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49166,port3,F]: received BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.255.1->172.31.255.10,49166,port3,F]: Sending BFD packet
2026-08-23 15:22:07 bfd_write:Session[172.31.254.1->172.31.254.10,49167,port4,10]: Sending BFD packet
2026-08-23 15:22:07 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49167,port4,10]: received BFD packet
2026-08-23 15:22:07 id=65308 trace_id=532 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.30:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=1424."
2026-08-23 15:22:07 id=65308 trace_id=532 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00000791, original direction"
2026-08-23 15:22:07 id=65308 trace_id=532 func=ipv4_fast_cb line=53 msg="enter fast path"

Now, why does this ping loss not happen with SNAT enabled, especially when the debug tells a different story? I genuinely do not know, but that is what I have observed.

Here are the debugs of an ISP failover:

ISP failover with SNAT

2026-08-22 20:03:40 bfd_session_inactivity_timer: Session[172.31.254.1->172.31.254.10,49209,port4,3A]: Inactivity timer expired
2026-08-22 20:03:40 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49209,port4,3A]: state UP -> DOWN local_diag=0x01
2026-08-22 20:03:40 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49209,port4,3A]: bfd_session_send_packet_now
2026-08-22 20:03:40 bfd_nl_event: processing nl event...
2026-08-22 20:03:40 bfd_nl_event: processing nl event type=2 pid=1629
2026-08-22 20:03:40 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49209,port4,3A]: state DOWN -> ADMIN DOWN local_diag=0x27
2026-08-22 20:03:40 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49209,port4,3A]: bfd_session_send_packet_now
2026-08-22 20:03:40 bfd_nl_process_unregister: reg 172.31.254.1->172.31.254.10 ifindex=8
2026-08-22 20:03:40 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:40 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:40 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:40 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:40 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:40 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:40 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:40 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:40 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:41 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:41 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:41 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:41 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:41 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:41 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:41 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:41 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:42 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:42 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:42 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:42 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:42 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:42 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:42 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:42 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:42 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:42 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:43 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:43 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:43 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:43 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:43 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:43 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:43 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:43 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:44 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:44 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:44 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:44 id=65308 trace_id=32 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.10:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=9245."
2026-08-22 20:03:44 id=65308 trace_id=32 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00004836, original direction"
2026-08-22 20:03:44 id=65308 trace_id=32 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.255.10 via port3"
2026-08-22 20:03:44 id=65308 trace_id=32 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-22 20:03:44 id=65308 trace_id=32 func=get_new_addr line=1331 msg="find SNAT: IP-172.31.255.1(from IPPOOL), port-5118"
2026-08-22 20:03:44 id=65308 trace_id=32 func=fw_strict_dirty_session_check line=297 msg="SNAT IP 172.31.254.1 != 172.31.255.1, drop"
2026-08-22 20:03:44 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:44 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:44 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:44 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:44 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:44 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:44 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:45 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:45 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:45 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:45 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:45 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:45 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:45 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:45 id=65308 trace_id=33 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.10:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=9246."
2026-08-22 20:03:45 id=65308 trace_id=33 func=init_ip_session_common line=6607 msg="allocate a new session-00004844"
2026-08-22 20:03:45 id=65308 trace_id=33 func=__vf_ip_route_input_rcu line=2116 msg="find a route: flag=00000000 gw-172.31.255.10 via port3"
2026-08-22 20:03:45 id=65308 trace_id=33 func=__iprope_tree_check line=529 msg="gnum-100004, use int hash, slot=31, len=4"
2026-08-22 20:03:45 id=65308 trace_id=33 func=get_new_addr line=1331 msg="find SNAT: IP-172.31.255.1(from IPPOOL), port-5118"
2026-08-22 20:03:45 id=65308 trace_id=33 func=fw_forward_handler line=1014 msg="Allowed by Policy-1: SNAT"
2026-08-22 20:03:45 id=65308 trace_id=33 func=ip_session_confirm_final line=3357 msg="npu_state=0x0, hook=4"
2026-08-22 20:03:45 id=65308 trace_id=33 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"
2026-08-22 20:03:45 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:45 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:46 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:46 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:46 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:46 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:46 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:46 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:46 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49208,port3,39]: received BFD packet
2026-08-22 20:03:46 bfd_write:Session[172.31.255.1->172.31.255.10,49208,port3,39]: Sending BFD packet
2026-08-22 20:03:46 id=65308 trace_id=34 func=print_pkt_detail line=6393 msg="vd-root:0 received a packet(proto=1, 192.168.1.231:1->10.10.100.10:2048) tun_id=0.0.0.0 from wan1. type=8, code=0, id=1, seq=9247."
2026-08-22 20:03:46 id=65308 trace_id=34 func=resolve_ip_tuple_fast line=6501 msg="Find an existing session, id-00004844, original direction"
2026-08-22 20:03:46 id=65308 trace_id=34 func=npu_handle_session44 line=1460 msg="Trying to offloading session from wan1 to port3, skb.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000000"
2026-08-22 20:03:46 id=65308 trace_id=34 func=ip_session_install_npu_session line=445 msg="npu session installation succeeded"
2026-08-22 20:03:46 id=65308 trace_id=34 func=fw_forward_dirty_handler line=445 msg="state=04010204, state2=00000001, npu_state=00000400"
2026-08-22 20:03:46 id=65308 trace_id=34 func=__ip_session_run_tuple line=3682 msg="SNAT 192.168.1.231->172.31.255.1:5118"

In the first line, the inactivity timer expires, and the BFD session for port4 (ISP2) goes down. On the next packet, it takes a new route (via port3, i.e. ISP1), with new SNAT, but gets dropped because the SNAT IP is wrong. This gets corrected on the next packet, is allowed by policy 1, and forwarding happens as we expect.

Here is the configuration at this point:

Case 3 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system ha
    set route-ttl 30
end
config system settings
    set bfd enable
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "172.31.255.10"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set soft-reconfiguration enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit "172.31.254.10"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set soft-reconfiguration enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.255.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!
config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.254.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

I don’t want to spend too much time figuring out why this behaviour is as it is, because we can do better and that requires a static BFD neighbour.

Static BFD neighbours

The documentation calls it “BFD enabled BGP graceful restart”, but I will call it static BFD going forward to keep it short.

You can read the explanation, but the two important parts are:

  • Static BFD neighbors will start in the INIT state instead of DOWN
  • If a discriminator (discr) is received that doesn’t exist, the BFD packet isn’t discarded, but a new session is intialized

In practice, this means that BFD sessions are immediately up on the new primary if an HA failover is performed and, in turn, the BGP session isn’t being taken down.

Two things of note with static BFD:

  • The neighbor needs to be in the same subnet as the port the neighbor is configured on.
    • This becomes a problem in a later case.
  • It is enough to configure static BFD on the FortiGate cluster; you do not need to configure it on any other device to benefit from it.
    • This point is especially helpful, because if it would require such a configuration on both sides, it would become less useful.

Here is a debug of this behaviour:

70G-BGP1(Secondary) # 2026-08-22 17:43:14 bfd_ha_event: Got HA MESSAGE, msg->nvdoms=1
2026-08-22 17:43:14 bfd_ha_event: HA state vfid=0 old=78336 new=8704
2026-08-22 17:43:14 bfd_ha_event: vd=root is master
2026-08-22 17:43:14 bfd_nl_event: processing nl event...
2026-08-22 17:43:14 bfd_nl_event: processing nl event type=5 pid=1654
2026-08-22 17:43:15 bfd_packet_process: Can't find session with discr 0x00000004
2026-08-22 17:43:15 bfd_packet_process: Choose session with discr 0x00000008 for 0x00000004
2026-08-22 17:43:15 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49159,port4,8]: state INIT -> UP local_diag=0x00
2026-08-22 17:43:15 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49159,port4,8]: bfd_session_send_packet_now
2026-08-22 17:43:15 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49159,port4,8]: received BFD packet
2026-08-22 17:43:15 bfd_packet_process: Can't find session with discr 0x00000003
2026-08-22 17:43:15 bfd_packet_process: Choose session with discr 0x00000007 for 0x00000003
2026-08-22 17:43:15 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49158,port3,7]: state INIT -> UP local_diag=0x00
2026-08-22 17:43:15 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49158,port3,7]: bfd_session_send_packet_now

The HA event gets triggered, no session is being found with discr 0x00000004, and the session with discr 0x00000008 is chosen. The BFD neighbor on port4 (ISP2) then transitions from INIT to UP, and the same process happens on port3 (ISP1)

An HA failover with regular BFD looks like this:

70G-BGP1(Secondary) # 2026-08-22 17:47:09 bfd_ha_event: Got HA MESSAGE, msg->nvdoms=1
2026-08-22 17:47:09 bfd_ha_event: HA state vfid=0 old=78336 new=8704
2026-08-22 17:47:09 bfd_ha_event: vd=root is master
2026-08-22 17:47:09 bfd_cmdb_vd_callback: VD=root no bfd neighbours or multihops.
2026-08-22 17:47:15 bfd_nl_event: processing nl event...
2026-08-22 17:47:15 bfd_nl_event: processing nl event type=1 pid=1629
2026-08-22 17:47:15 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49160,port4,9]: bfd_session_send_packet_now
2026-08-22 17:47:15 bfd_nl_process_register: reg 172.31.254.1->172.31.254.10 ifindex=8
2026-08-22 17:47:15 bfd_session_change_state: Session[172.31.254.1->172.31.254.10,49160,port4,9]: state DOWN -> UP local_diag=0x00
2026-08-22 17:47:15 bfd_session_send_packet_now: Session[172.31.254.1->172.31.254.10,49160,port4,9]: bfd_session_send_packet_now
2026-08-22 17:47:15 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:15 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:16 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:16 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:16 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:16 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:16 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:16 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:16 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:16 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:17 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:17 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:17 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:17 bfd_read_ipv4: Session[172.31.254.1->172.31.254.10,49160,port4,9]: received BFD packet
2026-08-22 17:47:17 bfd_write:Session[172.31.254.1->172.31.254.10,49160,port4,9]: Sending BFD packet
2026-08-22 17:47:17 bfd_nl_event: processing nl event...
2026-08-22 17:47:17 bfd_nl_event: processing nl event type=1 pid=1629
2026-08-22 17:47:17 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49161,port3,A]: bfd_session_send_packet_now
2026-08-22 17:47:17 bfd_nl_process_register: reg 172.31.255.1->172.31.255.10 ifindex=7
2026-08-22 17:47:17 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49161,port3,A]: state DOWN -> INIT local_diag=0x00
2026-08-22 17:47:17 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49161,port3,A]: bfd_session_send_packet_now
2026-08-22 17:47:17 bfd_read_ipv4: Session[172.31.255.1->172.31.255.10,49161,port3,A]: received BFD packet
2026-08-22 17:47:17 bfd_session_change_state: Session[172.31.255.1->172.31.255.10,49161,port3,A]: state INIT -> UP local_diag=0x00
2026-08-22 17:47:17 bfd_session_send_packet_now: Session[172.31.255.1->172.31.255.10,49161,port3,A]: bfd_session_send_packet_now

The only interesting part, except the missing discriminator lines, is that neighbors start in the DOWN state.

Let’s enable static BFD for case 3 and see what the results are.

Case 4: GR enabled, BGP on interface, BFD enabled in BGP and static BFD, switched links

Refer to case 2 for the topology.

Static BFD is now enabled alongside BGP BFD.

Results:

  • HA failover: No pings lost with or without SNAT
  • ISP failover:
    • 4 pings lost with SNAT
    • 3 pings lost without SNAT

It is a bit pointless to show, but the BFD sessions are immediately up on the new primary, as we expect from the way static BFD works. BGP needs to re-establish, however, so keep that in mind when configuring route-ttl.

There isn’t that much to say here. I can already tell you that this will be one of the recommended configurations, except for one thing that gets mentioned in the “Can one do better?” section.

Case 4 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system settings
    set bfd enable
end
config router bfd
    config neighbor
        edit 172.31.254.10
            set interface "port4"
        next
        edit 172.31.255.10
            set interface "port3"
        next
    end
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "172.31.255.10"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit "172.31.254.10"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set soft-reconfiguration enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.255.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65004
    set router-id 192.0.2.4
    set scan-time 5
    config neighbor
        edit "172.31.254.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

Case 5: GR enabled, BGP on loopback, BFD enabled in BGP with multihop, switched links

Multi-homed BGP on loopbacks

So, you configured BGP on loopbacks. Great, I love loopbacks, but it comes with a bit of baggage. First, you have to configure BFD multihop; otherwise, BFD will not establish. Second, you cannot configure static BFD.

As mentioned in the static BFD section, the neighbors need to be in the same subnet as the interface you configure them on. Configuring loopbacks with anything but a /32 leads to hilarious results, like the FortiGate answering to its own pings and trying to establish a BGP peering with itself.

Here is an example on the HA cluster (loopback IP 192.0.2.1)

70G-BGP1(Primary) # execute ping 192.0.2.3
PING 192.0.2.3 (192.0.2.3): 56 data bytes
64 bytes from 192.0.2.3: icmp_seq=0 ttl=255 time=0.0 ms
64 bytes from 192.0.2.3: icmp_seq=1 ttl=255 time=0.0 ms
64 bytes from 192.0.2.3: icmp_seq=2 ttl=255 time=0.0 ms
64 bytes from 192.0.2.3: icmp_seq=3 ttl=255 time=0.0 ms
64 bytes from 192.0.2.3: icmp_seq=4 ttl=255 time=0.0 ms

--- 192.0.2.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

[root] BGP: [NETWORK] Accept Thread: Incoming conn from host 192.0.2.1 (FD=30 VRF=0)
[root] BGP: [NETWORK] Accept Thread: 192.0.2.1 - No such Peer configured

Note the constant RTT of 0.0 ms, despite pinging a different host, and the incoming peer connection from 192.0.2.1.

This means static BFD is out of the question.

On to the results.

Results:

  • HA failover:
    • 1 ping lost with SNAT
    • 6 pings lost without SNAT
  • ISP failover:
    • 4 pings lost with SNAT
    • 3 pings lost without SNAT

This case is overall very similar to case 3, except it runs on loopbacks.

Case 5 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system ha
    set route-ttl 30
end
config system settings
    set bfd enable
end
config router bfd
    config multihop-template
        edit 1
            set src 192.0.2.0 255.255.255.0
            set dst 192.0.2.0 255.255.255.0
        next
    end
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "192.0.2.3"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set soft-reconfiguration enable
            set interface "LO_BGP"
            set remote-as 65003
            set connect-timer 1
            set update-source "LO_BGP"
        next
        edit "192.0.2.4"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set soft-reconfiguration enable
            set interface "LO_BGP"
            set remote-as 65004
            set connect-timer 1
            set update-source "LO_BGP"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config router bfd
    config multihop-template
        edit 1
            set src 192.0.2.0 255.255.255.0
            set dst 192.0.2.0 255.255.255.0
        next
    end
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set ebgp-enforce-multihop enable
            set soft-reconfiguration enable
            set interface "LO_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "LO_BGP"
        next
    end
end

!!!ISP2!!!

config router bfd
    config multihop-template
        edit 1
            set src 192.0.2.0 255.255.255.0
            set dst 192.0.2.0 255.255.255.0
        next
    end
end
config router bgp
    set as 65004
    set router-id 192.0.2.4
    set scan-time 5
    config neighbor
        edit "192.0.2.1"
            set advertisement-interval 3
            set bfd enable
            set capability-graceful-restart enable
            set capability-default-originate enable
            set ebgp-enforce-multihop enable
            set soft-reconfiguration enable
            set interface "LO_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "LO_BGP"
        next
    end
end

No Graceful Restart cases

Let’s see what happens if GR is disabled, because the results are surprising.

Case 6: GR disabled, BGP on loopback, BFD enabled in BGP with multihop, switched links

Refer to case 5 for the topology.

Results:

  • HA failover:
    • 0 pings lost with SNAT
    • 6 pings lost without SNAT
  • ISP failover:
    • 4 pings lost with SNAT
    • 3 pings lost without SNAT

On an HA failover, BFD and BGP go down, and once the BGP session is up, traffic is being forwarded once again.

SNAT continues to be in its own world on an HA failover.

Case 3 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system ha
    set route-ttl 30
end
config system settings
    set bfd enable
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "172.31.255.10"
            set advertisement-interval 3
            set bfd enable
            set soft-reconfiguration enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit "172.31.254.10"
            set advertisement-interval 3
            set bfd enable
            set soft-reconfiguration enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.255.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.254.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

Case 7: GR disabled, BGP on interface, BFD enabled in BGP, switched links

Refer to case 2 for the topology.

Results:

  • HA failover:
    • 1 ping lost with SNAT
    • 6 pings lost without SNAT
  • ISP failover:
    • 4 pings lost with SNAT
    • 3 pings lost without SNAT

The results are very similar to what we’ve seen before.

No configuration provided, because it’s the config from case 3 without GR.

Case 8: GR disabled, BGP on interface, static BFD only, switched links

Refer to case 2 for the topology.

Results:

  • HA failover: Complete traffic loss after route-ttl expires
  • ISP failover: Complete traffic loss

For the HA failover scenario, traffic forwarding works until the synchronized routes expire, so technically you can work around this by increasing the route-ttl setting. BFD sessions are up on the new primary.

During an ISP failover, BGP waits for the hold timer to expire, leading to traffic being blackholed.

There is no difference between SNAT and no-SNAT.

Case 8 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system settings
    set bfd enable
end
config router bfd
    config neighbor
        edit 172.31.254.10
            set interface "port4"
        next
        edit 172.31.255.10
            set interface "port3"
        next
    end
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "172.31.255.10"
            set advertisement-interval 3
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit "172.31.254.10"
            set advertisement-interval 3
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.255.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65004
    set router-id 192.0.2.4
    set scan-time 5
    config neighbor
        edit "172.31.254.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

Case 9: GR disabled, BGP on interface, BFD enabled in BGP and static BFD, switched links

Refer to case 2 for the topology.

Results:

  • HA failover: 0 pings lost with and without SNAT
  • ISP failover:
    • 3 pings lost with SNAT
    • 2 pings lost without SNAT

The ISPs never lose the routes during an HA failover, by the way.

This is a contender for the best configuration.

Case 9 configuration for cluster and ISPs

!!!HA CLUSTER!!!

config system global
    set snat-route-change enable
end
config system settings
    set bfd enable
end
config router bfd
    config neighbor
        edit 172.31.254.10
            set interface "port4"
        next
        edit 172.31.255.10
            set interface "port3"
        next
    end
end
config router bgp
    set as 65001
    set router-id 192.0.2.1
    set ebgp-multipath enable
    set additional-path enable
    set scan-time 5
    config neighbor
        edit "172.31.255.10"
            set advertisement-interval 3
            set bfd enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "port3"
            set remote-as 65003
            set connect-timer 1
            set update-source "port3"
        next
        edit "172.31.254.10"
            set advertisement-interval 3
            set bfd enable
            set soft-reconfiguration enable
            set interface "port4"
            set remote-as 65004
            set connect-timer 1
            set update-source "port4"
        next
    end
    config network
        edit 1
            set prefix 192.168.1.0 255.255.255.0
        next
    end
end

!!!ISP1!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65003
    set router-id 192.0.2.3
    set scan-time 5
    config neighbor
        edit "172.31.255.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

!!!ISP2!!!

config system settings
    set bfd enable
end
config router bgp
    set as 65004
    set router-id 192.0.2.4
    set scan-time 5
    config neighbor
        edit "172.31.254.1"
            set advertisement-interval 3
            set bfd enable
            set capability-default-originate enable
            set soft-reconfiguration enable
            set link-down-failover enable
            set interface "SW_BGP"
            set remote-as 65001
            set connect-timer 1
            set update-source "SW_BGP"
        next
    end
end

Can one do better?

BGP settings

I have already set the most relevant BGP timers to low or minimum settings (scan-time, advertisement-interval, and connect-timer), and outside of setting the advertisement-interval to 1, there isn’t anything of relevance we can tweak, that I am aware of at least. The keepalive and holdtime-timer settings should never be relevant in a configured scenario (it would matter for case 8, for example, but you can easily do something better).

BFD timers

I did change the default BFD timers from 250/250/3 to 100/100/2 for a few tests, and I sometimes noticed less traffic disruption, but this is quite difficult to test in a lab environment, because it can just be a fluke, and the lower your BFD timers are, the higher the risk of tearing the session down by accident. Do test it yourself, however and see if it makes a meaningful difference.

FortiGate Session Life Support Protocol (FGSP)

Before starting my tests, I thought that FGSP would be the better option, but now I am not sure. With static BFD, the problem regarding BFD sessions not being synchronized is nonexistent, so I don’t know if FGSP brings any meaningful benefits here. You still need to take care of the first hop on the inside with FGSP, and I don’t know if VRRP will perform better than FGCP.

Disabling hardware offloading

This one’s a doozy. If I disabled hardware offloading in a policy (set auto-asic-offload disable), I consistently lost fewer pings. Obviously, this comes with the problem of traffic not being offloaded, but if you can stomach it for some traffic, this seems like a viable option. The reason for less traffic disruption is probably that everything is being handled in the kernel, instead of the kernel + the NPU, but I am not 100% sure.

I did not test this extensively.

Conclusion and case recommendations

With all that done, what conclusions can we draw?

  • Enable BFD in BGP and static BFD, if available
    • You can’t use static BFD with loopbacks
  • It seemingly does not matter if you enable or disable GR to get the best outcome presented here, but the rest of your configuration needs to be correct.
  • Static BFD is great

The main decision is: Are you running BGP on your interface or on loopbacks?

If it’s BGP on interface: Look at case 4 or case 9. The only difference is that case 9 does not enable GR. I am more comfortable recommending case 4, because I just feel better with GR enabled.

If it’s BGP on loopback: Look at case 5 or case 6. Again, the only difference is GR not being enabled in one case (6), and I feel better recommending case 5, because GR is enabled.

Wrapping up

This was effectively my first commenter request, which feels good, but this post took way longer than it should have due to my testing process being bad and me constantly coming up with new things to test. A lot of surprises were discovered along the way, and now I’ve got it behind me, and I hope you, dear reader, come away with more, just as I did.

If you are curious, I used Drawbridge to create the diagrams for this post and will do so moving forward. Drawbridge is a project by Per Rosenlind, a good acquaintance. Check it out, and if you like it, support him.

Comments

Be civil. Stay on topic. Don’t lie.

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts