BGP GR

网络需求

FortiGate HA集群与上下游路由器运行BGP协议,并开启GR。在FortiGate的HA发生主备切换时,GR需要确保每个路由器(包括FortiGate及其它路由器)在HA切换期间保持其路由表中的BGP路由(GR Helper),以避免流量中断。

在FortiGate HA集群中,BGP进程相关工作仅在主设备上运行。当发生HA切换时,将在新选举的主机上启动一个新的BGP协商过程,在此过程中,新的主机需要在路由Kernel表中维持从原主机学习到的BGP路由,直至新的BGP邻居形成。

对于BGP的GR,有以下几个计时器需要了解:

  • holdtime-timer(默认180s):将BGP邻居标记为死亡的秒数。即在考虑与对等方的连接关闭之前,在保持活动、更新或通知消息之间等待的秒数。
  • graceful-restart-time(默认120s):邻居重新启动所需的时间,即GR-Helper在删除过时路由之前等待OPEN消息的秒数。请确保graceful-restart-time 小于或等于 holdtime-timer
  • graceful-update-delay(默认120s):GR完成后,路由宣告/学习的延迟时间。在HA故障切换后,由于此设置的计时器,新主机上的路由会有一定的延迟。
  • graceful-stalepath-time(默认360s):在GR-Helper上保持GR邻居的过时路由的时间,即在删除之前保持过时路由的最大总时间。

网络拓扑

image-20240115105813078

  1. FW1与FW2建立A-P模式的HA。
  2. FW1/FW2与FW3、FW4分别建立IBGP邻居,FW3、FW4宣告各自的网段。
  3. FW1/FW2与FW3、FW4开启BGP Graceful Restart,FW1/FW2发生主备切换后,FW3、FW4可以进入GR Helper模式,协助FW1/FW2的GR过程。

配置步骤

  1. 基础网络配置(略)。

  2. FW1与FW2的HA配置(略)。

  3. 安全策略配置(略)。

  4. 配置FW1/FW2的BGP,开启GR功能,restart-period设置为600s,计时器配置均保持默认,邻居配置中开启路由反射器(RR),将两个IBGP邻居的路由互相反射,反射时将下一跳改为自己。

    config router bgp
        set as 65001
        set router-id 202.103.12.1
        set graceful-restart enable
        set graceful-restart-time 120
        set graceful-stalepath-time 360
        set graceful-update-delay 120
        config neighbor
            edit "202.103.12.2"
                set capability-graceful-restart enable
                set next-hop-self-rr enable
                set soft-reconfiguration enable
                set remote-as 65001
                set route-reflector-client enable
            next
            edit "202.103.13.2"
                set capability-graceful-restart enable
                set next-hop-self-rr enable
                set soft-reconfiguration enable
                set remote-as 65001
                set route-reflector-client enable
            next
        end
    end
    
  5. 配置FW3的BGP,宣告内网网络,路由器作为GR-Helper也需要开启GR功能,计时器配置均保持默认。

    config router bgp
        set as 65001
        set router-id 202.103.12.2
        set graceful-restart enable
        config neighbor
            edit "202.103.12.1"
                set capability-graceful-restart enable
                set soft-reconfiguration enable
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 10.10.1.0 255.255.255.0
            next
        end
    end
    
  6. 配置FW4的BGP,宣告内网网络,路由器作为GR-Helper也需要开启GR功能,计时器配置均保持默认。

    config router bgp
        set as 65001
        set router-id 202.103.13.2
        set graceful-restart enable
        config neighbor
            edit "202.103.13.1"
                set capability-graceful-restart enable
                set soft-reconfiguration enable
                set remote-as 65001
            next
        end
        config network
            edit 1
                set prefix 10.10.2.0 255.255.255.0
            next
        end
    end
    
  7. 配置FW1/FW2 HA配置中的route-ttl为360s(默认10s),在此时间内,新的主机需要在路由Kernel表中维持从原主机学习到的BGP路由,直至新的BGP邻居形成,并完成BGP路由学习。

    重要步骤,防止HA切换后,新主机建立新的BGP邻居前,路由消失导致业务中断。

    config system ha
        set route-ttl 360
    end
    

结果验证

  1. 初始状态下,FW1为主设备,FW2为备设备。

    FW1 # diagnose sys ha status
    HA information
    Statistics
            traffic.local = s:0 p:42869 b:39035389
            traffic.total = s:0 p:42867 b:39033889
            activity.ha_id_changes = 2
            activity.fdb  = c:0 q:0
    
    Model=80008, Mode=2 Group=7 Debug=0
    nvcluster=1, ses_pickup=0, delay=0
    
    [Debug_Zone HA information]
    HA group member information: is_manage_primary=1.
    FGVM08TM23000175:      Primary, serialno_prio=1, usr_priority=128, hostname=FW1
    FGVM08TM23000176:    Secondary, serialno_prio=0, usr_priority=100, hostname=FW2
    
    [Kernel HA information]
    vcluster 1, state=work, primary_ip=169.254.0.2, primary_id=0
    FGVM08TM23000175:      Primary, ha_prio/o_ha_prio=0/0
    FGVM08TM23000176:    Secondary, ha_prio/o_ha_prio=1/1
    
  2. 初始状态下查看FW3的BGP邻居状态与路由信息,邻居状态为已建立状态,BGP路由学习正常。

    FW3 # get router info bgp summary 
    
    VRF 0 BGP router identifier 202.103.12.2, local AS number 65001
    BGP table version is 3
    1 BGP AS-PATH entries
    0 BGP community entries
    
    Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    202.103.12.1 4      65001    1229    1229        2    0    0 16:55:54        1
    
    Total number of neighbors 1
    
    FW3 # get router info routing-table bgp
    Routing table for VRF=0
    B       10.10.2.0/24 [200/0] via 202.103.12.1 (recursive is directly connected, port2), 00:02:15, [1/0]
    
  3. 初始状态下查看FW4的BGP邻居状态与路由信息,邻居状态为已建立状态,BGP路由学习正常。

    FW4 # get router info bgp summary 
    
    VRF 0 BGP router identifier 202.103.13.2, local AS number 65001
    BGP table version is 3
    1 BGP AS-PATH entries
    0 BGP community entries
    
    Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    202.103.13.1 4      65001    1232    1233        2    0    0 17:01:22        1
    
    Total number of neighbors 1
    
    FW4 # get router info routing-table bgp
    Routing table for VRF=0
    B       10.10.1.0/24 [200/0] via 202.103.13.1 (recursive is directly connected, port2), 00:03:00, [1/0]
    
  4. 初始状态下查看FW1的BGP邻居状态与路由信息,邻居状态为已建立,BGP路由学习正常。

    FW1 # get router info bgp summary 
    
    VRF 0 BGP router identifier 202.103.12.1, local AS number 65001
    BGP table version is 1
    1 BGP AS-PATH entries
    0 BGP community entries
    
    Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    202.103.12.2 4      65001    1232    1241        1    0    0 17:03:12        1
    202.103.13.2 4      65001    1233    1238        1    0    0 17:03:11        1
    
    Total number of neighbors 2
    
    FW1 # get router info routing-table bgp
    Routing table for VRF=0
    B       10.10.1.0/24 [200/0] via 202.103.12.2 (recursive is directly connected, port2), 17:02:56, [1/0]
    B       10.10.2.0/24 [200/0] via 202.103.13.2 (recursive is directly connected, port3), 17:02:55, [1/0]
    
  5. 使FW1和FW2发生HA主备倒换,此时FW2成为HA主设备。

    FW2 # diagnose sys ha status
    HA information
    Statistics
            traffic.local = s:0 p:6233 b:4732229
            traffic.total = s:0 p:6286 b:4735596
            activity.ha_id_changes = 4
            activity.fdb  = c:0 q:0
    
    Model=80008, Mode=2 Group=7 Debug=0
    nvcluster=1, ses_pickup=0, delay=0
    
    [Debug_Zone HA information]
    HA group member information: is_manage_primary=1.
    FGVM08TM23000176:      Primary, serialno_prio=0, usr_priority=100, hostname=FW2
    FGVM08TM23000175:    Secondary, serialno_prio=1, usr_priority=128, hostname=FW1
    
    [Kernel HA information]
    vcluster 1, state=work, primary_ip=169.254.0.1, primary_id=0
    FGVM08TM23000176:      Primary, ha_prio/o_ha_prio=0/0
    FGVM08TM23000175:    Secondary, ha_prio/o_ha_prio=1/1
    
  6. 查看FW2的BGP Debug信息,FW2成为HA主后,向BGP邻居发送BGP OPEN消息,其中包含Cap GR,Restarting时间为120s。

    FW2 # diagnose ip router bgp level info
    FW2 # diagnose ip router bgp all enable
    FW2 # diagnose debug console time enable
    ......
    2024-01-16 14:53:33 BGP: 202.103.12.2-Outgoing [ENCODE] Open: Ver 4 MyAS 65001 Holdtime 180
    ......
    2024-01-16 14:53:33 BGP: 202.103.12.2-Outgoing [DECODE] Open Opt: Option Type 2, Option Len 8
    2024-01-16 14:53:33 BGP: 202.103.12.2-Outgoing [DECODE] Open Cap: Cap Code 64, Cap Len 6
    2024-01-16 14:53:33 BGP: 202.103.12.2-Outgoing [DECODE] Cap GR: Restart Flag Off, Restart Time 120
    diagnose ip2024-01-16 14:53:33 BGP: 202.103.12.2-Outgoing [DECODE] Cap GR: AFI/SAFI 1/1 Fwd-state Flag 1, action: Set
    ......
    2024-01-16 14:53:34 2024-01-16 14:53:34 BGP: 202.103.12.2-Outgoing [GRST] GRST Init Announce:GRST Defer Announce Timer(120) Started
    id=20300 msg="BGP: %BGP-5-ADJCHANGE: VRF 0 neighbor 202.103.12.2 Up "
    ......
    2024-01-16 14:53:59 BGP: 202.103.12.2-Outgoing [FSM] Update: IPv4 Unicast End-Of-Rib Marker Received
    2024-01-16 14:53:59 BGP: 202.103.12.2-Outgoing [FSM] Process End-of-RIB: Received for afi/safi: 1/1
    
  7. 该报文格式如下图所示。

    image-20240116150338749

  8. 在FW2进入GR状态后,查看FW3的BGP邻居状态与BGP路由状态,FW3进入GR Helper状态,开始与FW2建立新的BGP邻居,之前已学习到的BGP路由未消失,被打上S(Stale)标记。(FW4也是一样的状态,这里就不再赘述)

    FW3 # get router info bgp summary
    
    VRF 0 BGP router identifier 202.103.12.2, local AS number 65001
    BGP table version is 10
    1 BGP AS-PATH entries
    0 BGP community entries
    
    Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    202.103.12.1 4      65001     101     104        0    0    0    never OpenConfirm
    
    Total number of neighbors 1
    
    FW3 # get router info bgp network
    VRF 0 BGP table version is 7, local router ID is 202.103.12.2
    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                  S Stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
    
       Network          Next Hop            Metric     LocPrf Weight RouteTag Path
    *> 10.10.1.0/24     0.0.0.0                       100  32768        0 i <-/1>
    S>i10.10.2.0/24     202.103.12.1    0             100      0        0 i <-/1>
    
    Total number of prefixes 2
    
    FW3 # get router info routing-table bgp
    Routing table for VRF=0
    B       10.10.2.0/24 [200/0] via 202.103.12.1 (recursive is directly connected, port2), 00:01:06, [1/0]
    
  9. 在FW3上查看BGP的Debug信息,FW3与FW1的BGP连接断开后,进入GR Helper模式,GR倒计时120s。(FW4也是一样的状态,这里就不再赘述)

    FW3 # diagnose ip router bgp level info
    FW3 # diagnose ip router bgp all enable
    FW3 # diagnose debug console time enable
    
    2024-01-16 15:13:53 BGP: [NETWORK] Accept Thread: Incoming conn from host 202.103.12.1 (FD=26 VRF=0)
    2024-01-16 15:13:53 BGP: 202.103.12.1-Outgoing [FSM] State: Established Event: 14
    ......
    2024-01-16 15:14:22 BGP: 202.103.12.1-Outgoing [RIB] Update: Received Prefix 10.10.2.0/24 path_id 0
    ......
    
  10. 等待FW2与FW3的BGP邻居建立完成后,FW2的GR完成,FW3退出GR Helper状态,期间被打上S标记的旧BGP路由没有消失,重新和FW2建立BGP邻居并学习BGP路由后,S标记变为*标记。(FW4也是一样的状态,这里就不再赘述)

    FW3 # get router info bgp summary
    
    VRF 0 BGP router identifier 202.103.12.2, local AS number 65001
    BGP table version is 7
    1 BGP AS-PATH entries
    0 BGP community entries
    
    Neighbor     V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
    202.103.12.1 4      65001      79      81        0    0    0 00:00:05        1
    
    FW3 # get router info bgp network 
    VRF 0 BGP table version is 12, local router ID is 202.103.12.2
    Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
                 S Stale
    Origin codes: i - IGP, e - EGP, ? - incomplete
    
      Network          Next Hop            Metric     LocPrf Weight RouteTag Path
    *> 10.10.1.0/24     0.0.0.0                       100  32768        0 i <-/1>
    *>i10.10.2.0/24     202.103.12.1    0             100      0        0 i <-/1>
    
    Total number of prefixes 2
    
    FW3 # get router info routing-table bgp
    Routing table for VRF=0
    B       10.10.2.0/24 [200/0] via 202.103.12.1 (recursive is directly connected, port2), 00:01:06, [1/0]
    
  11. 在FW3上查看BGP的Debug信息,FW3从FW2完成了新的BGP路由学习后,向FW2发送了End-of-RIB报文,报文格式如下(空的BGP Update报文),通知其路由已经学习完成。

    2024-01-16 15:14:22 BGP: VRF 0 NSM announce: 10.10.2.0/24
    2024-01-16 15:14:22 BGP: 202.103.12.1-Outgoing [FSM] Update: IPv4 Unicast End-Of-Rib Marker Received
    2024-01-16 15:14:22 BGP: 202.103.12.1-Outgoing [FSM] Process End-of-RIB: Received for afi/safi: 1/1
    

    image-20240116153557391

  12. 在FW3上查看BGP邻居状态,其中包含GR相关信息,如邻居的GR时间为120s。

    FW3 # get router info bgp neighbors 
    VRF 0 neighbor table:
    BGP neighbor is 202.103.12.1, remote AS 65001, local AS 65001, internal link
      BGP version 4, remote router ID 202.103.12.1
      BGP state = Established, up for 00:04:58
      Last read 00:00:02, hold time is 180, keepalive interval is 60 seconds
      Configured hold time is 180, keepalive interval is 60 seconds
     ......
     For address family: IPv4 Unicast
      BGP table version 13, neighbor version 12
      Index 1, Offset 0, Mask 0x2
      AF-dependant capabilities:
        Graceful restart: advertised, received, negotiated
    ......
     Connections established 13; dropped 12
     Graceful-restart Status:
      Remote restart-time is 120 sec
    ......
    
  13. GR期间,FW2从FW1重启前同步的路由一直存在(route-ttl 360,prio=2164260865)。

    FW2 # get router info kernel | grep 10.10.
    tab=254 vf=0 scope=0 type=1 proto=30 prio=2164260865 0.0.0.0/0.0.0.0/0->10.10.1.0/24 pref=0.0.0.0 gwy=202.103.12.2 dev=4(port2)
    tab=254 vf=0 scope=0 type=1 proto=30 prio=2164260865 0.0.0.0/0.0.0.0/0->10.10.2.0/24 pref=0.0.0.0 gwy=202.103.13.2 dev=5(port3)
    
  14. GR结束后,学到了新的BGP路由(prio=1)。

    FW2 # get router info kernel | grep 10.10.
    tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->10.10.1.0/24 pref=0.0.0.0 gwy=202.103.12.2 dev=4(port2)
    tab=254 vf=0 scope=0 type=1 proto=30 prio=2164260865 0.0.0.0/0.0.0.0/0->10.10.1.0/24 pref=0.0.0.0 gwy=202.103.12.2 dev=4(port2)
    tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->10.10.2.0/24 pref=0.0.0.0 gwy=202.103.13.2 dev=5(port3)
    tab=254 vf=0 scope=0 type=1 proto=30 prio=2164260865 0.0.0.0/0.0.0.0/0->10.10.2.0/24 pref=0.0.0.0 gwy=202.103.13.2 dev=5(port3)
    
  15. 整个HA切换引起的GR期间,所有设备路由转发表未发生实际变化,流量不会中断。

Copyright © 2024 Fortinet Inc. All rights reserved. Powered by Fortinet TAC Team.
📲扫描下方二维码分享此页面👇
该页面修订于: 2024-01-17 16:44:45

results matching ""

    No results matching ""