Ha config not in sync

[Pras]

Hi Guys.

I have a Palo 220 in HA A/P managed by the panorama.

The customer made mgmt IP change and Added a Zone but then ever since the config is out of Sync Between the HA pairs.

So all the articles are referenced, request high-availability sync-to-remote running-config' has been performed from both passive and active fw, force committed, pushed the template values from Panorama with all the force values and others selected, nothing works.

Pano is on 9.1.16 and the Firewalls are on 9.1.14-h4.

the only option left is to manual sync from the xml file which the customer is hesitant to do.

ha-agent logs gives below error from the passive Firewall

(Peer namespace on peer device missing too long, trying to restart)

LV[3]: type 11 (SYSD_PEER_DOWN); len 4; value:
00000001

Msg Hdr
-------
version : 1
groupID : 1
type : Hello (2)
token : 0x1b4e
flags : 0x1 (req:)
length : 122

Hello Msg
---------
flags : 0x1 (preempt:)
state : Active (5)
priority : 100
cookie : 17043
num tlvs : 3
Printing out 3 tlvs
TLV[1]: type 62 (CONFIG_MD5_PRE); len 33; value:
62656362 63383863 64663634 36636336 39373337 32356162
39373436 64333362 00
TLV[2]: type 2 (CONFIG_MD5SUM); len 33; value:
35653537 63313638 36646165 66623137 39323163 38306263
31663966 33333466 00
TLV[3]: type 11 (SYSD_PEER_DOWN); len 4; value:
00000001

2023-10-13 13:11:25.309 +1100 Error: ha_peer_hello_callback(src/ha_peer.c:5076): Group 1 (HA1-MAIN): Peer namespace on peer device missing too long, trying to restart
2023-10-13 13:11:25.309 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3353): Attempting 1 modify for sw.sysd.peers
2023-10-13 13:11:25.309 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3394): Setting up to modify sw.sysd.peers to peer. -> peerip:10…..xxx; sourceip:10.117.21.XXX; port:0x6e64
2023-10-13 13:11:25.309 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3418): Setting sysd node to: { 'peer.': { 'peerip': 10…..'port': 28260, 'reset': True, 'sourceip': 10.xxxXXX, }, }
2023-10-13 13:11:25.309 +1100 debug: sysd_queue_wr_event_add(sysd_queue.c:915): QUEUE: queue write event already added
2023-10-13 13:11:25.329 +1100 debug: ha_sysd_peerip_modify_callback(src/ha_sysd.c:3322): Successfully modified sw.sysd.peers
2023-10-13 13:12:45.388 +1100 Error: ha_peer_hello_callback(src/ha_peer.c:5076): Group 1 (HA1-MAIN): Peer namespace on peer device missing too long, trying to restart
2023-10-13 13:12:45.388 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3353): Attempting 1 modify for sw.sysd.peers
2023-10-13 13:12:45.389 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3394): Setting up to modify sw.sysd.peers to peer. -> peerip:10.xxxxxx; sourceip:10…xxx; port:0x6e64
2023-10-13 13:12:45.389 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3418): Setting sysd node to: { 'peer.': { 'peerip': 10….. Xxx, 'port': 28260, 'reset': True, 'sourceip': 10…XXX, }, }
2023-10-13 13:12:45.389 +1100 debug: sysd_queue_wr_event_add(sysd_queue.c:915): QUEUE: queue write event already added
2023-10-13 13:12:45.408 +1100 debug: ha_sysd_peerip_modify_callback(src/ha_sysd.c:3322): Successfully modified sw.sysd.peers

2023-10-13 13:14:05.466 +1100 Error: ha_peer_hello_callback(src/ha_peer.c:5076): Group 1 (HA1-MAIN): Peer namespace on peer device missing too long, trying to restart
2023-10-13 13:14:05.466 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3353): Attempting 1 modify for sw.sysd.peers
2023-10-13 13:14:05.467 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3394): Setting up to modify sw.sysd.peers to peer. -> peerip:10.xx; sourceip:10..1.XXX; port:0x6e64
2023-10-13 13:14:05.467 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3418): Setting sysd node to: { 'peer.': { 'peerip': 10.117… 'port': 28260, 'reset': True, 'sourceip': 10.117…XXX, }, }
2023-10-13 13:14:05.467 +1100 debug: sysd_queue_wr_event_add(sysd_queue.c:915): QUEUE: queue write event already added
2023-10-13 13:14:05.486 +1100 debug: ha_sysd_peerip_modify_callback(src/ha_sysd.c:3322): Successfully modified sw.sysd.peers
^Z2023-10-13 13:15:25.568 +1100 Error: ha_peer_hello_callback(src/ha_peer.c:5076): Group 1 (HA1-MAIN): Peer namespace on peer device missing too long, trying to restart
2023-10-13 13:15:25.568 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3353): Attempting 1 modify for sw.sysd.peers
2023-10-13 13:15:25.569 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3394): Setting up to modify sw.sysd.peers to peer. -> peerip:10.117…; sourceip:10.117….XXX; port:0x6e64
2023-10-13 13:15:25.569 +1100 debug: ha_sysd_peerip_modify(src/ha_sysd.c:3418): Setting sysd node to: { 'peer.': { 'peerip': 10.117.. 'port': 28260, 'reset': True, 'sourceip': 10.117….XX, }, }
2023-10-13 13:15:25.569 +1100 debug: sysd_queue_wr_event_add(sysd_queue.c:915): QUEUE: queue write event already added
2023-10-13 13:15:25.589 +1100 debug: ha_sysd_peerip_modify_callback(src/ha_sysd.c:3322): Successfully modified sw.sysd.peers
^PA-220-02(passive)>
PA-220-02(passive)>
PA-220-02(passive)> debug software resstart process management-server

Many Thanks,

@kiwi 

@BPry 

[Raido_Rattameister]

Firewalls are fully managed from Panorama so zone was added into Panorama template and pushed to firewall?

Management IP change was done inside active firewall right? Not in the Panorama. Mgmt IP needs to be different on both firewalls (management interface IP is not syncronized with HA sync).

[Pras]

@Raido_Rattameister 

Yes thats correct. 
ips are different in both Fws and Yes, zone pushed from Panorama.   Also this was working fine before. 
Forgot to mention, with all this happening with HA ,the Panorama actually says its in sync and theres no issue there.

Note: No zombie processes are running on the firewalls but the sysd msg and "Peer namespace on peer device missing too long, trying to restart" msg seem to be the clue for the issue.

Thanks

[MayurLaddha4545]

We had similar issue.

The fix is to reboot both firewalls in the HA pair as SYSD_PEER_DOWN.

Reboot will fix this issue right away.

Tried restarting manual sync, mgmt server reboot before reboot of PAN and no luck. 

Hope this helps

Cheers,

Mayur

[Pras · Accepted solution]

@MayurLaddha45454545: Thanks !!!!  I forgot to update here but that's exactly what was done to resolve the issue, manually sync'd the config and then restarted the Firewalls, as you said.