PAN 0S 7.1.1 Mode active/passive Still Display Not Synchronized

[plebras]

Hello

After Upgrading To 7.1.1 The 2 devices PA-500 mode Active/Passive. The display keep Not Synchronized.

The command -show job id x- for HA sync display OK (no warnings).

The command -show high-availability state-synchronization- show all synchronized except A/A session stup A/A session stats and A/Packet (I am in A/P so it is normal).

Is It a bug ? Any Ideas ?

Thank you

------------------------------------

Mode Active-passive
Local Active
Peer (10.253.254.201) Passive
Running Config Not synchronized Sync to peer
App Version Match
Threat Version Match
Antivirus Version Match
PAN-OS Version Match
GlobalProtect Version Match
HA1 Up
HA2 Up

------------------------------------------------

Enqueued Dequeued ID Type
Status Result Completed
--------------------------------------------------------------------------------
----------------------------------------------
2016/05/06 08:50:10 08:50:10 91 HA-Sync
FIN OK 08:52:42
Warnings:
Details:Configuration committed successfully

--------------------------------------------------------------------------------
State Synchronization Status: Complete
--------------------------------------------------------------------------------
state synchronization to peer device enabled: no (device not in active state)
--------------------------------------------------------------------------------
state synchronization messages processed since system up

message enable version sent received

--------------------------------------------------------------------------------
session setup yes 8 0 2868121

session teardown yes 8 0 2867822

session update yes 8 0 17753292

predict session add yes 8 0 2363

predict session delete yes 8 0 2344

predict session update yes 8 0 19

ARP update yes 1 0 161604

ARP delete yes 1 0 0

MAC update yes 1 0 0

MAC delete yes 1 0 0

IPSec sequence number update yes 3 0 47245

ND update yes 1 0 0

ND delete yes 1 0 0

DoS Aggregate entry update yes 1 0 0

DoS Class Tbl IP update yes 1 0 0

DoS Class Tbl IP delete yes 1 0 0

DoS Block Tbl IP update yes 1 0 0

DoS Block Tbl IP delete yes 1 0 0

A/A session setup no 8 0 0

A/A session statistics no 8 0 0

A/A packet forward using HA2 no 8 0 0

Return MAC Update yes 1 0 0

Return MAC Delete yes 1 0 0

V6 Return MAC Update yes 1 0 0

V6 Return MAC Delete yes 1 0 0

HA2 monitor message yes 1 0 0

predict session modify yes 8 0 0

--------------------------------------------------------------------------------

[BenjAudy.MTL]

Hi,

The state synchronization refers to things like sessions, which is different from the config synchronization. Have you tried to do a config audit between the local running config and the peer's running config? Maybe something will stand out (apart from the usual private-key and other unique config items). If you see something missing, fail over the other firewall and apply the missing config, it might help with the synchronization. That's what I did when it happened to me, but in my case it was v6.

Regards,

Benjamin

[JDominguez]

An Audit will do well!

Also check the Passive device's "Tasks" to see if the commit is failing.

You can also run the following command to follow the ha-agent log to get a bit more info:

>tail follow yes mp-log ha_agent.log

[plebras]

Thanks you.

Config audit does not display any differences except key, IP, name.

I have tried to restart management service on peer. Same. I have tried to commit  before on the peer then  resync. Same.

On the peer passive , the  sync commit task from the peer active display successful (configuration commited successfully). It seems the config is duplicated (see logs) but some erros are reported :

The command 

>tail follow yes mp-log ha_agent.log

display on the active PA:

tail follow yes mp-log ha_agent.log
00000000

2016-05-09 10:40:29.082 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:40:29.082 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:40:29.082 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:279): Group 1: cancel config sync timer
2016-05-09 10:40:29.082 +0200 debug: ha_sysd_dev_cfgsync_update(src/ha_sysd.c:1383): Set dev cfgsync to Out-of-Sync
2016-05-09 10:40:29.089 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:40:32.999 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:40:32.999 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:40:32.999 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:47:18.493 +0200 debug: ha_sysd_mgmt_dosync_notifier_callback(src/ha_sysd.c:2449): Received external triggered dosync
2016-05-09 10:47:18.493 +0200 debug: ha_sysd_dev_cfgsync_update(src/ha_sysd.c:1383): Set dev cfgsync to Committing
2016-05-09 10:49:30.639 +0200 debug: ha_peer_recv_hello(src/ha_peer.c:4998): Group 1 (HA1-MAIN): Receiving hello message

Msg Hdr
-------
version : 1
groupID : 1
type : Hello (2)
token : 0xc89e
flags : 0x1 (req:)
length : 122

Hello Msg
---------
flags : 0x0 ()
state : Passive (4)
priority : 101
cookie : 10787
num tlvs : 3
Printing out 3 tlvs
TLV[1]: type 62 (CONFIG_MD5_PRE); len 33; value:
32383631 66396666 62626564 35353563 37363362 62643262
34333532 30646133 00
TLV[2]: type 2 (CONFIG_MD5SUM); len 33; value:
61323137 33393930 35316331 65306164 61316566 36306631
63663337 34336634 00
TLV[3]: type 11 (SYSD_PEER_DOWN); len 4; value:
00000000

2016-05-09 10:49:48.533 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:49:48.533 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:49:48.533 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:279): Group 1: cancel config sync timer
2016-05-09 10:49:48.533 +0200 debug: ha_sysd_dev_cfgsync_update(src/ha_sysd.c:1383): Set dev cfgsync to Out-of-Sync
2016-05-09 10:49:48.533 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:49:52.701 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:49:52.701 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:49:52.701 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request

 --------------------------------------------

On the passive PA

tail follow yes mp-log ha_agent.log

2016-05-09 10:40:11.492 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync failure
2016-05-09 10:40:26.940 +0200 debug: ha_sysd_config_status_notifier_callback(src/ha_sysd.c:2793): Ending monitor increase holdup on commit end
2016-05-09 10:40:26.940 +0200 debug: ha_state_stop_increase_monitor_holdup(src/ha_state.c:1220): Ending monitor holdup increase after commit in 60 seconds
2016-05-09 10:40:29.065 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:40:29.065 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:40:29.065 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:40:32.984 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:40:32.984 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:40:32.984 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:48:05.908 +0200 debug: cfgagent_flags_callback(pan_cfgagent.c:225): ha_agent: cfg agent received flags from server
2016-05-09 10:48:05.909 +0200 debug: cfgagent_flags_callback(pan_cfgagent.c:229): new flags=0x4
2016-05-09 10:48:05.914 +0200 debug: cfgagent_config_callback(pan_cfgagent.c:252): ha_agent: cfg agent received configuration from server
2016-05-09 10:48:05.915 +0200 debug: cfgagent_config_callback(pan_cfgagent.c:274): config length=81656
2016-05-09 10:48:05.917 +0200 debug: ha_cfgagent_phase1(src/ha_cfgagent.c:557): start
2016-05-09 10:48:05.918 +0200 debug: ha_cfgagent_phase1_callback(src/ha_cfgagent.c:496): start
2016-05-09 10:48:05.970 +0200 debug: ha_cfgagent_phase1_callback(src/ha_cfgagent.c:528): sending back true for p1done
2016-05-09 10:49:14.038 +0200 debug: ha_sysd_config_status_notifier_callback(src/ha_sysd.c:2801): Starting monitor increase holdup on phase2 start
2016-05-09 10:49:14.038 +0200 debug: ha_state_start_increase_monitor_holdup(src/ha_state.c:1198): Starting monitor holdup increase during commit
2016-05-09 10:49:14.085 +0200 debug: cfgagent_flags_callback(pan_cfgagent.c:225): ha_agent: cfg agent received flags from server
2016-05-09 10:49:14.085 +0200 debug: cfgagent_flags_callback(pan_cfgagent.c:229): new flags=0x0
2016-05-09 10:49:14.088 +0200 debug: ha_cfgagent_phase2(src/ha_cfgagent.c:749): start
2016-05-09 10:49:14.088 +0200 debug: ha_cfgagent_phase2_callback(src/ha_cfgagent.c:697): start
2016-05-09 10:49:14.093 +0200 debug: ha_cfgagent_phase2_callback(src/ha_cfgagent.c:726): sending back true for p2done
2016-05-09 10:49:15.254 +0200 Received HA2 MAC address: d4:f4:be:12:e2:16
2016-05-09 10:49:15.255 +0200 Received HA2 MAC address: d4:f4:be:12:e2:16
2016-05-09 10:49:30.620 +0200 debug: ha_state_cfg_md5_set(src/ha_state_cfg.c:458): We were out of sync and now we are out of sync; autocommit no; ha-sync yes; panorama no; cfg-sync-off no; pre-old-insync no; pre-new-insync no
2016-05-09 10:49:30.620 +0200 debug: ha_sysd_mgmt_dosync_trigger(src/ha_sysd.c:808): Sending start sync to mgmtsrvr: False
2016-05-09 10:49:30.621 +0200 debug: ha_peer_send_hello(src/ha_peer.c:4945): Group 1 (HA1-MAIN): Sending hello message

Hello Msg
---------
flags : 0x0 ()
state : Passive (4)
priority : 101
cookie : 10787
num tlvs : 3
Printing out 3 tlvs
TLV[1]: type 62 (CONFIG_MD5_PRE); len 33; value:
32383631 66396666 62626564 35353563 37363362 62643262
34333532 30646133 00
TLV[2]: type 2 (CONFIG_MD5SUM); len 33; value:
61323137 33393930 35316331 65306164 61316566 36306631
63663337 34336634 00
TLV[3]: type 11 (SYSD_PEER_DOWN); len 4; value:
00000000

2016-05-09 10:49:30.626 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync failure
2016-05-09 10:49:45.646 +0200 debug: ha_sysd_config_status_notifier_callback(src/ha_sysd.c:2793): Ending monitor increase holdup on commit end
2016-05-09 10:49:45.646 +0200 debug: ha_state_stop_increase_monitor_holdup(src/ha_state.c:1220): Ending monitor holdup increase after commit in 60 seconds
2016-05-09 10:49:48.511 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:49:48.511 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:49:48.511 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request
2016-05-09 10:49:52.681 +0200 debug: ha_sysd_mgmt_finsync_notifier_callback(src/ha_sysd.c:2482): Mgmtsrvr sent finsync success
2016-05-09 10:49:52.681 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:253): group 1: mgmtsrvr insync: NO; insync2: NO
2016-05-09 10:49:52.681 +0200 debug: ha_state_cfg_check_insync(src/ha_state_cfg.c:287): failure for config sync request

Thank you for looking

[rmonvon]

Hi...Can you switch to private/incognito mode on your browser to see if it's a caching issue please.  If the CLI output is correct and the GUI is not, it may be the browser.  

[plebras]

Thank you but the some cli command seems correct but the logs shows errors (see post) and I do not know what is the meeaning of these errors. When syncing there is a display "synchronization in progress" so it is not a cache problem (we can refresh the display in the IHM). I've tried with private chrome browser, but it is the same.

We was in 7.05 release and all was correct. the problem came after the upgrade in 7.1.1.  (With this release , we also have another problem: the Cisco Vpn Client on windows does not work any more -I know, we can use the GP client...- , but? not completely tested release ?). 

[rmonvon]

Since both PAs are running with the same config, you may try sync'ing from passive to active device.  Also, your log does not show the same error as mentioned in this link, but restarting the management server may resolve the issue: 

https://live.paloaltonetworks.com/t5/Management-Articles/Cannot-Sync-from-Active-to-Passive-Member-i...

[plebras]

I haved already try to restart management agent on passive peer and to reboot it. No result. The sync from to the passive peer to the active gives the same result (no sync).  I will try to reboot the active peer when it will be possible.

[plebras · Accepted solution]

Upgrade to 7.1.2 resolve the problem. Bug 7.1.1

[Ian_Lobdell]

The IPSec issue (cisco vpn client not connecting) was also fixed in 7.1.2