11-26-2013 05:06 AM
Hi All,
I've a HA Active/Active pair of PA-3020's managed via Panorama.
All was well until I had to replace one of the pair, I brought the new device into Panorama, added it to the correct group/templates and then finished the device config (IP addresses etc).
The last step was to configure it as part of the HA.
However the two systems are failing to sync policy.
So far all I can see is the entries in the system log widget:
HA Group 1: Running configuration not synchronized after retries 11/26 12:25:23
running configuration synchronized with HA peer by admin 11/26 12:10:23
Where do I go next to find out why they won't sync?
Phil.
11-26-2013 07:17 AM
Hi Vince,
Yes that's active - the push from Panorama works fine - it's the HA peer sync of the running config that's failing.
Phil
11-26-2013 07:25 AM
Hello Pcook,
Here is the command to try form the CLI,
> request high-availability sync-to-remote running-config
Also in the ms.log we will see details what is happening with error details and logs indicating the fail for the HA-sync
Run the below command first " tail follow yes mp-log ms.log" and then try to do HA Sync to see the logs dynamically. This would give some explanation.
May also have to look at any other pending job is running on the peer device
show jobs pending
Hope this would give a path.
Thanks
11-26-2013 08:09 AM
Thanks Phoenix, that helped in that it's a useful command but I'm still not sure where the error is. There are some error lines at the start but the sync seems to run after that.
Do you think I should raise a support ticket?
Phil.
tail follow yes mp-log ms.log wrote:
Nov 26 15:32:49 Getting authorization info for user cookp succeeded.
Nov 26 15:32:49 cms-client: Found role superuser for user cookp
Nov 26 15:32:49 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:32:51 Error: pan_xml_get_node_by_xpath(pan_xml_helper.c:570): cannot evalute xpath expression:/config/devices/entry[@name='localhost.localdomain']/deviceconfig/system
Nov 26 15:32:52 Error: pan_xml_get_node_by_xpath(pan_xml_helper.c:570): cannot evalute xpath expression:/config/devices/entry[@name='localhost.localdomain']/deviceconfig/system
Nov 26 15:32:58 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:2025): failed to fetch: NO_MATCHES
Nov 26 15:32:58 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:2047): failed to fetch: NO_MATCHES
Nov 26 15:33:00 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:2025): failed to fetch: NO_MATCHES
Nov 26 15:33:00 report generation started for 'acc-summary'
Nov 26 15:33:00 generating report for time from 1385476380 to 1385479979
Nov 26 15:33:00 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:33:00 report generation: vectorization started for 'acc-summary'
Nov 26 15:33:00 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:33:00 report generation: sorting started for 'acc-summary'
Nov 26 15:33:00 report generation: sorting ended for 'acc-summary'
Nov 26 15:33:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:33:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:33:11 report generation started for 'acc-summary'
Nov 26 15:33:11 generating report for time from 1385476391 to 1385479990
Nov 26 15:33:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:33:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:33:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:33:11 report generation: sorting started for 'acc-summary'
Nov 26 15:33:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:33:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:33:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:33:11 report generation started for 'acc-summary'
Nov 26 15:33:11 generating report for time from 1385476391 to 1385479990
Nov 26 15:33:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:33:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:33:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:33:11 report generation: sorting started for 'acc-summary'
Nov 26 15:33:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:33:12 Error: pan_logjobmgr_print_job(pan_log_mgr.c:2173): no results entry for log job id 1055
Nov 26 15:33:12 Error: pan_mgmtop_show_query(pan_ops_common.c:2105): Unable to find query job
Nov 26 15:33:20 client dagger reported op command was SUCCESSFUL
Nov 26 15:34:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:34:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:34:11 report generation started for 'acc-summary'
Nov 26 15:34:11 generating report for time from 1385476451 to 1385480050
Nov 26 15:34:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:34:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:34:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:34:11 report generation: sorting started for 'acc-summary'
Nov 26 15:34:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:35:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:35:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:35:11 report generation started for 'acc-summary'
Nov 26 15:35:11 generating report for time from 1385476511 to 1385480110
Nov 26 15:35:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:35:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:35:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:35:11 report generation: sorting started for 'acc-summary'
Nov 26 15:35:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:36:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:36:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:36:11 report generation started for 'acc-summary'
Nov 26 15:36:11 generating report for time from 1385476571 to 1385480170
Nov 26 15:36:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:36:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:36:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:36:11 report generation: sorting started for 'acc-summary'
Nov 26 15:36:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:37:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:37:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:37:11 report generation started for 'acc-summary'
Nov 26 15:37:11 generating report for time from 1385476631 to 1385480230
Nov 26 15:37:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:37:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:37:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:37:11 report generation: sorting started for 'acc-summary'
Nov 26 15:37:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:38:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:38:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:38:11 report generation started for 'acc-summary'
Nov 26 15:38:11 generating report for time from 1385476691 to 1385480290
Nov 26 15:38:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:38:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:38:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:38:11 report generation: sorting started for 'acc-summary'
Nov 26 15:38:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:39:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:39:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:39:11 report generation started for 'acc-summary'
Nov 26 15:39:11 generating report for time from 1385476751 to 1385480350
Nov 26 15:39:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:39:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:39:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:39:11 report generation: sorting started for 'acc-summary'
Nov 26 15:39:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:40:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:40:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:40:11 report generation started for 'acc-summary'
Nov 26 15:40:11 generating report for time from 1385476811 to 1385480410
Nov 26 15:40:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:40:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:40:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:40:11 report generation: sorting started for 'acc-summary'
Nov 26 15:40:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:41:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:41:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:41:11 report generation started for 'acc-summary'
Nov 26 15:41:11 generating report for time from 1385476871 to 1385480470
Nov 26 15:41:11 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:41:11 report generation: vectorization started for 'acc-summary'
Nov 26 15:41:11 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:41:11 report generation: sorting started for 'acc-summary'
Nov 26 15:41:11 report generation: sorting ended for 'acc-summary'
Nov 26 15:42:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:42:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:42:12 report generation started for 'acc-summary'
Nov 26 15:42:12 generating report for time from 1385476932 to 1385480531
Nov 26 15:42:12 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:42:12 report generation: vectorization started for 'acc-summary'
Nov 26 15:42:12 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:42:12 report generation: sorting started for 'acc-summary'
Nov 26 15:42:12 report generation: sorting ended for 'acc-summary'
Nov 26 15:43:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:43:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:43:12 report generation started for 'acc-summary'
Nov 26 15:43:12 generating report for time from 1385476992 to 1385480591
Nov 26 15:43:12 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:43:12 report generation: vectorization started for 'acc-summary'
Nov 26 15:43:12 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:43:12 report generation: sorting started for 'acc-summary'
Nov 26 15:43:12 report generation: sorting ended for 'acc-summary'
Nov 26 15:44:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:44:11 client dagger reported op command was SUCCESSFUL
Nov 26 15:44:12 report generation started for 'acc-summary'
Nov 26 15:44:12 generating report for time from 1385477052 to 1385480651
Nov 26 15:44:12 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:44:12 report generation: vectorization started for 'acc-summary'
Nov 26 15:44:12 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:44:12 report generation: sorting started for 'acc-summary'
Nov 26 15:44:12 report generation: sorting ended for 'acc-summary'
Nov 26 15:44:46 client routed reported op command was SUCCESSFUL
Nov 26 15:45:42 client dagger reported op command was SUCCESSFUL
Nov 26 15:46:22 dnscfgmod: FQDN Refresh: Periodic TTL Expiry Refresh
Nov 26 15:46:22 dnscfgmod: Main refresh function: (TTL Expiry)
Nov 26 15:46:22 dnscfgmod: No fqdns used in this config. Skip config push to device
Nov 26 15:53:01 client dagger reported op command was SUCCESSFUL
Nov 26 15:53:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:53:02 report generation started for 'acc-summary'
Nov 26 15:53:02 generating report for time from 1385477582 to 1385481181
Nov 26 15:53:02 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:53:02 report generation: vectorization started for 'acc-summary'
Nov 26 15:53:02 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:53:02 report generation: sorting started for 'acc-summary'
Nov 26 15:53:02 report generation: sorting ended for 'acc-summary'
Nov 26 15:54:01 client dagger reported op command was SUCCESSFUL
Nov 26 15:54:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:54:01 report generation started for 'acc-summary'
Nov 26 15:54:01 generating report for time from 1385477641 to 1385481240
Nov 26 15:54:01 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:54:01 report generation: vectorization started for 'acc-summary'
Nov 26 15:54:01 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:54:01 report generation: sorting started for 'acc-summary'
Nov 26 15:54:01 report generation: sorting ended for 'acc-summary'
Nov 26 15:55:01 client dagger reported op command was SUCCESSFUL
Nov 26 15:55:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date
Nov 26 15:55:01 report generation started for 'acc-summary'
Nov 26 15:55:01 generating report for time from 1385477701 to 1385481300
Nov 26 15:55:01 report generation: query on type appstat started for 'acc-summary'
Nov 26 15:55:01 report generation: vectorization started for 'acc-summary'
Nov 26 15:55:01 report generation: pre-sort-hook started for 'acc-summary'
Nov 26 15:55:01 report generation: sorting started for 'acc-summary'
Nov 26 15:55:01 report generation: sorting ended for 'acc-summary'
11-26-2013 08:45 AM
Hello Pcook,
Did the CLI command help in getting the HA(High availability ) pair to be in Sync.
If not at this point a support case is the option. The reason is an analysis can be obtained by running the logs at debug level for ms.log and also find what is happening in the "ha-agent.log" ( This log has all the details about HA operation on Management Plane ), "pan_dha.log" ( HA logs on Data plane )
and verify if any known issues are there for your software version and so on.
Thanks
11-26-2013 09:28 AM
Hello,
You could try an another command to see if this helps: PAN>> request high-availability sync-to-remote runtime-state.
Once synchronized message is displayed, execute the following command to all parameter with peers node and also check in the Dashboard.
> show high-availability state
Thanks
11-28-2013 01:44 AM
Just to close this off (I hope)
I tried the above suggestions and the commit was still failing.
However today when I picked up the issue again i decided to try and run the commit from the second box rather than the first and to my surprise it succeeded - the systems now report that they are in sync.
I've no real idea what was going on but I'm going to accept that the issue has resolved itself.
Thanks to everyone for their time - I've got some good tools to at least start investigating the HA next time.
Phil.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
