topic Troubleshooting HA Sync in General Topics

Troubleshooting HA Sync

pcook — Tue, 26 Nov 2013 13:06:00 GMT

Hi All,

I've a HA Active/Active pair of PA-3020's managed via Panorama.

All was well until I had to replace one of the pair, I brought the new device into Panorama, added it to the correct group/templates and then finished the device config (IP addresses etc).

The last step was to configure it as part of the HA.

However the two systems are failing to sync policy.

So far all I can see is the entries in the system log widget:

HA Group 1: Running configuration not synchronized after retries 11/26 12:25:23

running configuration synchronized with HA peer by admin 11/26 12:10:23

Where do I go next to find out why they won't sync?

Phil.

Re: Troubleshooting HA Sync

VinceM — Tue, 26 Nov 2013 15:02:10 GMT

Hi,

Are you sure you have activate the panorama pbject sharing on local device ?

Hope help.

Re: Troubleshooting HA Sync

pcook — Tue, 26 Nov 2013 15:17:18 GMT

Hi Vince,

Yes that's active - the push from Panorama works fine - it's the HA peer sync of the running config that's failing.

Phil

Re: Troubleshooting HA Sync

Phoenix — Tue, 26 Nov 2013 15:25:34 GMT

Hello Pcook,

Here is the command to try form the CLI,

> request high-availability sync-to-remote running-config

Also in the ms.log we will see details what is happening with error details and logs indicating the fail for the HA-sync

Run the below command first " tail follow yes mp-log ms.log" and then try to do HA Sync to see the logs dynamically. This would give some explanation.

May also have to look at any other pending job is running on the peer device

show jobs pending

Hope this would give a path.

Thanks

Re: Troubleshooting HA Sync

pcook — Tue, 26 Nov 2013 16:09:35 GMT

Thanks Phoenix, that helped in that it's a useful command but I'm still not sure where the error is. There are some error lines at the start but the sync seems to run after that.

Do you think I should raise a support ticket?

Phil.

tail follow yes mp-log ms.log wrote:

Nov 26 15:32:49 Getting authorization info for user cookp succeeded.

Nov 26 15:32:49 cms-client: Found role superuser for user cookp

Nov 26 15:32:49 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:32:51 Error: pan_xml_get_node_by_xpath(pan_xml_helper.c:570): cannot evalute xpath expression:/config/devices/entry[@name='localhost.localdomain']/deviceconfig/system

Nov 26 15:32:52 Error: pan_xml_get_node_by_xpath(pan_xml_helper.c:570): cannot evalute xpath expression:/config/devices/entry[@name='localhost.localdomain']/deviceconfig/system

Nov 26 15:32:58 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:2025): failed to fetch: NO_MATCHES

Nov 26 15:32:58 Error: pan_cfg_mgr_get_tpl_disabled(pan_cfg_mgr.c:2047): failed to fetch: NO_MATCHES

Nov 26 15:33:00 Error: pan_cfg_mgr_get_sp_disabled(pan_cfg_mgr.c:2025): failed to fetch: NO_MATCHES

Nov 26 15:33:00 report generation started for 'acc-summary'

Nov 26 15:33:00 generating report for time from 1385476380 to 1385479979

Nov 26 15:33:00 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:33:00 report generation: vectorization started for 'acc-summary'

Nov 26 15:33:00 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:33:00 report generation: sorting started for 'acc-summary'

Nov 26 15:33:00 report generation: sorting ended for 'acc-summary'

Nov 26 15:33:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:33:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:33:11 report generation started for 'acc-summary'

Nov 26 15:33:11 generating report for time from 1385476391 to 1385479990

Nov 26 15:33:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:33:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:33:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:33:11 report generation: sorting started for 'acc-summary'

Nov 26 15:33:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:33:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:33:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:33:11 report generation started for 'acc-summary'

Nov 26 15:33:11 generating report for time from 1385476391 to 1385479990

Nov 26 15:33:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:33:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:33:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:33:11 report generation: sorting started for 'acc-summary'

Nov 26 15:33:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:33:12 Error: pan_logjobmgr_print_job(pan_log_mgr.c:2173): no results entry for log job id 1055

Nov 26 15:33:12 Error: pan_mgmtop_show_query(pan_ops_common.c:2105): Unable to find query job

Nov 26 15:33:20 client dagger reported op command was SUCCESSFUL

Nov 26 15:34:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:34:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:34:11 report generation started for 'acc-summary'

Nov 26 15:34:11 generating report for time from 1385476451 to 1385480050

Nov 26 15:34:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:34:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:34:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:34:11 report generation: sorting started for 'acc-summary'

Nov 26 15:34:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:35:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:35:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:35:11 report generation started for 'acc-summary'

Nov 26 15:35:11 generating report for time from 1385476511 to 1385480110

Nov 26 15:35:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:35:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:35:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:35:11 report generation: sorting started for 'acc-summary'

Nov 26 15:35:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:36:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:36:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:36:11 report generation started for 'acc-summary'

Nov 26 15:36:11 generating report for time from 1385476571 to 1385480170

Nov 26 15:36:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:36:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:36:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:36:11 report generation: sorting started for 'acc-summary'

Nov 26 15:36:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:37:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:37:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:37:11 report generation started for 'acc-summary'

Nov 26 15:37:11 generating report for time from 1385476631 to 1385480230

Nov 26 15:37:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:37:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:37:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:37:11 report generation: sorting started for 'acc-summary'

Nov 26 15:37:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:38:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:38:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:38:11 report generation started for 'acc-summary'

Nov 26 15:38:11 generating report for time from 1385476691 to 1385480290

Nov 26 15:38:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:38:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:38:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:38:11 report generation: sorting started for 'acc-summary'

Nov 26 15:38:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:39:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:39:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:39:11 report generation started for 'acc-summary'

Nov 26 15:39:11 generating report for time from 1385476751 to 1385480350

Nov 26 15:39:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:39:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:39:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:39:11 report generation: sorting started for 'acc-summary'

Nov 26 15:39:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:40:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:40:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:40:11 report generation started for 'acc-summary'

Nov 26 15:40:11 generating report for time from 1385476811 to 1385480410

Nov 26 15:40:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:40:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:40:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:40:11 report generation: sorting started for 'acc-summary'

Nov 26 15:40:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:41:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:41:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:41:11 report generation started for 'acc-summary'

Nov 26 15:41:11 generating report for time from 1385476871 to 1385480470

Nov 26 15:41:11 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:41:11 report generation: vectorization started for 'acc-summary'

Nov 26 15:41:11 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:41:11 report generation: sorting started for 'acc-summary'

Nov 26 15:41:11 report generation: sorting ended for 'acc-summary'

Nov 26 15:42:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:42:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:42:12 report generation started for 'acc-summary'

Nov 26 15:42:12 generating report for time from 1385476932 to 1385480531

Nov 26 15:42:12 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:42:12 report generation: vectorization started for 'acc-summary'

Nov 26 15:42:12 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:42:12 report generation: sorting started for 'acc-summary'

Nov 26 15:42:12 report generation: sorting ended for 'acc-summary'

Nov 26 15:43:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:43:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:43:12 report generation started for 'acc-summary'

Nov 26 15:43:12 generating report for time from 1385476992 to 1385480591

Nov 26 15:43:12 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:43:12 report generation: vectorization started for 'acc-summary'

Nov 26 15:43:12 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:43:12 report generation: sorting started for 'acc-summary'

Nov 26 15:43:12 report generation: sorting ended for 'acc-summary'

Nov 26 15:44:11 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:44:11 client dagger reported op command was SUCCESSFUL

Nov 26 15:44:12 report generation started for 'acc-summary'

Nov 26 15:44:12 generating report for time from 1385477052 to 1385480651

Nov 26 15:44:12 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:44:12 report generation: vectorization started for 'acc-summary'

Nov 26 15:44:12 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:44:12 report generation: sorting started for 'acc-summary'

Nov 26 15:44:12 report generation: sorting ended for 'acc-summary'

Nov 26 15:44:46 client routed reported op command was SUCCESSFUL

Nov 26 15:45:42 client dagger reported op command was SUCCESSFUL

Nov 26 15:46:22 dnscfgmod: FQDN Refresh: Periodic TTL Expiry Refresh

Nov 26 15:46:22 dnscfgmod: Main refresh function: (TTL Expiry)

Nov 26 15:46:22 dnscfgmod: No fqdns used in this config. Skip config push to device

Nov 26 15:53:01 client dagger reported op command was SUCCESSFUL

Nov 26 15:53:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:53:02 report generation started for 'acc-summary'

Nov 26 15:53:02 generating report for time from 1385477582 to 1385481181

Nov 26 15:53:02 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:53:02 report generation: vectorization started for 'acc-summary'

Nov 26 15:53:02 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:53:02 report generation: sorting started for 'acc-summary'

Nov 26 15:53:02 report generation: sorting ended for 'acc-summary'

Nov 26 15:54:01 client dagger reported op command was SUCCESSFUL

Nov 26 15:54:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:54:01 report generation started for 'acc-summary'

Nov 26 15:54:01 generating report for time from 1385477641 to 1385481240

Nov 26 15:54:01 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:54:01 report generation: vectorization started for 'acc-summary'

Nov 26 15:54:01 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:54:01 report generation: sorting started for 'acc-summary'

Nov 26 15:54:01 report generation: sorting ended for 'acc-summary'

Nov 26 15:55:01 client dagger reported op command was SUCCESSFUL

Nov 26 15:55:01 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:363): failed to fetch cfg.gpdatafile-release-date

Nov 26 15:55:01 report generation started for 'acc-summary'

Nov 26 15:55:01 generating report for time from 1385477701 to 1385481300

Nov 26 15:55:01 report generation: query on type appstat started for 'acc-summary'

Nov 26 15:55:01 report generation: vectorization started for 'acc-summary'

Nov 26 15:55:01 report generation: pre-sort-hook started for 'acc-summary'

Nov 26 15:55:01 report generation: sorting started for 'acc-summary'

Nov 26 15:55:01 report generation: sorting ended for 'acc-summary'

Re: Troubleshooting HA Sync

Phoenix — Tue, 26 Nov 2013 16:45:43 GMT

Hello Pcook,

Did the CLI command help in getting the HA(High availability ) pair to be in Sync.

If not at this point a support case is the option. The reason is an analysis can be obtained by running the logs at debug level for ms.log and also find what is happening in the "ha-agent.log" ( This log has all the details about HA operation on Management Plane ), "pan_dha.log" ( HA logs on Data plane )

and verify if any known issues are there for your software version and so on.

Thanks

Re: Troubleshooting HA Sync

HULK — Tue, 26 Nov 2013 17:28:54 GMT

Hello,

You could try an another command to see if this helps: PAN>> request high-availability sync-to-remote runtime-state.

Once synchronized message is displayed, execute the following command to all parameter with peers node and also check in the Dashboard.

> show high-availability state

Thanks

Re: Troubleshooting HA Sync

pcook — Thu, 28 Nov 2013 09:44:24 GMT

Just to close this off (I hope)

I tried the above suggestions and the commit was still failing.

However today when I picked up the issue again i decided to try and run the commit from the second box rather than the first and to my surprise it succeeded - the systems now report that they are in sync.

I've no real idea what was going on but I'm going to accept that the issue has resolved itself.

Thanks to everyone for their time - I've got some good tools to at least start investigating the HA next time.

Phil.