topic Re: Active/Passive HA Sync Issues in General Topics

Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 16:15:21 GMT

I'm in the process of testing out two PAN-M-100's in the lab and more specifically testing the HA functionality at this point.

The issue that I am running into:

I have changed the Primary to Passive and the Secondary to Active, made a change to the Active/Secondary and then reverted the M-100's back to Active/Primary - Passive/Secondary. After doing this, instead of the Active/Primary pulling the latest config from the Passive/Secondary, it tries to overwrite the config with it's own. So in a nut shell, when we are failed over to our secondary M-100, all the changes we make will have to be redone on the Primary upon fail back.

Running version 5.1.3 (STIG compliance disallows us to upgrade, trust me I wish I could).

Any thoughts?

Re: Active/Passive HA Sync Issues

HULK — Fri, 11 Jul 2014 16:34:47 GMT

Hello Davecorwin,

Could you please try below mentioned command before doing a failover.

admin@114-PANORAMA> request high-availability sync-to-remote

> candidate-config Sync candidate configuration to peer

> clock Sync the local time and date to the peer

> running-config Sync running configuration to peer

admin@114-PANORAMA> request high-availability sync-to-remote running-config

admin@114-PANORAMA> show jobs all --- just to ensure that sync job has been completed.

Then do a failover test and let us know the result.

Thanks

Re: Active/Passive HA Sync Issues

HULK — Fri, 11 Jul 2014 16:40:42 GMT

Forgot to mention, please verify JOBS on the secondary box as well. It should show that, Secondary received a config-sync job from primary and completed successfully.

Thanks

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 16:45:51 GMT

Roger that...stand by...

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 17:08:47 GMT

Yeah, only the PEER will show the sync job. We have successfully performed the sync. Our next step is to unplug the primary M-100 from the switch (totally take it off the network) to cause the secondary to take over as Active/Passive on it's own. I will then make a config change on the Active/Secondary. Once that is complete, I am going to plug the Primary back into the switch...this should automatically make the Primary Active. The issue is that when we do this, the Primary wants to overwrite the config.

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 17:42:08 GMT

Ok, so when the Primary came back in line, as assumed it went straight into active mode. When you go to sync it overwrites the changes you made on the secondary. I was able to get the primary, once back online, to go into passive state and push the sync from the secondary, which worked! The issue is, after only a minute or two, the primary automatically reverts back to active. The M-100 is currently in preemptive mode, so I don't see why this is happening. These devices should successfully/correctly sync without me having to do all of this extra.

Thoughts?

Re: Active/Passive HA Sync Issues

hyadavalli — Fri, 11 Jul 2014 17:55:01 GMT

Hello Dave,

In your situation try disabling pre-emptive on both firewalls.

Regards,

Hari Yadavalli

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 18:06:30 GMT

I actually just got done doing that and disconnected the primary from the switch. The secondary automatically switched to active (as expected) and I created another rule. Once the commit is done, I will plug the primary back into the network. Hopefully the primary stays as passive (since preemptive is turned off). I also hope that the sync process kicks off automatically.

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 18:12:33 GMT

So, once the primary was plugged back into the network, it automatically went into ACTIVE mode...how is this?? That tells me that there is absolutely NOTHING different between preemptive and non-preemptive.

What we are trying now is to leave preemptive off on the primary but turn it on the secondary and see what happens.

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 18:58:07 GMT

Still the same issue. We set up a case with Palo...hopefully they can figure out the issue.

Re: Active/Passive HA Sync Issues

hyadavalli — Fri, 11 Jul 2014 19:13:35 GMT

Hello Dave,

Make sure you commit the changes and disabled pre-emptive on both firewalls.

Regards,

Hari Yadavalli

Re: Active/Passive HA Sync Issues

DaveCorwin — Fri, 11 Jul 2014 19:19:37 GMT

I've done all that...makes no difference. I have tried every kind of way imaginable and the outcome is always the same: The primary automatically switches back to active and the changes on the secondary don't sync with the primary. I have to manually push the sync from the cli of the secondary to properly sync them...I shouldn't have to do that.

Re: Active/Passive HA Sync Issues

ajbool — Fri, 11 Jul 2014 20:05:39 GMT

Perhaps worth checking the time on the two units.

If the primary has a clock time further into the future than the secondary; perhaps this causing its config to be considered more fresh than the secondary and therefore the version that gets pushed out?

Re: Active/Passive HA Sync Issues

Wenar — Mon, 14 Jul 2014 08:26:30 GMT

The preemptive feature has to be activated on both devices to use it. If it's activated the device with the higher priority(lower number) becomes active/(primary). If it's not active the device with the longer uptime and lower MAC will be active. Can you post a screenshot of the HA configuration?

Re: Active/Passive HA Sync Issues

DaveCorwin — Mon, 14 Jul 2014 15:42:36 GMT

I understand all of that...the issue is, I want the "Secondary" to be able to send any new configs to the now, "Active/Primary" upon reinstatement. With the preemptive setting in place, upon reinstatement of the Primary device (making it Active), that device wants to sync it's config with the secondary...I don't want that! By this happening, I'm erasing any config changes that I made, on the secondary, while the primary was down.

Re: Active/Passive HA Sync Issues

ajbool — Mon, 14 Jul 2014 16:42:04 GMT

Could you share the output of "show high-availability all" from both firewalls?

Re: Active/Passive HA Sync Issues

DaveCorwin — Mon, 14 Jul 2014 17:16:19 GMT

These are not firewalls, they are M-100 Panoramas (managers). Below is the the "show high-availability all" of our current VM enviroment. The M-100's that we are testing in the lab have been uploaded with the exact configuration snap-shot of our live VM Panoramas. I cannot provide the read out from the M-100s but it shouldn't matter since everything is the same (Config, licneses, software, dynamic updates, etc).

PRIMARY:

High-Availability:

Local Information:

Version: 1

State: active

Device Information:

Election Option Information:

Priority: primary

Preemptive: yes

Promotion Hold Interval: 2000 ms

Hello Message Interval: 8000 ms

Heartbeat Ping Interval: 1000 ms

Preemption Hold Interval: 1 min

Monitor Fail Hold Up Interval: 0 ms

Addon Master Hold Up Interval: 7000 ms

Version Information:

Build Release: 5.1.3

URL Database: Not Installed

Application Content: 445-2292

Threat Content: 0

Anti-Virus: 1317-1787

Version Compatibility:

Software Version: Match

Application Content Compatibility: Match

Threat Content Compatibility: Match

Anti-Virus Compatibility: Match

Peer Information:

Connection status: up

Version: 1

State: passive

Device Information:

Connection up; Primary HA1 link

Election Option Information:

Priority: secondary

Preemptive: yes

Version Information:

Build Release: 5.1.3

URL Database: Not Installed

Application Content: 445-2292

Threat Content: 0

Anti-Virus: 1317-1787

Configuration Synchronization:

Enabled: yes

Running Configuration: synchronized

SECONDARY:

High-Availability:

Local Information:

Version: 1

State: passive

Last suspended state reason: User requested

Device Information:

Election Option Information:

Priority: secondary

Preemptive: yes

Promotion Hold Interval: 2000 ms

Hello Message Interval: 8000 ms

Heartbeat Ping Interval: 1000 ms

Preemption Hold Interval: 1 min

Monitor Fail Hold Up Interval: 0 ms

Addon Master Hold Up Interval: 7000 ms

Version Information:

Build Release: 5.1.3

URL Database: Not Installed

Application Content: 445-2292

Threat Content: 0

Anti-Virus: 1317-1787

Version Compatibility:

Software Version: Match

Application Content Compatibility: Match

Threat Content Compatibility: Match

Anti-Virus Compatibility: Match

Peer Information:

Connection status: up

Version: 1

State: active

Last suspended state reason: User requested

Device Information:

Connection up; Primary HA1 link

Election Option Information:

Priority: primary

Preemptive: yes

Version Information:

Build Release: 5.1.3

URL Database: Not Installed

Application Content: 445-2292

Threat Content: 0

Anti-Virus: 1317-1787

Path Monitoring Information:

Enabled: yes

Failure condition: any

Groups:

No path monitoring groups

Configuration Synchronization:

Enabled: yes

Running Configuration: synchronized

Again, the preemptive function works correctly up to the point where it doesn't Sync the recently config changes from the secondary to the primary. I have to manually run the sync command from the secondary CLI in order to make the sync work.

Re: Active/Passive HA Sync Issues

ajbool — Mon, 14 Jul 2014 19:03:31 GMT

Looks pretty good to me (I have lots of A/P devices; but my VMware Panorama is just standalone - so can't compare directly). Clocks synced between the two devices OK?

Re: Active/Passive HA Sync Issues

DaveCorwin — Mon, 14 Jul 2014 19:12:42 GMT

Yes, the clocks are synced. It's almost as if there is a bug with the software version that isn't allowing changes made on the secondary device (while in active mode) to transfer to the primary device upon reinstatement.

What's even stranger, is that after turning OFF the preemptive mode on both devices (committing, etc), the primary device still automatically becomes active upon reinstatement to the network. It's like there is no difference between preemptive and non-preemptive.

Re: Active/Passive HA Sync Issues

Wenar — Tue, 15 Jul 2014 07:31:38 GMT

Ok I overlooked that you said it's a M-100. The passive/secondary isn't able to manage policies and if the active/primary comes online again it will sync it's configuration because it's the only device which is allowed to manage the policies.

Panorama Administrator's Guide 6.0 (English) Page 127

I think you have to put them in active/primary and active/secondary if you want the secondary device to be able to sync the configuration.

Whoops my mistake. That is not possible. If this still doesn't work if preemptive is deactivated on both devices you should open a case.

I can't test this here and see how a suspended device behaves if the other peer commits a configuration. Could you see if there is a job running on the suspended device after you commit the configuration on the secondary?

If the guide is right the secondary should still be active after you make the primary functional again and preemptive is disabled on both devices.