problems after RMA of an active-passive pair

[RREALICA]

just to give a baseline - hardware are pa-5220, running 10.1.6-h6, HA (active/passive), device managed by Panorama.

Long story short - one of the firewalls got stuck on a reboot cycle where the firewall would reboot every 85 minutes. The data plane would not come up during this time and support processed a RMA. 

For the fun part:

our initial attempt to restore, after upgrading the same software level as working firewall, we used a device state backup, everything sync up properly, when we tried to pass traffic through the new firewall we saw the following strange behavior:

1. source zone and destination zone was not showing up in the logs
2. some zones were being identified but was incorrect, for instance DMZ, is showing up as Trust
3. destination country was reporting incorrectly, for instance traffic for United States is showing up for Uruguay
4. traffic seems to be flowing properly even though rules were not being identified in the logs

2nd restore attempt we used a back from a day before the firewall failed, we had no issues with synching up in this case as well. this gave us a better experience.

1. all traffic flows looks good on the log, zones and rules were being identified properly
2. destination country looks good as well
3. the one problem we noticed was that the IPSEC tunnels would not come up - after troubleshooting with support even a simple command "ping source x.x.x.x host y.y.y.y" where x.x.x.x is one of the firewall interface ip results in "bind: Cannot assign requested address".   We checked for "untagged sub interface" like suggested by others but none found.

Any feedback, suggestions would be appreciated. A TAC support is currently open and waiting feedback as well.

Thanks.

[JayGolf]

Hi @RREALICA ,

Have you received any feedback from TAC yet? 

[RREALICA]

Issues has been resolved - that's the good news. We are going to push to get a RCA in this case since the problems encountered does not seem normal.

1. To fix the error "bind: Cannot assign requested address"- create a dummy/temporary L3 interface, commit, test and then remove the dummy/temporary L3 interface.  No amount of reboots, "commit force" by different levels of TAC did not make a difference. Also TAC mentioned there's a cli command to refresh the interface and zones but we did not try this.

2. Another problem came about where the IPSEC VPNs were not coming - this turns out the ISAKMP process is not properly starting up or not listening for IPSEC traffic.  Check for ISAKMP via "show netstat listening yes | match isakmp" to see if the firewall is listening for the traffic - restart of ISAKMP process resolved the issue - a reboot of the firewall also may have been best after fixing the interface related issues.

Thanks.