Yes, it happened. The primary PA FW was crashed and we couldn't recover. We had a huge network outage, it all happened with a faulty network switch which continuously sends the topology change packets to the entire network resulting in a network loop. At some point the FW was not responding, did a reboot after the backup. It never came back, the status lights were off except the power light. Even nothing from the console port as well, no messages or boot information showed up. Luckily we had the PA OSS device. This is what we did. 1. Raised a case with the support at the highest priority 2. We powered on the OSS device and confirmed the PAN OS version 3. There was a version difference between these 2 devices 4. Downloaded the base PAN OS and the updates to match the version of PAN OSS to the live unit 5. With the help of mgmt port, updated the OS of OSS unit 6. Restored latest backup to the OSS unit 7. We did received some errors during the restoration, I believe those were some Firewall security rules, which was easy to troubleshoot 8. The support guys came into the scenario 9. For some reason, the OSS unit was not showing in device list. May be it wasn't added during the time of purchase 10. They added the unit with the SN # information 9. Downloaded the latest AV, URL, GP, Clientless VPN and other licenses from the portal and restored to the OSS unit 10. Removed the existing license information of faulty unit from the Portal 11. Assigned the license to the OSS unit and renamed the Unit to Production one 12. All services were restored 13. RMA was issued, but it took 2 weeks or so 14. The core team was unable to analyze the firewall as it was completely given up The overall process might have taken 2-3 hours. As we were doing this for the first time on working hours it was a difficult one. Having the support engineer on the right time saved us a big time or else we could have spend more hours figuring out on how to transfer the licenses.
... View more