Dynamic Updates and Device Registration Failing in GCP. HELP!!!

[SCantwell_IM]

Ok Gurus,  (including @BPry ) I got a need for some help and visibility on this one... please... 

My FWs are virtualized within a open source (KVM) environment, that was placed into GCP back in June 2019.

Somewhere along this week, when I first noticed that I am not getting my Dynamic Updates, cannot retrieve license keys from license server, nothing... 

I looked at a different set of FWs that are in GCP (when they were booted up on Jan 12, and they have not gotten their updates either.  

We think there is something going on with GCP, because it was working fine for nearly 6 months. 

I am wondering anyone on this forum is running into similar "weird" stuff going on within GCP.

There has been literally no changes made to the virtualized pods.

I am asking for confirmation that someone (please) test with their GCP hosted virtual FWs and let me know.

I have wiresharks from internet facing GCP port, and the inside interface (facing the FW) and I am seeing re-transmissions from GCP. 

I have a ticket open with PANW TAC and they are not able to confirm this issue.... they showed me there was connectivity to their servers.

For example... it is hit or miss, but if I do a "request content upgrade check" will complete... but other times, it just hangs and I get the "generic communication" error message. 

Yet this happens on EVERY SINGLE virtualized firewall that we have deployed... going back to Jan 12th.

Again, please do testing or advising your thoughts.

Thank you.

Steve

[SCantwell_IM]

Ok Gurus,  (including @BPry ) I got a need for some help and visibility on this one... please... 

My FWs are virtualized within a open source (KVM) environment, that was placed into GCP back in June 2019.

Somewhere along this week, when I first noticed that I am not getting my Dynamic Updates, cannot retrieve license keys from license server, nothing... 

I looked at a different set of FWs that are in GCP (when they were booted up on Jan 12, and they have not gotten their updates either.  

We think there is something going on with GCP, because it was working fine for nearly 6 months. 

I am wondering anyone on this forum is running into similar "weird" stuff going on within GCP.

There has been literally no changes made to the virtualized pods.

I am asking for confirmation that someone (please) test with their GCP hosted virtual FWs and let me know.

I have wiresharks from internet facing GCP port, and the inside interface (facing the FW) and I am seeing re-transmissions from GCP. 

I have a ticket open with PANW TAC and they are not able to confirm this issue.... they showed me there was connectivity to their servers.

For example... it is hit or miss, but if I do a "request content upgrade check" will complete... but other times, it just hangs and I get the "generic communication" error message. 

Yet this happens on EVERY SINGLE virtualized firewall that we have deployed... going back to Jan 12th.

Again, please do testing or advising your thoughts.

Thank you.

Steve

[BPry]

@SCantwell_IM,

Wish I could help, but I don't have anything currently spun up in GCP to verify this behavior. PAN itself is extremely invested in GCP, so I find it doubtful that it would be a systemic issue across GCP in its entirety. 

You might want to try creating a service route to move this traffic off of the management interface, and then create a PCAP directly on the firewall to capture this traffic. That might show you something more definitive to show either Google on PAN support. 

[SutareMayur]

@SCantwell_IMHey, can you please try to change service route for Palo Alto Network updates and check if you are able to update?

- Mayur

[SCantwell_IM]

Thanks @BPry !

I have tried many different iterations, changing service routes, around, etc.

I even changed the default GW of the Mgmt interface, so that it bypasses the FW itself, and issue continues.

Strangeness in that we noticed an old VM firewall that was spun up (pretty much untouched) since Jan 12, and it cannot get updates as well.

We also know that the updates.paloaltonetworks.com is also in GCP now, so maybe some intern-GCP routing issues.

We have tried staticupdates.paloaltonetworks.com and various public IPs, and it (for what seems to be) system wide, hence my email to all GCP spun up firewall ppl, to test... maybe it is our KVM, but yet, other traffic works fine.

Weird... just leaving this up for others to comment in.