topic Re: Package manager upgrade failures to certain sites in Next-Generation Firewall Discussions

Package manager upgrade failures to certain sites

brianhill88 — Wed, 16 Oct 2024 23:04:55 GMT

Hello,

We have a case open for this that has been turned over to internal dev but I thought I would post this here to see if anyone else was experiencing this issue. This on a 5260 running 10.2.7-h3.

We been tracking down this issue for a long time. It started as Mac build servers that reach out to the Internet using brew to install package upgrades. These would fail periodically but because it was working more than not it wasn't reported at first. The problem was eventually reported and we had both our network team and platform team start looking at it. We conducted numerous packet captures and also worked with our network hardware vendor to make sure the problem wasn't the network itself. When we started focusing on the Palo Alto we noticed missing return packets in the Palo Alto captures on the box. Palo Alto thought that was because the traffic was being offloaded to hardware. So in an effort to find the missing packets we did a session where we turned off hardware offloading. Unexpectedly the package upgrades with brew no longer failed. So it appears there is some kind of issue with the traffic being offloaded to hardware but not conclusive yet. We did notice we do not have the same issue on a 3260 running the same code train.

Since then we have seen other 443 issues with other package managers. We also wondering if there are other 443 issues just not being reported because they work the majority of the time.

Re: Package manager upgrade failures to certain sites

brianhill88 — Thu, 17 Oct 2024 04:54:42 GMT

Interestingly it looks like we solved our issue this evening. We wanted to try and upgrade the firewalls this evening to 10.2.9-h11. Before upgrading I wanted to reload the firewall first and test again on the off chance a reload fixed it (we had failed the firewalls over before but never reloaded). After reloading the passive and then making it active the problem no longer occurs. If we fail over to the one that hasn't been reloaded the problem comes back. Seems to indicate there is some kind of memory leak or some kind of uptime issue with this code train.

Re: Package manager upgrade failures to certain sites

brianhill88 — Fri, 18 Oct 2024 18:05:24 GMT

Unfortunately the solution was short lived. Within a day the problem started happening again. Last night we moved forward with upgrading the firewalls to 10.2.9-h11. The problem still persists as of testing this morning.

Re: Package manager upgrade failures to certain sites

brianhill88 — Mon, 28 Oct 2024 16:50:40 GMT

Update:

The update to 10.2.9-h11 actually introduced a new problem where all TLS traffic stopped working after 6 days. This is a known issue and they are releasing a hotfix for it. We ended up rolling back to the a previous version.

Solution:

PA engineers were able to give us a fix for the original issue we were experiencing.

The solution was to change the LAG flow from type tag to type tuple:

set session lag-flow-key-type tuple

show session lag-flow-key-type

After setting it to tuple, initial testing shows we are no longer seeing the issue. This appears to keep each unique session on a particular link in a LAG.

Re: Package manager upgrade failures to certain sites

daniel0021 — Fri, 18 Apr 2025 10:25:18 GMT

Thanks for sharing this, Brian. Sounds like you've been doing some serious deep tracking—like trying to find a parcel with a UMAC tracking number that randomly vanishes and reappears! Interesting that disabling hardware offloading fixed it. We've seen similar intermittent failures with package managers, but nothing consistent enough to pin down. Definitely keeping an eye on this thread for updates!