Slack hooks server certificate invalid

[rlarose]

Our firewalls cannot send to hooks.slack.com since they refreshed their cert yesterday (3/14/2023).

I suspect a problem with the way their chain is signing X1 root CA but until they fix it, is there a way to allow the log forwarding service to ignore the invalid cert and send anyway?  I see a kb article about doing this for decryption profiles, but not sure if it applies here.

Also is there any debugging that can be done on the palo to get more specific detail about what its problem is with the cert?

Thanks in advance for anyone who can advise.

[rlarose]

The feature request they're referring to from the closed case is not related to this problem.  It's about avoiding the infinite loop of system logs trying to send to a dead log server, which fails & raises a system log which it tries to send to the log server, which fails & raises a system log..... etc forever.

[scottymuse]

Ah, yes, now I'm the one confused. Either way, I also referenced your current active case and suggested that properly validating certificates should be a priority. We'll see how they respond.

[scottymuse]

Good news, TAC responded to me and called this an "outbreak globally for the slack integration users". They are looking into the cause. My guess is this will be fixed in a future update, but they'll let me know.

[rlarose]

Well acknowledging the noise we're making is a good first step.  They need to band-aid it right away, though.

[onercan]

This is really goods news, they finally heard our voice. I hope the update will be posted as soon as possible.

[VladimirL]

Joining with same issue, Pan-OS v10.1.3

TAC working on a workaround, no proper ETA yet

[rlarose]

Welcome to the party @VladimirL  please share any progress you're able to make on your case.

[onercan]

Hello everyone,

Is the slack issue solved?

[rlarose]

Hi @onercan - no, not at this time.  I've been advised by Palo Support that PanOS 10.2 does not have the issue, but can't verify that as we're on actively-supported v9.1.15.

I saw 9.1.16 dropped this week --the release notes mention expired root cert handing for ssl decryption but not specifically with HTTPS log forwarding process..

[onercan]

@rlarosedo you think PanOS support team tested it on v10.2? We will upgrade from v10.0 to 10.2 if it works on v10.2.

[rlarose]

The *impression* I got was that someone over there had tested in on 10.2 and decided it didn't have the same problem.  But I can't swear to that.  The fact that it's a problem at all suggests they have not been thorough with their testing/validating.

[onercan]

Hello everyone,

We upgraded from 10.0.8 to v10.2.4. It's working fine. Our SOC is active. we are leaving to party 😂

[btsecurity]

Awesome . . .

[rlarose]

@onercan congratulations, that's good news.

I was away last week but did receive a reasonable RCA from Palo, as well as a workaround to restore successful connectivity in PanOS versions prior to 10.2.  Early test of their workaround was successful and I'll be deploying today.

Did y'all also get the email I got from Slack saying that by May 9th all clients will need to have a working X1 Root certificate?  That to me suggests they are going to switch from LE long-chain to LE short-chain after all, which will presumably render this problem moot.  I have not received direct contact from my Slack account or support folks about this, though, so only speculating what the broadcast email means.

[EO-AlexJinks]

Following this - Running 10.1.9-h1 and still having this issue. Has anyone received a workaround or ETA on fix?