Slack hooks server certificate invalid

[rlarose]

Our firewalls cannot send to hooks.slack.com since they refreshed their cert yesterday (3/14/2023).

I suspect a problem with the way their chain is signing X1 root CA but until they fix it, is there a way to allow the log forwarding service to ignore the invalid cert and send anyway?  I see a kb article about doing this for decryption profiles, but not sure if it applies here.

Also is there any debugging that can be done on the palo to get more specific detail about what its problem is with the cert?

Thanks in advance for anyone who can advise.

[rlarose]

I'm advised by Slack and the LetsEncrypt folks that the "long chain" certificate format being used is valid, so I guess I need a way to tell the firewall that this is okay.

We're running PanOS 9.1.x -- possible this is addressed in a later OS update?

[scottymuse]

We're running 10.1.8-h2 but having the same issue.

[rlarose]

Well there go my hopes for an upgrade solution.

Testing against any LE long-chain server (e.g., letsencrypt.org slack.com nba.com) results in failure.  Testing against any LE short-chain (e.g., la-sso.bounce51.com) or non-LE (e.g., gmail.com) results in successful validation.

So it does appear to be tied to how Palo's Log Forwarding HTTPS process interprets that long-chain LetsEncrypt cert with the expired X3 root.

Do you have a PaloAlto support case open?  We should reference each other so they know this is not us, it's them.

[onercan]

Hello everyone,

We're running 10.0.8-h4 but having the same issue for 3 days. We follow some logs with push notification from Slack.

If you find the solution to the problem, can you share it here?

I hope this issue will be resolved as soon as possible.

[rlarose]

@onercan and @scottymuse Can you provide your PaloAlto suport case #numbers?  I'd like to make sure they are aware this is a PAN-OS issue, not any of our specific configurations.

[rlarose]

Can you both provide your PaloAlto suport case #numbers?  I'd like to make sure they are aware this is a PAN-OS issue, not any of our specific configurations.

[scottymuse]

I just created case 02499701

[onercan]

We still haven't solved the issue. And you?@scottymuse @rlarose

[rlarose]

No solution yet @onercan -- can you share your PaloAlto support case number?  It will help lend weight when we can make clear that this is not an individual config problem, but rather a PanOS problem.

[onercan]

We can't open to case for 10.0.8-h4 End of Support. 

Did palo alto engineers respond to the case? @scottymuse

[rlarose]

We know it's not inherent to the PanOS version, as I'm runing 9.1 and @scottymuse is running 10.1.

Are your firewalls just not under support at all?  As long as they are, even on the end-of-support OS, you should be able to raise a case and it would help put pressure on Palo to acknowledge and address it.

Also press the issue with Slack -- it's their change that broke things.

[scottymuse]

Here is the latest response I received:

Greetings!

As you mentioned earlier there is a workaround going on related to this issue.

It is related to a feature request.

I have checked the case associated with Rlarose and the case was closed.

Kindly Let me know if you have any concerns regarding this issue I will be happy to assist you.

Have a great day!

I'm not exactly happy with that reply. The workaround I mentioned to TAC was we stopped using it temporarily while it is broken and modified our workflow. I guess properly reading certs is a feature request now?

[rlarose]

My case has not been closed -- they're referring to the case about 2 weeks prior when slack also jiggled the handle on their cert (maybe a dry-run?) which caused me some trouble.

My open, active, unresovled case number you can reference is 02496793

[scottymuse]

Yeah, I figured there was some confusion (to put it mildly) on that reply regarding your case. I've replied asking for a time frame I could expect this feature request to be fulfilled.