topic Re: Slack hooks server certificate invalid in Log Forwarding Discussions

Slack hooks server certificate invalid

rlarose — Wed, 15 Mar 2023 13:44:50 GMT

Our firewalls cannot send to hooks.slack.com since they refreshed their cert yesterday (3/14/2023).

I suspect a problem with the way their chain is signing X1 root CA but until they fix it, is there a way to allow the log forwarding service to ignore the invalid cert and send anyway? I see a kb article about doing this for decryption profiles, but not sure if it applies here.

Also is there any debugging that can be done on the palo to get more specific detail about what its problem is with the cert?

Thanks in advance for anyone who can advise.

Re: Slack hooks server certificate invalid

rlarose — Wed, 15 Mar 2023 15:00:58 GMT

I'm advised by Slack and the LetsEncrypt folks that the "long chain" certificate format being used is valid, so I guess I need a way to tell the firewall that this is okay.

We're running PanOS 9.1.x -- possible this is addressed in a later OS update?

Re: Slack hooks server certificate invalid

scottymuse — Wed, 15 Mar 2023 20:02:26 GMT

We're running 10.1.8-h2 but having the same issue.

Re: Slack hooks server certificate invalid

rlarose — Wed, 15 Mar 2023 20:19:34 GMT

Well there go my hopes for an upgrade solution.

Testing against any LE long-chain server (e.g., letsencrypt.org slack.com nba.com) results in failure. Testing against any LE short-chain (e.g., la-sso.bounce51.com) or non-LE (e.g., gmail.com) results in successful validation.

So it does appear to be tied to how Palo's Log Forwarding HTTPS process interprets that long-chain LetsEncrypt cert with the expired X3 root.

Do you have a PaloAlto support case open? We should reference each other so they know this is not us, it's them.

Re: Slack hooks server certificate invalid

onercan — Thu, 16 Mar 2023 13:42:21 GMT

Hello everyone,

We're running 10.0.8-h4 but having the same issue for 3 days. We follow some logs with push notification from Slack.

If you find the solution to the problem, can you share it here?

I hope this issue will be resolved as soon as possible.

Re: Slack hooks server certificate invalid

rlarose — Thu, 16 Mar 2023 14:16:25 GMT

@onercan and @scottymuse Can you provide your PaloAlto suport case #numbers? I'd like to make sure they are aware this is a PAN-OS issue, not any of our specific configurations.

Re: Slack hooks server certificate invalid

rlarose — Thu, 16 Mar 2023 14:16:35 GMT

Can you both provide your PaloAlto suport case #numbers? I'd like to make sure they are aware this is a PAN-OS issue, not any of our specific configurations.

Re: Slack hooks server certificate invalid

scottymuse — Thu, 16 Mar 2023 20:13:10 GMT

I just created case 02499701

Re: Slack hooks server certificate invalid

onercan — Fri, 17 Mar 2023 16:31:00 GMT

We still haven't solved the issue. And you?@scottymuse @rlarose

Re: Slack hooks server certificate invalid

rlarose — Fri, 17 Mar 2023 17:03:27 GMT

No solution yet @onercan -- can you share your PaloAlto support case number? It will help lend weight when we can make clear that this is not an individual config problem, but rather a PanOS problem.

Re: Slack hooks server certificate invalid

onercan — Mon, 20 Mar 2023 16:51:58 GMT

We can't open to case for 10.0.8-h4 End of Support.

Did palo alto engineers respond to the case? @scottymuse

Re: Slack hooks server certificate invalid

rlarose — Mon, 20 Mar 2023 17:04:37 GMT

We know it's not inherent to the PanOS version, as I'm runing 9.1 and @scottymuse is running 10.1.

Are your firewalls just not under support at all? As long as they are, even on the end-of-support OS, you should be able to raise a case and it would help put pressure on Palo to acknowledge and address it.

Also press the issue with Slack -- it's their change that broke things.

Re: Slack hooks server certificate invalid

scottymuse — Mon, 20 Mar 2023 17:14:01 GMT

Here is the latest response I received:

Greetings! As you mentioned earlier there is a workaround going on related to this issue. It is related to a feature request. I have checked the case associated with Rlarose and the case was closed. Kindly Let me know if you have any concerns regarding this issue I will be happy to assist you. Have a great day!

I'm not exactly happy with that reply. The workaround I mentioned to TAC was we stopped using it temporarily while it is broken and modified our workflow. I guess properly reading certs is a feature request now?

Re: Slack hooks server certificate invalid

rlarose — Mon, 20 Mar 2023 17:19:06 GMT

My case has not been closed -- they're referring to the case about 2 weeks prior when slack also jiggled the handle on their cert (maybe a dry-run?) which caused me some trouble.

My open, active, unresovled case number you can reference is 02496793

Re: Slack hooks server certificate invalid

scottymuse — Mon, 20 Mar 2023 17:28:12 GMT

Yeah, I figured there was some confusion (to put it mildly) on that reply regarding your case. I've replied asking for a time frame I could expect this feature request to be fulfilled.

Re: Slack hooks server certificate invalid

rlarose — Mon, 20 Mar 2023 17:35:59 GMT

The feature request they're referring to from the closed case is not related to this problem. It's about avoiding the infinite loop of system logs trying to send to a dead log server, which fails & raises a system log which it tries to send to the log server, which fails & raises a system log..... etc forever.

Re: Slack hooks server certificate invalid

scottymuse — Mon, 20 Mar 2023 17:39:48 GMT

Ah, yes, now I'm the one confused. Either way, I also referenced your current active case and suggested that properly validating certificates should be a priority. We'll see how they respond.

Re: Slack hooks server certificate invalid

scottymuse — Fri, 24 Mar 2023 18:31:01 GMT

Good news, TAC responded to me and called this an "outbreak globally for the slack integration users". They are looking into the cause. My guess is this will be fixed in a future update, but they'll let me know.

Re: Slack hooks server certificate invalid

rlarose — Fri, 24 Mar 2023 18:33:51 GMT

Well acknowledging the noise we're making is a good first step. They need to band-aid it right away, though.

Re: Slack hooks server certificate invalid

onercan — Fri, 24 Mar 2023 18:51:42 GMT

This is really goods news, they finally heard our voice. I hope the update will be posted as soon as possible.