Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

Reply
L2 Linker

Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

There was a massive outage on Microsoft sites. It has been resolved now, but I was wondering if this something related to Palo Alto Dynamic updates.


https://www.reddit.com/r/sysadmin/comments/9nc9oj/microsoft_authentication_issues_with_akamai_ips/
We just got nailed this morning with issues caused by Palo Alto Firewalls adding an Akamai IP/IP-range to it's blocked IP definitions in it's regular definition updates sometime in the past 12 hours. This caused a blank page for redirection to o365 authentication prior to ADFS for owa/onedrive/sharepoint/etc so users couldn't connect to these services. Just a heads up for anyone else that might be bumping into this, I hear of a number of other local orgs bumping up against this so I'm guessing it's reasonably widespread.

 

 

 

L3 Networker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

Office 365 (and some other MS Online stuff) had an issue, and in our case authentications were going to microsoftonline-p.com addresses. It was categorized as phishing by Palo. I whitelisted it and put in a URL reclassification request which has already been processed.

 

But really, "microsoftonline-p.com" looks like a fake MS site to me at first blush as well, but in searching I found it was legit:

https://support.microsoft.com/en-us/help/2655102/internet-accessible-urls-required-for-connectivity-...

L3 Networker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

Looking deeper in my logs it looks like we started blocking when our URL DB was upgraded to 20181010:40185

L2 Linker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

I've got an update from the TAC as below.

#
The reason for the Microsoft issues is that an Office365 host was incorrectly categorized as 'Phishing' in an earlier PAN-DB update. 
This has since been flipped back to the correct category of 'computer-and-internet-info' in a subsequent update. 

The issue is still being analyzed as we speak. 
We would be posting an update to the Customer Advisory board once we have more details surrounding the incident and an RCA is determined. 

L3 Networker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

Where is the "Customer Advisory Board"?

L2 Linker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

L3 Networker

Re: Microsoft authentication issues with Akamai IPs blocked by Palo Alto (?)

https://live.paloaltonetworks.com/t5/Customer-Advisories/Important-customer-information-regarding-PA...

 

Dear Palo Alto Networks customer,

 

On Thursday, October 11, 2018 at 7:01 AM PT, a Microsoft URL used by Office 365 (secure.aadcdn.microsoftonline-p.com) was inadvertently categorized from “Computer-and-internet-info” to “Phishing,” which may have impacted the availability of Office 365. As soon as we discovered the issue at 10:22 AM PT, we re-categorized the URL to “Computer-and-internet-info,” restoring access to impacted users.

 

During the impacted timeframe, customers who have PAN-DB policies to block the “phishing” category would have been unable to access secure.aadcdn.microsoftonline-p.com, potentially preventing authentication to Microsoft Office 365.

 

Root Cause

 

We immediately began investigating the root cause of this important issue and determined it was due to a human error that resulted in the re-categorization from “Computer-and-internet-info” to “Phishing.”

 

We apologize for any inconvenience this may have caused and thank you for your patience as we worked to resolve this issue. Should you have any questions, please don’t hesitate to reach out to your support provider or the Palo Alto Networks Support Team at https://support.paloaltonetworks.com.  

 

10:22am PDT? That's around the time I submitted the URL re-classification ;)

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!