Office 365 URL feeds include Dropbox and itunes?

[Sec101]

Just started going through the new miners and looking over docs, we are not in production on our deployment for O365 yet.  I understand there are app-Id's that would catch most of these, but I noticed that the URL Minemeld feed for the "any-any" version includes quite a few URLS....specifically including:

www.dropbox.com

www.youtube.com

*.itunes.apple.com

I understand that the best way to police this type of access is to use app-id and decryption in a single ruleset that would include these lists.  I just don't understand why items like the above are in this list getting mined directly from microsoft.....  

I'm guessing that SSL decryption is absolutely essential before allowing anything out to these lists, and that you would only use these lists in your O365 app-id rulesets?- not the dependencies?  

It appears that in the ruleset setup in the below article banking on the dependencies for O365-SSL and Web-browsing, as the firewall would read-down the ruleset, it would see your O365 enterprise-access custom app- and hit that rule, and then proceed to your dependencies.    The catch all dependencies rule, just makes me a bit nervous,

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkzCAC 

Long story short- are people creating exclude lists, to Omit some of these IP/URL's that they don't want to mine? 

[lmori]

Hi @Sec101,

you can generate a feed with 3rd party removed by using the output o365-api.feed-no-3rdparty prototype: https://github.com/PaloAltoNetworks/minemeld-node-prototypes/blob/master/prototypes/o365-api.yml#L50...

It applies some dump heuristics to detect if the IP Address/URLs belongs to MSFT or to 3rd parties.

[Sec101]

If I'm reading this right, that miner is keying off of the keyword  "integration"  in the link that Microsoft provides, right?  Is this marked experimental due to that- or that they could change that keyword at any time?

PS- Thank you Lmori  for the quick replies. Your a wizard on this.

[lmori]

That's correct. I am checking with MSFT if there is a better way. Also note there is a bug in that prototype, I will fix in the  next release.

[taustin]

We've run into the same 3rd party issue. Therefore, per this thread, we implemented the "...no-3rdparty" output but it is not removing the 3rd parties like Dropbox.com. The thread also mentioned an issue with this output, is there an ETA on when the issue will be resolved?  Thanks

[lmori]

We have added a new feature that will be shipped in the next release (0.9.52 - by the end of the week). The O365 Miners now have an "Integrations" flag on the WebUI. By disabling the Integrations, 3rd party URLs will be removed.

[ice-quake]

After I toggle "integrations" the # of indicators drops to 618 from 654, but still includes youtube.com and itunes.com etc.  What is the logic used to exclude 3rd party integrations?

I think I'm going to create some new nodes to match the "required" note in the category field.

https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges?redirectSourcePath=...

Category: Shows whether the endpoint set is categorized as “Optimize”, “Allow”, or “Default”. You can read about these categories and guidance for management of them at http://aka.ms/pnc. This column also lists which endpoint sets are required to have network connectivity. For endpoint sets which are not required to have network connectivity, we provide notes in this field to indicate what functionality would be missing if the endpoint set is blocked. If you are excluding an entire service area, the endpoint sets listed as required do not require connectivity.

[lmori]

Hi @ice-quake,

the nodes look for the word "integration" in the notes attribute of the URLs/IPs sections. This is far from optimal, but today there is no way to identify programmatically if a URLs/IPs in that list belongs to MSFT or not. The "required" field has a different meaning, not only includes 3rd party accessory sites but also optional MSFT sites.

If you are ok with dropping *all* the optional sites, you can change the prototype of the Miner to drop all the indicators with o365_required attribute set to false. The O365 API Miners already expose that attribute in the indicators.