topic Re: Replacing office 365 from XML and RSS with Restful API in General Topics

Replacing office 365 from XML and RSS with Restful API

chtoh82 — Wed, 02 May 2018 09:05:59 GMT

Hi all,

In view of the changes Microsoft is going to make in future as describe in the following link, would the current miner for O365 still works?

https://support.office.com/en-gb/article/managing-office-365-endpoints-99cab9d4-ef59-4207-9f2b-3728eb46bf9a?ui=en-US&rs=en-GB&ad=GB#ID0EADAAA=4._Web_service&ID0EACAAA=4._Web_service

Re: Replacing office 365 from XML and RSS with Restful API

lmori — Fri, 04 May 2018 08:35:38 GMT

Hi @chtoh82,

the new MSFT API is not production-ready yet. But we have already implemented the Miner for it:

https://github.com/PaloAltoNetworks/minemeld-core/issues/267

We will include it in a new release early next week.

luigi

Re: Replacing office 365 from XML and RSS with Restful API

chtoh82 — Fri, 04 May 2018 09:00:20 GMT

Thanks! Looking forward to it!

Re: Replacing office 365 from XML and RSS with Restful API

KengSeng — Thu, 09 Aug 2018 08:58:23 GMT

Hi Luigi, as i can see the prototype for those miners based on China, Gemany, US and worldwide are still in experimental. May I know whether we already mined and feeding those IP and domains to the O365 output? Thanks.

Re: Replacing office 365 from XML and RSS with Restful API

Fille01 — Thu, 16 Aug 2018 11:30:26 GMT

I tested the O365-API worldwide miner on the AF hosted minemeld:

# HTTPError: 400 Client Error: Bad Request

2018-08-16T11:00:52 (12932)basepoller.hup INFO: O365-Worldwide-Any-Service - hup received, force polling
2018-08-16T11:00:52 (12932)basepoller._huppable_wait INFO: hup is clear: False
2018-08-16T11:00:52 (12932)basepoller._actor_loop INFO: O365-Worldwide-Any-Service - command: 1534417252635 poll
2018-08-16T11:00:52 (12932)basepoller._polling_loop INFO: Polling O365-Worldwide-Any-Service
2018-08-16T11:00:52 (12932)connectionpool._new_conn INFO: Starting new HTTPS connection (1): endpoints.office.com
2018-08-16T11:00:52 (12932)basepoller._poll ERROR: Exception in polling loop for O365-Worldwide-Any-Service: 400 Client Error: Bad Request
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.46/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 721, in _poll
performed = self._polling_loop()
File "/opt/minemeld/engine/0.9.46/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 571, in _polling_loop
iterator = self._build_iterator(now)
File "/opt/minemeld/engine/0.9.46/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 352, in _build_iterator
latest_version = self._check_version()
File "/opt/minemeld/engine/0.9.46/local/lib/python2.7/site-packages/minemeld/ft/o365.py", line 252, in _check_version
r.raise_for_status()
File "/opt/minemeld/engine/0.9.46/local/lib/python2.7/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 400 Client Error: Bad Request
2018-08-16T11:00:55 (12932)basepoller._polling_loop INFO: Polling O365-Worldwide-Any-Service
2018-08-16T11:00:55 (12932)connectionpool._new_conn INFO: Starting new HTTPS connection (1): endpoints.office.com
2018-08-16T11:00:55 (12932)basepoller._poll ERROR: Exception in polling loop for O365-Worldwide-Any-Service: 400 Client Error: Bad Request

Before posting this I tried China too, no difference. They all error out on the same thing.

Re: Replacing office 365 from XML and RSS with Restful API

fwmike — Tue, 25 Sep 2018 14:39:02 GMT

Same behavior on my MM.

Re: Replacing office 365 from XML and RSS with Restful API

xhoms — Tue, 25 Sep 2018 14:42:07 GMT

Hi @Fille01,

new O365 API Miners were officialy released with MineMeld version 0.9.50. Please, update your MineMeld instance before using the provided configuration files.

Re: Replacing office 365 from XML and RSS with Restful API

xhoms — Tue, 25 Sep 2018 14:43:08 GMT

Hi @fwmike,

( see https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Replacing-office-365-from-XML-and-RSS-with-Restful-API/m-p/232277/highlight/true#M2570 )

Re: Replacing office 365 from XML and RSS with Restful API

Fille01 — Wed, 26 Sep 2018 07:15:43 GMT

Hello,

I used the AF hosted minemeld. However I changed to hosting one myself and that was updated, works OK.

Re: Replacing office 365 from XML and RSS with Restful API

RamBalaji — Wed, 26 Sep 2018 13:40:27 GMT

I also caught up with this 400 client error:bad request after replacing the scripts. I couldn't find version upgrade option in autofocus minemeld, can you share the link.

Re: Replacing office 365 from XML and RSS with Restful API

Fille01 — Wed, 26 Sep 2018 13:42:31 GMT

@RamBalaji wrote:

I also caught up with this 400 client error:bad request after replacing the scripts. I couldn't find version upgrade option in autofocus minemeld, can you share the link.

I am pretty sure there is none.

Re: Replacing office 365 from XML and RSS with Restful API

lmori — Wed, 26 Sep 2018 14:21:50 GMT

Hi @RamBalaji,

which version of MineMeld for Autofocus are you running? Is it possible that you are running on version 0.9.46?

If that's the case you can upgrade MineMeld by going to Apps, stop MineMeld and restart it. DO NOT RESET!!

It's always better to do a full backup of MineMeld before doing a restart (under the tab SYSTEM).

Re: Replacing office 365 from XML and RSS with Restful API

Fille01 — Thu, 27 Sep 2018 06:59:26 GMT

I can confirm that this solution works! It upgraded to 0.9.50 and o365 miner works.