topic Re: Miner to collect Microsoft Public IP space in General Topics

Miner to collect Microsoft Public IP space

alyssonalmeida — Sun, 12 Nov 2017 19:37:49 GMT

Hi Luigi,

I need to create a miner on MineMeld containing all Microsoft Public IP addresses. Any plans to add a miner for it?
The URL source is https://www.microsoft.com/en-us/download/details.aspx?id=53602

The file is in CSV format. I noticed you created something similar for Azure IP addresses, but the file format for Azure is XML, not CSV.

I tried to create a new prototype but I couldn't make it work.

I am now converting the CSV to TXT using a bash script and then posting the TXT to a webserver.

curl -X GET `curl https://www.microsoft.com/en-us/download/confirmation.aspx?id=53602 2>&1 | sed -n 's/.*href="\([^"]*\).*/\1/p' |grep -m1 msft-public-ips.csv` | cut -d, -f1 > msft-public-ips.txt

Are you planning to add it?

Thanks,

Re: Miner to collect Microsoft Public IP space

xhoms — Sun, 12 Nov 2017 20:57:13 GMT

@alyssonalmeida : The best option for CSV formatted sources like the one you're referring to, is to create a prototype for the minemeld.CSVFT class. You can, for instance, use the sslabusech.ipblacklist prototype from the standard library as a base and create a new prototype with the following content. It will do the job for you right away.

attributes:
    confidence: 100
    share_level: green
    type: IPv4
fieldnames:
	- indicator
	- service_type
ignore_regex: ^((Prefix)|([a-z0-9]{4}:))
source_name: microsoft_public_ip
url: https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

Re: Miner to collect Microsoft Public IP space

alyssonalmeida — Sun, 12 Nov 2017 21:02:24 GMT

The issue with this approach is that the link you mentioned here is going to change as soon as they update the file.

https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

So, what you suggested is going to work, but it will not get any updates from Microsoft.

The only static link is the one I posted here.

https://www.microsoft.com/en-us/download/details.aspx?id=53602

Re: Miner to collect Microsoft Public IP space

xhoms — Mon, 13 Nov 2017 06:32:17 GMT

@alyssonalmeida : I'm not a MSFT content expert at all. Just made a couple of checks to realize that, aparently:

https://www.microsoft.com/en-us/download/details.aspx?id=53602 has a static reference to
- HREF "confirmation.aspx?id=53602"
"confirmation.aspx?id=53602" has a static reference to
- https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

Obviously, these apparent static references might be constructed at runtime by JavaScript DOM functions. Just wanted to let you know how I reached to the final URL.

Re: Miner to collect Microsoft Public IP space

xhoms — Wed, 17 Jan 2018 19:02:33 GMT

Hi @alyssonalmeida,

I took some time to create a Miner to process CSV files referenced in Microsoft Articles. The extension at https://github.com/xhoms/minemeld-msarticle provides you with a prototype called msarticle.msPublicSpace that fulfills your need.

Re: Miner to collect Microsoft Public IP space

Alex_Samad — Thu, 26 Jul 2018 08:43:09 GMT

Thanks for this. I have added you extension in, but I am unsure how to actually use it to create a minor for this

can you give an example

Re: Miner to collect Microsoft Public IP space

xhoms — Tue, 31 Jul 2018 09:02:12 GMT

@Alex_Samad,

once you install the extension (and activate it) a new Prototype called "msarticle.msPublicSpace" will be added to your MineMeld's instance library.

Find it clicking on the right/bottom icon on the Configuration Pane.

Then clone the prototype as a new working node

After commiting the changes you should see the new miner grabbing indicators from the article.

Re: Miner to collect Microsoft Public IP space

Alex_Samad — Tue, 31 Jul 2018 22:19:36 GMT

Thanks, yeah I did some more digging around got it working.

Any reason that the source isn't marked as high confidence ?

Re: Miner to collect Microsoft Public IP space

xhoms — Thu, 02 Aug 2018 07:04:29 GMT

Hi @Alex_Samad,

no reason at all I just forgot to add the `confidence` configuration attribute into the published prototype. Will update the extension in a few moments.

Thank you very much for sharing your findings.

Re: Miner to collect Microsoft Public IP space

Alex_Samad — Thu, 02 Aug 2018 07:08:49 GMT

thanks for the plugin

Re: Miner to collect Microsoft Public IP space

xhoms — Thu, 02 Aug 2018 07:29:47 GMT

It is done. Version 0.1.1 of the extension includes the modification in the prototype.

If you want to avoid going through an extension re-installation process just modify the file at /opt/minemeld/local/library/<uuid4>/MSArticle/prototypes/msarticle.yml and add the confidence attribute at the bottom.

      attributes:
        share_level: green
        confidence: 100

Re: Miner to collect Microsoft Public IP space

MAROUAN — Tue, 31 Dec 2019 20:41:43 GMT

Hello,

I really apperciate you help, i follow the steps but it's nor working for me , i don't know why.

could you help me please.

best regards