Miner to collect Microsoft Public IP space

[alyssonalmeida]

Hi Luigi,

I need to create a miner on MineMeld containing all Microsoft Public IP addresses.  Any plans to add a miner for it?
The URL source is https://www.microsoft.com/en-us/download/details.aspx?id=53602

The file is in CSV format. I noticed you created something similar for Azure IP addresses, but the file format for Azure is XML, not CSV.

I tried to create a new prototype but I couldn't make it work. 

I am now converting the CSV to TXT using a bash script and then posting the TXT to a webserver. 

 curl -X GET `curl https://www.microsoft.com/en-us/download/confirmation.aspx?id=53602 2>&1  | sed -n 's/.*href="\([^"]*\).*/\1/p' |grep -m1 msft-public-ips.csv`  |  cut -d, -f1 > msft-public-ips.txt

 Are you planning to add it?

Thanks,

[xhoms]

@alyssonalmeida : The best option for CSV formatted sources like the one you're referring to, is to create a prototype for the minemeld.CSVFT class. You can, for instance, use the sslabusech.ipblacklist prototype from the standard library as a base and create a new prototype with the following content. It will do the job for you right away.

attributes:
    confidence: 100
    share_level: green
    type: IPv4
fieldnames:
	- indicator
	- service_type
ignore_regex: ^((Prefix)|([a-z0-9]{4}:))
source_name: microsoft_public_ip
url: https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

[alyssonalmeida]

The issue with this approach is that the link you mentioned here is going to change as soon as they update the file.

https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

 So, what you suggested is going to work, but it will not get any updates from Microsoft.

The only static link is the one I posted here.

https://www.microsoft.com/en-us/download/details.aspx?id=53602

[xhoms]

@alyssonalmeida : I'm not a MSFT content expert at all. Just made a couple of checks to realize that, aparently:

• https://www.microsoft.com/en-us/download/details.aspx?id=53602 has a static reference to
  • HREF "confirmation.aspx?id=53602"
• "confirmation.aspx?id=53602" has a static reference to
  • https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.c...

Obviously, these apparent static references might be constructed at runtime by JavaScript DOM functions. Just wanted to let you know how I reached to the final URL.

[xhoms]

Hi @alyssonalmeida,

I took some time to create a Miner to process CSV files referenced in Microsoft Articles. The extension at https://github.com/xhoms/minemeld-msarticle provides you with a prototype called msarticle.msPublicSpace that fulfills your need.

[Alex_Samad]

Hi

Thanks for this. I have added you extension in, but I am unsure how to actually use it to create a minor for this 

can you give an example

[xhoms]

@Alex_Samad,

once you install the extension (and activate it) a new Prototype called "msarticle.msPublicSpace" will be added to your MineMeld's instance library.

Find it clicking on the right/bottom icon on the Configuration Pane.

Then clone the prototype as a new working node

After commiting the changes you should see the new miner grabbing indicators from the article.

[Alex_Samad]

Hi

Thanks, yeah I did some more digging around got it working.

Any reason that the source isn't marked as high confidence ?

[xhoms]

Hi @Alex_Samad,

no reason at all I just forgot to add the `confidence` configuration attribute into the published prototype. Will update the extension in a few moments.

Thank you very much for sharing your findings.

[Alex_Samad]

Np 

thanks for the plugin

[xhoms]

It is done. Version 0.1.1 of the extension includes the modification in the prototype.

If you want to avoid going through an extension re-installation process just modify the file at /opt/minemeld/local/library/<uuid4>/MSArticle/prototypes/msarticle.yml and add the confidence attribute at the bottom.

      attributes:
        share_level: green
        confidence: 100

[MAROUAN]

Hello, 

I really apperciate you help, i follow the steps but it's nor working for me , i don't know why.

could you help me please.

best regards