Miner to collect Microsoft Public IP space

Reply
L2 Linker

Miner to collect Microsoft Public IP space

Hi Luigi,

 

I need to create a miner on MineMeld containing all Microsoft Public IP addresses.  Any plans to add a miner for it?
The URL source is https://www.microsoft.com/en-us/download/details.aspx?id=53602


The file is in CSV format. I noticed you created something similar for Azure IP addresses, but the file format for Azure is XML, not CSV.


I tried to create a new prototype but I couldn't make it work. 

I am now converting the CSV to TXT using a bash script and then posting the TXT to a webserver. 

 curl -X GET `curl https://www.microsoft.com/en-us/download/confirmation.aspx?id=53602 2>&1  | sed -n 's/.*href="\([^"]*\).*/\1/p' |grep -m1 msft-public-ips.csv`  |  cut -d, -f1 > msft-public-ips.txt

 

 Are you planning to add it?

Thanks,

 

L5 Sessionator

Re: Miner to collect Microsoft Public IP space

@alyssonalmeida : The best option for CSV formatted sources like the one you're referring to, is to create a prototype for the minemeld.CSVFT class. You can, for instance, use the sslabusech.ipblacklist prototype from the standard library as a base and create a new prototype with the following content. It will do the job for you right away.

 

attributes:
    confidence: 100
    share_level: green
    type: IPv4
fieldnames:
	- indicator
	- service_type
ignore_regex: ^((Prefix)|([a-z0-9]{4}:))
source_name: microsoft_public_ip
url: https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv
L2 Linker

Re: Miner to collect Microsoft Public IP space

The issue with this approach is that the link you mentioned here is going to change as soon as they update the file.

 

 

 

https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

 So, what you suggested is going to work, but it will not get any updates from Microsoft.

 

The only static link is the one I posted here.

https://www.microsoft.com/en-us/download/details.aspx?id=53602

L5 Sessionator

Re: Miner to collect Microsoft Public IP space

@alyssonalmeida : I'm not a MSFT content expert at all. Just made a couple of checks to realize that, aparently:

Obviously, these apparent static references might be constructed at runtime by JavaScript DOM functions. Just wanted to let you know how I reached to the final URL.

L5 Sessionator

Re: Miner to collect Microsoft Public IP space

Hi @alyssonalmeida,

 

I took some time to create a Miner to process CSV files referenced in Microsoft Articles. The extension at https://github.com/xhoms/minemeld-msarticle provides you with a prototype called msarticle.msPublicSpace that fulfills your need.

 

 

L4 Transporter

Re: Miner to collect Microsoft Public IP space

Hi

 

Thanks for this. I have added you extension in, but I am unsure how to actually use it to create a minor for this 

 

can you give an example

 

L5 Sessionator

Re: Miner to collect Microsoft Public IP space

@Alex_Samad,

 

once you install the extension (and activate it) a new Prototype called "msarticle.msPublicSpace" will be added to your MineMeld's instance library.

 

Find it clicking on the right/bottom icon on the Configuration Pane.2018-07-31_10-40-24.png

 

 

 

 

Then clone the prototype as a new working node2018-07-31_10-41-07.png2018-07-31_10-41-59.png

 

After commiting the changes you should see the new miner grabbing indicators from the article.2018-07-31_10-57-17.png

 

 

 

L4 Transporter

Re: Miner to collect Microsoft Public IP space

Hi

 

Thanks, yeah I did some more digging around got it working.

 

Any reason that the source isn't marked as high confidence ?

L5 Sessionator

Re: Miner to collect Microsoft Public IP space

Hi @Alex_Samad,

 

no reason at all I just forgot to add the `confidence` configuration attribute into the published prototype. Will update the extension in a few moments.

 

Thank you very much for sharing your findings.

L4 Transporter

Re: Miner to collect Microsoft Public IP space

Np 

 

thanks for the plugin

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!