Miner to collect Microsoft Public IP space

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Miner to collect Microsoft Public IP space

L2 Linker

Hi Luigi,

 

I need to create a miner on MineMeld containing all Microsoft Public IP addresses.  Any plans to add a miner for it?
The URL source is https://www.microsoft.com/en-us/download/details.aspx?id=53602


The file is in CSV format. I noticed you created something similar for Azure IP addresses, but the file format for Azure is XML, not CSV.


I tried to create a new prototype but I couldn't make it work. 

I am now converting the CSV to TXT using a bash script and then posting the TXT to a webserver. 

 curl -X GET `curl https://www.microsoft.com/en-us/download/confirmation.aspx?id=53602 2>&1  | sed -n 's/.*href="\([^"]*\).*/\1/p' |grep -m1 msft-public-ips.csv`  |  cut -d, -f1 > msft-public-ips.txt

 

 Are you planning to add it?

Thanks,

 

11 REPLIES 11

L5 Sessionator

@alyssonalmeida : The best option for CSV formatted sources like the one you're referring to, is to create a prototype for the minemeld.CSVFT class. You can, for instance, use the sslabusech.ipblacklist prototype from the standard library as a base and create a new prototype with the following content. It will do the job for you right away.

 

attributes:
    confidence: 100
    share_level: green
    type: IPv4
fieldnames:
	- indicator
	- service_type
ignore_regex: ^((Prefix)|([a-z0-9]{4}:))
source_name: microsoft_public_ip
url: https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

The issue with this approach is that the link you mentioned here is going to change as soon as they update the file.

 

 

 

https://download.microsoft.com/download/B/2/A/B2AB28E1-DAE1-44E8-A867-4987FE089EBE/msft-public-ips.csv

 So, what you suggested is going to work, but it will not get any updates from Microsoft.

 

The only static link is the one I posted here.

https://www.microsoft.com/en-us/download/details.aspx?id=53602

@alyssonalmeida : I'm not a MSFT content expert at all. Just made a couple of checks to realize that, aparently:

Obviously, these apparent static references might be constructed at runtime by JavaScript DOM functions. Just wanted to let you know how I reached to the final URL.

Hi @alyssonalmeida,

 

I took some time to create a Miner to process CSV files referenced in Microsoft Articles. The extension at https://github.com/xhoms/minemeld-msarticle provides you with a prototype called msarticle.msPublicSpace that fulfills your need.

 

 

Hi

 

Thanks for this. I have added you extension in, but I am unsure how to actually use it to create a minor for this 

 

can you give an example

 

@Alex_Samad,

 

once you install the extension (and activate it) a new Prototype called "msarticle.msPublicSpace" will be added to your MineMeld's instance library.

 

Find it clicking on the right/bottom icon on the Configuration Pane.2018-07-31_10-40-24.png

 

 

 

 

Then clone the prototype as a new working node2018-07-31_10-41-07.png2018-07-31_10-41-59.png

 

After commiting the changes you should see the new miner grabbing indicators from the article.2018-07-31_10-57-17.png

 

 

 

Hi

 

Thanks, yeah I did some more digging around got it working.

 

Any reason that the source isn't marked as high confidence ?

Hi @Alex_Samad,

 

no reason at all I just forgot to add the `confidence` configuration attribute into the published prototype. Will update the extension in a few moments.

 

Thank you very much for sharing your findings.

Np 

 

thanks for the plugin

It is done. Version 0.1.1 of the extension includes the modification in the prototype.

 

If you want to avoid going through an extension re-installation process just modify the file at /opt/minemeld/local/library/<uuid4>/MSArticle/prototypes/msarticle.yml and add the confidence attribute at the bottom.

 

      attributes:
        share_level: green
        confidence: 100

Hello, 

I really apperciate you help, i follow the steps but it's nor working for me , i don't know why.

 

could you help me please.

 

best regards 

  • 12396 Views
  • 11 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!