External Dynamic List: add manual entries?

[scottsander]

I'm using MineMeld to scrape Microsoft's website to grab the list of Office 365 related URLs and IP addresses.  We've noticed that this feeds in *.outlook.com, but not outlook.com.  This results in the common Office 365 Outlook Web App URL https://outlook.com/owa/<realm>.mail.onmicrosoft.com not matching the External Dynamic List ("EDL") generated by MineMeld and isn't matching our Office 365 security policy in the firewall.

I see that there is a Manual Exception list for an EDL.  The help documentation isn't completely clear whether or not this is a way to manually add entries to an EDL or to manually remove EDL entries pulled in automatically.  I believe it is the latter, but I tried adding outlook.com/ to the Office 365 URLs Manual Exception list, but visits still aren't matching my Office 365 security policy.

Is there a way I can manually add outlook.com/ to the Office 365 URL EDL generated by MineMeld?  Or any other clever suggestion?

[BPry]

@scottsander,

The Manual Exception list does exactly what you believe, this is to exclude entries already in a EDL from being utilized by the fiewall. 

I would personally add a miner that utilizes the prototype stdlib.listDomainGeneric so that you can manually add indicators and simply include the new miner in the Aggregate node that you are using for the Office output node. This way you can add indicators when you need to and it will be incorporated into your EDL. 

[scottsander]

[UPDATE]: The issue I discuss in this particular post is solved. The resolution is in the following post.

Hm. OK, would you mind providing some guidance?  I am a MineMeld novice and don't find the interface very intuitive.

I previously followed PAN's guide on setting up MineMeld for Office 365.  So, based on that, I believe I somehow need to feed the office365_URLaggreggator (prototype: stdlib.aggregatorURL) this new miner you suggested.  The trouble is, I seem to be having problems with getting it created correctly.  After I create the new stdlib.listDomainGeneric miner and Commit, I don't seem to be able to add it to the aggregator.

Here are the steps I'm following to create the new miner.

1. In MineMeld, go to Config.
2. Scroll down and press the hamburger icon (tooltip: browse prototypes).
3. Locate stdlib.listDomainGeneric and click it.
4. Press Clone.
5. Name the new node office365_additionalurls.
6. Press Commit.
7. Go to Nodes.
8. Click on office365_additionalurls.
9. Click the Indicators icon.
10. Press the + icon, enter outlook.com in the Indicator field, and change the Share Level to Green.

The problem comes in when I try to then add the new office365_additionalurls miner as an Input on the office365_URLaggregator.  I click the list of inputs on the aggregator and find that office365_additionalurls is not an available selection.

[scottsander · Accepted solution]

I figured it out.  I noticed that office365_URLaggregator is based on the prototype stdlib.aggregatorURL which seems to accept only inputs of type "URL" and the prototype stdlib.listDomainGeneric is type "Domain".  I found a prototype called stdlib.listURLGeneric and based my new miner on that instead and was able to add it to the aggregator and then verify it was showing up in the resulting EDL.

TL;DR:

I used the stdlib.listURLGeneric protoype instead of your suggested stdlib.listDomainGeneric.

Thanks for getting me pointed in a useful direction!

[mcragg]

Thanks for the post.  New to Minemeld but learnng as I go.  I have setup this miner, processor, output node connection.  Where/How would you manually enter your entries?