MineMeld age_out not withdrawing ips

[PF]

I'm very new to MineMeld, and I am having issues withdrawing ip addresses from a list. 

The miner checks a local list, and the list has two ips in it currently. I'd like the ips to be age_out after 24 hours, even if they are still on the local list. 

In the logs I see TRACE / EMIT_WITHDRAW with the indicator of the ip, but then the very next log is TRACE / EMIT_UPDATE with the indicator of the ip, and the ip is never removed from the minemeld output. The miner says added 5 and removed 3, but the local list has been static. What am I missing? Thanks!

[lmori]

Hi @PF,

age out depends on the config and the type of output feeds. Example: standard feeds (stdlib.feed*) immediately remove expired indicators while other like taxiiDataFeed do not because their logic is different.

Could you share your config from CONFIG > EXPORT ? I can give you more details about the expected behavior.

[PF]

Thanks for getting back to me

nodes:
bunker_aggregator:
inputs:
- Bunker
output: true
prototype: stdlib.aggregatorIPv4Generic
Bunker:
inputs: []
output: true
prototype: minemeldlocal.bunker_banlist
bunker-output:
inputs:
- Bunker
output: false
prototype: stdlib.feedHCGreenWithValue

[lmori]

Hi @PF,

could you share more details about the minemeld.bunker_banlist prototype ? like class and full config ?

Thanks,

luigi

[PF]

--class--

minemeld.ft.http.HttpFT

--config--
age_out
default: first_seen+1d
interval: 1800
sudden_death: true
attributes
confidence: 100
direction: inbound
share_level: green
type: IPv4
ignore_regex ^#.*
interval 60
source_name bunker.banlist
url http://ip-address/banlist.txt

[lmori]

Hi @PF,

this is a bug, and I have already a fix for it. Would you be interested in testing the beta with the fix ?

luigi

[PF]

sure

[PF]

@lmori, Whats the process for testing the beta fix? I'm willing to give it a go. 

[lmori]

Hi @PF,

if you have installed MM from binaries (via OVA, CFN, AFM, ISO, apt repos, ...) you should subscribe your MM instance to the beta channel. Change the file /etc/minemeld-auto-updates.conf to this (basically change the value of "channel"):

{
  "minemeld-updates": {
    "baseurl": "http://minemeld-updates.panw.io/stage2",
    "channel": ["0_9", "beta0_9"]
  }
}

After that, force an update:

$ sudo -u minemeld /usr/sbin/minemeld-auto-update

[PF]

I changed the auto-update.conf and run the update command, but get this..

minemeld:/etc$ sudo -u minemeld /usr/sbin/minemeld-auto-update
Traceback (most recent call last):
File "/usr/sbin/minemeld-auto-update", line 787, in <module>
main()
File "/usr/sbin/minemeld-auto-update", line 738, in main
update_minemeld_package()
File "/usr/sbin/minemeld-auto-update", line 687, in update_minemeld_package
cache.update()
File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 418, in update
raise LockFailedException("Failed to lock %s" % lockfile)
apt.cache.LockFailedException: Failed to lock /var/lib/apt/lists/lock

[lmori]

Hi @PF,

most probably you have a process working on the apt database. See here: https://askubuntu.com/questions/335794/could-not-get-lock-var-lib-apt-lists-lock

Thanks,

luigi 

[akapucu]

Hi,

Can i drop IOCs from taxii feed after 90 days?

My source(soltra) is not providing that option i am hoping minemeld can do this?

[xhoms]

Hi @akapucu,

yes. It can. You just have to adjust the aging policy of your miner. Details at https://live.paloaltonetworks.com/t5/MineMeld-Articles/Configuring-nodes/ta-p/77185.

The "TaxiiClient" class extends the "BasePollerFT" which means it inherits all its capabilities including the indicators aging out engine.