topic Re: Mining Domains from mixed IP/Domain List in General Topics

Mining Domains from mixed IP/Domain List

danilo.souza — Thu, 09 Aug 2018 12:12:07 GMT

Hi guys,

I would appreciate some help withe this issue. I wanto to mine domains from a list that has IP/Domais mixed. The list has on the top a short description whose lines start with the "#" character. Thats fine, I can use the "ignore_regex ^#" to avoid it. But the list is composed by:

10.200.1.36 domain1.com

172.31.31.31 domain2.com

192.168.0.55 domain3.com

10.10.10.11 domain_n.com

How do I avoid the IPs and get only the domains? I belive it is not so difficult. I already got the IPs using "indicator regex: ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" .

Thank you in advanced.

Best regards.

Re: Mining Domains from mixed IP/Domain List

lmori — Fri, 10 Aug 2018 15:24:35 GMT

Hi @danilo.souza,

simple solution: mine the feed twice, once for the IPs and once for the domains 🙂

Re: Mining Domains from mixed IP/Domain List

danilo.souza — Fri, 10 Aug 2018 16:18:05 GMT

Hi @lmori

thank you for the reply. Sure, that is what I'm doing. The point is: the correct "regular expression" to get the domains (beginning include with numbers) without the space " " between the IPs and the domains itself. Until now, I was only able to get the domains including the space " ". Does the feed node know how to deal with it? The space " " at the begin is eliminated by him when producing the link to be ingested by PANOS in an EDL?

Thank you in advanced.

Best regards.

Re: Mining Domains from mixed IP/Domain List

lmori — Mon, 13 Aug 2018 19:51:16 GMT

Hi @danilo.souza,

you should be able to use the following to get the domains:

indicator:
  regex: '[\w\.]+\s(.*)'
  transform: '\1'

Re: Mining Domains from mixed IP/Domain List

chirss — Thu, 16 Aug 2018 16:11:11 GMT

It's a bit more complicated, but maybe this can help as well?

^((?:((\d{1,3}.){3}\d{1,3}\s))(.*))$

I haven't tested it yet since I don't have a feed like this, but it seems to match and non-capture group on what it needs to.