Create new prototype for Rest API and simple URL

[jilim]

Hi Experts,

I have a customer who wants to create new prototype for this customer. 

customer requirement is very simple but, it's very hard to me. 

first of one,

customer said spunk is using Rest API, below is feeds from splunk 

curl -k https://splunk_IP_address/services/search/jobs/export -d "search=| inputlookup autofocus_lookup" -d output_mode=xml -u test:'test_poc!!@@##'

Second of one, 

Value of DNS TTL in below URL is within 5min. So, customer would like to polling DNS IP address resolve every 1min. 

image.watchon.cjem.skcdn.com

image.ytn.co.kr 

Java-buildpack.cloudfoundry.org

download.run.pivotal.io

bitbucket.oksusu.com

photo.jtbc.joins.com

I know how to create new simple miner like below URL

https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner#the-node

but, I really don't know What prototype I should use and customazing. I'm not a used develope code.

Please guide me how to create prototype for this customer and will be appreciated.

Thanks

Jihoon

[xhoms]

Hi @jilim,

first one: just share with us an example of the output provided by the curl command and we'll be able to figure out the prototype needed to that feed.

second one: it doen't look like a good idea to try to get IP addresses from reverse DNS queries. You should ask these providers for an API MineMeld could connect to to download the current IP addresses for their services.

[jilim]

Hi Xhoms, 

I sent a curl command to your e-mail and, for second one why customer wants reserve DNS,

these URL is public URL and, when connect to these URL, Domain IP address is keeping change because of CDN in 1min or 2mins. 

So, to use our URL filtering, there is challange becasue minimum value of  IP resolve cache is 10mins.( there is around 8~9mins gap)

So, to use miner of minemeld, miner is polling Domain IP address to these URL every 1min and, our firewall connects minemeld by EDL.

This idea is  from our minemeld can get IP address for O365.

Thanks

Jihoon

[jilim]

as below, IP address is keep change.  So, customer wants to allow this URL only "image.watchon.cjem.skcdn.com" .

round-trip min/avg/max/stddev = 4.087/6.010/8.045/1.288 ms

SINMACF079HTDH:~ jilim$ ping image.watchon.cjem.skcdn.com

PING image.watchon.cjem.skcdn.com (211.110.212.66): 56 data bytes

64 bytes from 211.110.212.66: icmp_seq=0 ttl=57 time=8.566 ms

64 bytes from 211.110.212.66: icmp_seq=1 ttl=57 time=5.382 ms

64 bytes from 211.110.212.66: icmp_seq=2 ttl=57 time=7.669 ms

64 bytes from 211.110.212.66: icmp_seq=3 ttl=57 time=6.834 ms

^C

--- image.watchon.cjem.skcdn.com ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 5.382/7.113/8.566/1.172 ms

SINMACF079HTDH:~ jilim$ ping image.watchon.cjem.skcdn.com

PING image.watchon.cjem.skcdn.com (121.156.105.170): 56 data bytes

64 bytes from 121.156.105.170: icmp_seq=0 ttl=53 time=6.666 ms

64 bytes from 121.156.105.170: icmp_seq=1 ttl=53 time=8.423 ms

64 bytes from 121.156.105.170: icmp_seq=2 ttl=53 time=10.876 ms

^C

--- image.watchon.cjem.skcdn.com ping statistics ---

3 packets transmitted, 3 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 6.666/8.655/10.876/1.727 ms

SINMACF079HTDH:~ jilim$ ping image.watchon.cjem.skcdn.com

PING image.watchon.cjem.skcdn.com (121.156.105.170): 56 data bytes

64 bytes from 121.156.105.170: icmp_seq=0 ttl=53 time=5.745 ms

64 bytes from 121.156.105.170: icmp_seq=1 ttl=53 time=7.923 ms

64 bytes from 121.156.105.170: icmp_seq=2 ttl=53 time=24.452 ms

64 bytes from 121.156.105.170: icmp_seq=3 ttl=53 time=4.846 ms

^C

--- image.watchon.cjem.skcdn.com ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 4.846/10.742/24.452/7.994 ms

SINMACF079HTDH:~ jilim$ ping image.watchon.cjem.skcdn.com

PING image.watchon.cjem.skcdn.com (110.45.211.29): 56 data bytes

64 bytes from 110.45.211.29: icmp_seq=0 ttl=54 time=5.840 ms

64 bytes from 110.45.211.29: icmp_seq=1 ttl=54 time=4.893 ms

64 bytes from 110.45.211.29: icmp_seq=2 ttl=54 time=8.397 ms

^C

--- image.watchon.cjem.skcdn.com ping statistics ---

3 packets transmitted, 3 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 4.893/6.377/8.397/1.480 ms

SINMACF079HTDH:~ jilim$

SINMACF079HTDH:~ jilim$