ZPA Minemeld feed from json source truncated to last record

[SlowTypist]

Problem Summary:

Trying to locally convey - as a feed - all subnet block ranges from https://ips.zscaler.net/zpa/json - but only getting the last presented.

URL Being referenced: https://ips.zscaler.net/zpa/json

Example Content:

{"Cloud Name":"zscaler.net","Content":[{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["8.25.203.0/24","8.34.34.0/24","8.35.35.0/24","52.18.93.240/32","52.19.38.71/32","52.197.70.230/32","52.198.19.112/32","52.198.72.244/32","52.207.198.29/32","52.209.45.220/32","52.210.11.225/32","52.220.100.223/32","52.220.100.69/32","52.220.99.252/32","52.24.149.190/32","52.25.2.198/32","52.28.207.67/32","52.28.37.10/32","52.29.240.114/32","52.29.98.93/32","52.33.154.59/32","52.4.154.137/32","52.5.144.98/32","52.52.92.202/32","52.52.95.220/32","52.52.95.235/32","52.52.96.24/32","52.58.125.47/32","52.58.78.135/32","52.63.157.237/32","52.63.158.184/32","52.63.58.54/32","52.65.142.146/32","52.65.152.196/32","52.65.40.115/32","52.66.115.172/32","52.66.116.178/32","52.66.123.138/32","52.66.51.4/32","52.67.117.30/32","52.67.117.80/32","52.67.78.111/32","52.67.87.60/32","52.68.138.157/32","52.68.4.241/32","52.69.146.228/32","52.74.48.141/32","52.74.58.135/32","52.74.92.94/32","52.78.59.243/32","52.78.73.223/32","52.78.79.105/32","52.78.81.101/32","52.79.50.105/32","52.79.52.245/32","52.8.120.78/32","52.8.174.227/32","52.88.221.173/32","52.89.25.231/32","52.89.62.191/32","52.89.62.191/32","54.154.100.194/32","54.154.100.215/32","54.86.169.181/32","54.87.158.111/32","72.37.140.0/24","89.167.129.0/24","89.191.7.16/28","94.188.139.64/26","94.188.248.64/26","104.129.192.0/20","128.177.125.0/24","128.177.129.0/24","128.177.135.0/24","128.177.136.0/24","165.225.0.0/17","165.225.192.0/18","165.225.36.0/23","185.46.212.0/22","185.46.212.0/23","185.46.214.0/23","188.116.35.32/28","199.168.148.0/22","209.51.184.0/26","213.152.228.0/24","216.66.5.0/24"],"Date Added":"Initial Publication"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["13.59.180.7/32","13.59.141.201/32","13.59.14.90/32","13.58.243.5/32","35.182.57.197/32","35.182.72.155/32","35.182.41.239/32","35.182.113.223/32","35.176.70.72/32","35.176.178.43/32","35.176.174.248/32","35.176.170.178/32","13.64.250.38/32","40.86.176.165/32","40.86.182.64/32","40.86.183.239/32","104.45.131.108/32","104.45.128.192/32","104.45.151.52/32","104.45.148.204/32","52.169.125.252/32","13.74.157.78/32","13.79.33.253/32","40.113.92.79/32","40.68.30.189/32","23.101.72.77/32","40.68.25.125/32","23.100.7.240/32","52.175.24.162/32","52.175.26.143/32","52.175.30.139/32","52.175.29.8/32","52.187.19.12/32","52.187.23.160/32","52.187.17.199/32","52.187.66.156/32","52.240.159.223/32","52.240.157.136/32","52.240.154.114/32","52.240.155.200/32","13.65.36.86/32","13.85.19.207/32","13.65.33.5/32","13.85.78.38/32","52.173.149.37/32","52.165.218.125/32","52.173.147.246/32","52.165.216.94/32","40.84.53.118/32","13.77.82.151/32","13.77.86.84/32","13.77.82.96/32","13.71.158.244/32","13.73.1.205/32","13.78.126.65/32","13.71.159.30/32","104.215.27.73/32","104.215.31.13/32","104.215.26.115/32","104.215.26.249/32","104.41.24.112/32","104.41.26.126/32","104.41.27.137/32","104.41.31.133/32","13.75.143.33/32","13.75.136.115/32","13.75.137.223/32","13.75.143.22/32","13.70.159.20/32","13.77.5.206/32","13.70.184.227/32","13.77.7.178/32","52.172.216.84/32","52.172.209.202/32","52.172.209.243/32","52.172.209.244/32","13.71.121.83/32","52.172.50.146/32","52.172.54.58/32","52.172.53.133/32","104.211.186.221/32","104.211.187.48/32","104.211.188.142/32","104.211.188.122/32","52.237.19.166/32","52.237.21.25/32","52.233.42.219/32","52.237.30.86/32","52.242.19.28/32","52.235.43.198/32","52.235.43.151/32","52.235.43.152/32","52.161.100.200/32","52.161.97.167/32","52.161.99.87/32","52.161.97.78/32","52.183.125.224/32","52.175.255.83/32","52.229.39.139/32","52.175.208.105/32","51.141.55.81/32","51.141.42.174/32","51.141.46.82/32","51.141.43.174/32","51.140.74.255/32","51.140.122.102/32","51.140.125.127/32","51.140.114.120/32","52.231.27.82/32","52.231.26.225/32","52.231.25.14/32","52.231.34.139/32","52.231.204.27/32","52.231.201.255/32","52.231.206.16/32","52.231.202.42/32"],"Date Added":"September 2017"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["13.127.148.174/32","13.127.212.107/32","13.127.26.17/32","13.127.99.160/32","18.195.128.118/32","18.197.86.201/32","18.216.119.57/32","18.216.189.99/32","18.218.12.27/32","18.218.255.136/32","18.219.166.28/32","18.219.20.193/32","35.154.244.217/32","52.193.218.29/32","52.21.189.133/32","52.29.32.101/32","52.30.84.113/32","52.57.178.48/32","52.57.7.227/32","52.58.125.47/32","52.58.193.16/32","52.58.74.51/32","52.59.55.235/32","52.6.210.8/32","52.63.135.169/32","52.66.161.176/32","52.76.31.172/32","52.78.18.147/32","52.79.166.240/32","52.79.199.218/32","54.154.61.187/32"],"Date Added":"April 2018"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["35.180.108.229/32","35.180.12.19/32","35.180.16.134/32","35.180.49.249/32","35.180.59.62/32","35.180.59.240/32","52.47.53.30/32","52.47.207.196/32","52.47.104.132/32","52.47.109.64/32"],"Date Added":"June 2018"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["58.220.95.0/24","54.200.239.74/32","54.201.110.181/32","54.201.127.141/32","54.201.165.179/32","54.201.165.199/32","54.201.165.200/32","54.201.92.80/32"],"Date Added":"September 2018"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["13.53.102.181/32","13.53.105.156/32","13.53.115.185/32","13.53.120.157/32","13.53.141.39/32","13.53.160.23/32","13.53.163.129/32","13.53.167.43/32","13.53.58.60/32","13.53.88.9/32","54.219.164.222/32"],"Date Added":"January 2019"},{"IP Protocol":"TCP","Port":443,"Source":"Connector, Zscaler App","Domains":"*.prod.zpath.net,*.private.zscaler.com","IPs":["137.83.128.0/18","211.144.19.123/32","211.144.19.124/32","211.144.19.125/32","211.144.19.126/32"],"Date Added":"Feburary 2019"}]}

What we want to get?

List of all IP address ranges - eg.

192.168.1.0/24
172.16.2.0/24

To become something like....

192.168.1.1 - 192.168.1.254
172.16.2.1 - 172.16.2.254

ie. all subnet ranges within Content[].IPs[] ranges of the json input.

What was done?

Step 1: Created Inital Prototype

- Started with copy of "itcertpa.IP"
- Clicked New
- Details:

Name = minemeldlocal.SL-ZPA-proto5

MINEREXPERIMENTAL
ABOUT minemeldlocal
Local prototype library managed via MineMeld WebUI
ABOUT minemeldlocal.SL-ZPA-proto5
Proto 5
CLASS
minemeld.ft.http.HttpFT
INDICATOR TYPES
IPv4
TAGS
ConfidenceHighShareLevelGreen
CONFIG
age_out
default: null
interval: 270
sudden_death: true
attributes
confidence: 100
share_level: green
type: IPv4
extractor Content[].IPs[]
indicator
regex: (.*\")([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/[0-9]{1,2})(\".*)
transform: \2
prefix zs
source_name zscaler
url https://ips.zscaler.net/zpa/json

Step 2 - Created Miner Node

Name = SL-ZPA-Miner5
STATUS
CLASS minemeld.ft.http.HttpFT
PROTOTYPE minemeldlocal.SL-ZPA-proto5
STATE STARTED
LAST RUN 2020-02-13 14:59:29 +0800 WAITING
# INDICATORS 1
OUTPUT ENABLED
INPUTS none

Step 3 - Created Aggregator Prototype/Processor Node

Name = minemeldlocal.SL-ZPA-AggProto5

PROCESSORSTABLE
ABOUT minemeldlocal
Local prototype library managed via MineMeld WebUI
ABOUT minemeldlocal.SL-ZPA-AggProto5
Generic Aggregator for IPv4 indicators. Inputs with names starting with "wl" will be interpreted as whitelists.
CLASS
minemeld.ft.ipop.AggregateIPv4FT
INDICATOR TYPES
IPv4
TAGS
None
CONFIG
infilters
NAME CONDITIONS ACTIONS
accept withdraws
__method == 'withdraw'
accept
accept IPv4
type == 'IPv4'
accept

Step 4 - Created Aggregator Node

Name = SL-ZPA-Agg5

STATUS
CLASS minemeld.ft.ipop.AggregateIPv4FT
PROTOTYPE minemeldlocal.SL-ZPA-AggProto5
STATE STARTED
# INDICATORS 1
OUTPUT ENABLED
INPUTS
SL-ZPA-Miner5

Step 5 - Created Output Node

Name = SL-ZPA-Out5
STATUS
CLASS minemeld.ft.redis.RedisSet
PROTOTYPE minemeldlocal.SL-ZPA-OutProto5
STATE STARTED
FEED BASE URL https://192.168.19.144/feeds/SL-ZPA-Out5
TAGS
# INDICATORS 1
OUTPUT DISABLED
INPUTS
SL-ZPA-Agg5

Step 6 - I pressed "Commit" - this resulted in the stop/restart & reported no errrors....but

The result  presented at https://192.168.19.144/feeds/SL-ZPA-Out5:

     211.144.19.126-211.144.19.126

So it looks like it has retained the last line. The interpretation of the mask looks corrrect - but I need to
see all ip ranges.

If I am reading the meaning of the Indicators value correctly it looks like there has only been one
subnet value presented from the start of the action by the Miner ( although I may be misunderstanding the
relevant sequence of processing ).

Can anyone shed any light on where I am going wrong?

Many Thanks.

[SlowTypist · Accepted solution]

Hi,

After looking around at lots of other prototype definitions and running some more tests I found "a" solution.

I moved to replication of a prototype with class = minemeld.ft.json.SimpleJSON".

After this I just used the simple extractor line ; Content[].IPs[].{"indicator":@}

All good after this.

Thanks.