DBL Formatting

[nbilal]

We're having an issue getting firewalls running PANOS 6.1 and 7.0 to consume the DBL's created by Minemeld output nodes. None of the output node URL's end with .txt, which appears to be a requirement according to the live documents below. However, .txt cannot be added to the output node name in Minemeld. Is there some configuration we're missing?

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Configure-Dynamic-Block-List-DBL-...

https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Videos/PAN-OS-7-1-URL-Filtering-Dynamic-Block-List-E...

For example, when I point my browser at http://myminemeld/feeds/output_node_name, I am able to display the without issue. However, when I configure a DBL in PANOS 6.1 or 7.0 for that same URL and perform a request system external-list refresh... then request system external-list show... I'll get an error "Server error : external list file not found", which I assume is because PANOS is looking for a .txt.

Any ideas? I haven't tried this on PANOS 7.1, as customer will not be moving to that train until we've seen a few more maintenance releases.

[Greg_R]

Have  you added a policy referencing the EBL/DBL? It won't load until there is a policy calling it.

[nbilal]

Hey Greg! Thanks for the input. 

Yes, it is referenced in an active policy. In the event that a policy is not available, you'd see a different error message, something like "EBL not referenced in an active rule."

[Greg_R]

Have you tried using the 'request system external-list url-test' command to see if the url is accessible?

Additionally I had an almost identical issue last week and the issue was related to the firewall having multi vsys support enabled. The command 'set system setting target-vsys <vsys1>' ended up fixing my problem. I found the info in this article:

https://live.paloaltonetworks.com/t5/Learning-Articles/Working-with-External-Block-List-EBL-Formats-...

[lmori]

Hi Nasir,

could you check the detailed error in ms.log on PAN-OS ?

I have tested the URLs with PAN-OS 7.0 and I had no issues with the missing ".txt".

[nbilal]

This DBL is not readable by single or multi-vsys systems. 

request system external-list url-test {dbl}

Returns "URL is accessible"

[nbilal]

bilalnas@fw01(active-primary)> tail mp-log ms.log
2016-05-09 14:28:47.588 -0400 Error: ebl_verify_new_fetched_copy(pan_cfg_ebl.c:730): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) No valid entries found.
2016-05-09 14:28:47.588 -0400 Error: ebl_update_local_file(pan_cfg_ebl.c:1001): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Unable to fetch external list. Using old copy for refresh.
2016-05-09 14:28:47.589 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Unable to open EBL(/opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl)
2016-05-09 14:28:47.939 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Build ips node(1)
2016-05-09 14:28:49.759 -0400 dnscfgmod: FQDN Refresh: Periodic Force Refresh
2016-05-09 14:28:49.759 -0400 dnscfgmod: Main refresh function: (Force Refresh)
2016-05-09 14:28:49.759 -0400 dnscfgmod: No fqdns used in this config. Skip config push to device
2016-05-09 14:28:55.475 -0400 client device reported Phase 1 was SUCCESSFUL
2016-05-09 14:28:55.693 -0400 Error: ebl_refresh(pan_cfg_ebl.c:1329): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) EBLRefresh job failed. No valid IPs found in list
2016-05-09 14:28:55.996 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) timer init expires(0, Mon May 9 15:00:55 2016)

It's saying "No valid IP's found in list". Here's a sample of what's available at the configured URL:

104.47.0.0-104.47.127.255
13.107.6.152-13.107.6.152
13.107.6.153-13.107.6.153
13.107.9.152-13.107.9.152
13.107.9.153-13.107.9.153
131.253.33.215-131.253.33.215
132.245.1.128-132.245.111.255
132.245.113.128-132.245.127.255
132.245.129.128-132.245.160.255
132.245.162.128-132.245.164.255
132.245.165.128-132.245.191.255
132.245.193.128-132.245.207.255
132.245.209.128-132.245.255.255
134.170.101.0-134.170.101.255
134.170.140.0-134.170.140.255
134.170.171.0-134.170.171.255
134.170.68.0-134.170.69.255
157.55.11.0-157.55.11.127
157.55.133.0-157.55.133.127
157.55.157.128-157.55.157.255
157.55.158.0-157.55.159.255
157.55.206.0-157.55.207.255
157.55.224.128-157.55.224.255
157.55.225.0-157.55.225.127
157.55.234.0-157.55.234.255
157.55.47.0-157.55.47.255
157.55.49.0-157.55.49.255
157.55.61.0-157.55.61.255
157.55.9.128-157.55.9.255
157.56.106.128-157.56.106.143
157.56.108.0-157.56.108.255
157.56.110.0-157.56.110.255
157.56.111.0-157.56.111.255
157.56.112.0-157.56.112.255
157.56.116.0-157.56.116.127
157.56.120.0-157.56.120.127
157.56.206.0-157.56.206.255
157.56.208.0-157.56.211.255
157.56.232.0-157.56.239.255
157.56.24.0-157.56.24.127
157.56.240.0-157.56.255.255
157.56.73.0-157.56.73.255
157.56.87.192-157.56.87.255
157.56.96.16-157.56.96.31
157.56.96.224-157.56.96.231
157.56.96.232-157.56.96.239
191.232.96.0-191.232.127.255
191.234.140.0-191.234.143.255
191.234.224.0-191.234.227.255
191.234.6.152-191.234.6.152

If I copy and paste this list into a static text file I have sitting on another web server, it works fine as a DBL.

[lmori]

Could you include some earlier log messages before ?

Is PAN-OS downloading the DBLs via a proxy ?

[nbilal]

No proxies between firewalls and Minemeld, and "request system external-list url-test" returns "URL is accessible" for the DBL. Here are the last 200 lines of ms.log, which encompasses two CLI-forced DBL refreshes:

bilalnas@fw01(active-primary)> tail lines 200 mp-log ms.log
2016-05-09 14:10:38.635 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:10:38.635 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:10:38.635 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-3.1.0-actual.xsl provided by cms
2016-05-09 14:10:38.635 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0-actual.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:10:38.636 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:10:38.637 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.1.0.xsl provided by cms
2016-05-09 14:10:38.637 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:10:38.637 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:10:38.638 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0-actual.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-5.1.0.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0-actual.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.0.0.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:10:38.639 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-3.1.0.xsl provided by cms
2016-05-09 14:10:38.640 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:10:38.640 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-3.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0-actual.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.1.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:10:38.641 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:10:38.642 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:10:38.642 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:15:00.847 -0400 Checking to purge appstatdb logtype
2016-05-09 14:16:37.007 -0400 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:367): failed to fetch cfg.platform.uuid
2016-05-09 14:16:37.007 -0400 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:367): failed to fetch cfg.platform.cpuid
2016-05-09 14:16:42.146 -0400 Warning: pan_sigdb_get_idsev_map(pan_sigdb.c:887): /opt/pancfg/mgmt/global/wpc.xml.sev doesn't exist
2016-05-09 14:16:42.146 -0400 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1310): failed to get wpc idsev map
2016-05-09 14:16:42.146 -0400 Warning: pan_sigdb_get_wpcdb(pan_sigdb.c:1049): /opt/pancfg/mgmt/global/wpc.xml.db doesn't exist
2016-05-09 14:16:42.146 -0400 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1314): failed to get wpcinfo db
2016-05-09 14:16:42.198 -0400 Warning: pan_sigdb_get_idsev_map(pan_sigdb.c:887): /opt/pancfg/mgmt/global/wpc.xml.sev doesn't exist
2016-05-09 14:16:42.198 -0400 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1310): failed to get wpc idsev map
2016-05-09 14:16:42.198 -0400 Warning: pan_sigdb_get_wpcdb(pan_sigdb.c:1049): /opt/pancfg/mgmt/global/wpc.xml.db doesn't exist
2016-05-09 14:16:42.198 -0400 Warning: _pan_sigdb_get_hash(pan_sigdb.c:1314): failed to get wpcinfo db
2016-05-09 14:16:42.961 -0400 Update logforward config, flags: mdata[1], log setting[0]
2016-05-09 14:16:42.971 -0400 Commit job enqued. type=4
2016-05-09 14:16:43.052 -0400 Content not configured to sync to peer. Skipping
2016-05-09 14:16:45.024 -0400 client device reported Phase 1 was SUCCESSFUL
2016-05-09 14:16:45.235 -0400 Error: pan_mgmt_get_sysd_string(pan_cfg_status_handler.c:367): failed to fetch cfg.gpdatafile-release-date
2016-05-09 14:16:45.235 -0400 Got HA info from sysd: Local_state: active-primary, Peer_state: active-primary2016-05-09 14:16:45.429 -0400 wrote /opt/pancfg/mgmt/global/panorama-custom-reports.xml provided by cms
2016-05-09 14:16:45.429 -0400 wrote /opt/pancfg/mgmt/global/panorama-reportconfig.xml provided by cms
2016-05-09 14:16:45.429 -0400 wrote /opt/pancfg/mgmt/transforms/identity.xsl provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/transforms/global-transform.xsl provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/global/dg-custom-reportconfig.xml provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/global/merged_dg_ctxt.xml provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-5.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.430 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-3.1.0-to-3.0.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-6.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.431 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.0.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-3.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.432 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.433 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.433 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:16:45.433 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.433 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.433 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.434 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.434 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.434 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.435 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.0.0.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.436 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.437 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-3.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.438 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.439 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.439 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.439 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.439 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.443 -0400 wrote /opt/pancfg/mgmt/global/panorama-custom-reports.xml provided by cms
2016-05-09 14:16:45.443 -0400 wrote /opt/pancfg/mgmt/global/panorama-reportconfig.xml provided by cms
2016-05-09 14:16:45.443 -0400 wrote /opt/pancfg/mgmt/transforms/identity.xsl provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/transforms/global-transform.xsl provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/global/dg-custom-reportconfig.xml provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/global/merged_dg_ctxt.xml provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.444 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-5.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-3.1.0-to-3.0.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-6.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.445 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.0.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.446 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.446 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.446 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.446 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.0.0-to-3.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.446 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.447 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.448 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.448 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.448 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.449 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.449 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.449 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.449 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.449 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-4.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-6.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-4.1.0-to-3.0.0.xsl provided by cms
2016-05-09 14:16:45.450 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-5.0.0.xsl provided by cms
2016-05-09 14:16:45.451 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.451 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-3.1.0.xsl provided by cms
2016-05-09 14:16:45.451 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-7.0.0.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-6.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-6.1.0.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-3.1.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0-actual.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-7.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.0.0-to-5.1.0.xsl provided by cms
2016-05-09 14:16:45.452 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.1.0-to-4.1.0.xsl provided by cms
2016-05-09 14:16:45.453 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-5.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.453 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-7.0.0-to-5.0.0-actual.xsl provided by cms
2016-05-09 14:16:45.453 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.0.0.xsl provided by cms
2016-05-09 14:16:45.453 -0400 wrote /opt/pancfg/mgmt/transforms/tpl-transform-6.1.0-to-6.0.0.xsl provided by cms
2016-05-09 14:16:45.453 -0400 wrote /opt/pancfg/mgmt/transforms/sp-transform-5.0.0-to-4.1.0.xsl provided by cms
2016-05-09 14:28:47.371 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) flushing timer
2016-05-09 14:28:47.380 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) calling /usr/local/bin/newpanupdater.sh -xyes -Eyes -turl -L6500000 -T30 -z'https://vmwl5426/feeds/list_o365_ipv4' 2>/dev/null 1>/opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmpxx
$//g' /opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmpxx 2>/dev/null > /opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmp
2016-05-09 14:28:47.588 -0400 Error: ebl_verify_new_fetched_copy(pan_cfg_ebl.c:730): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) No valid entries found.
2016-05-09 14:28:47.588 -0400 Error: ebl_update_local_file(pan_cfg_ebl.c:1001): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Unable to fetch external list. Using old copy for refresh.
2016-05-09 14:28:47.589 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Unable to open EBL(/opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl)
2016-05-09 14:28:47.939 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) Build ips node(1)
2016-05-09 14:28:49.759 -0400 dnscfgmod: FQDN Refresh: Periodic Force Refresh
2016-05-09 14:28:49.759 -0400 dnscfgmod: Main refresh function: (Force Refresh)
2016-05-09 14:28:49.759 -0400 dnscfgmod: No fqdns used in this config. Skip config push to device
2016-05-09 14:28:55.475 -0400 client device reported Phase 1 was SUCCESSFUL
2016-05-09 14:28:55.693 -0400 Error: ebl_refresh(pan_cfg_ebl.c:1329): EBL entry(0xe4cad90, 0x1ccebe00, 0x145ad000 vsys1/DBL_O365_IPv4, 1, 1) EBLRefresh job failed. No valid IPs found in list
2016-05-09 14:28:55.996 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) timer init expires(0, Mon May 9 15:00:55 2016)
2016-05-09 14:30:00.536 -0400 Checking to purge appstatdb logtype
2016-05-09 14:43:02.022 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) flushing timer
2016-05-09 14:43:02.431 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) calling /usr/local/bin/newpanupdater.sh -xyes -Eyes -turl -L6500000 -T30 -z'https://vmwl5426/feeds/list_o365_ipv4' 2>/dev/null 1>/opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmpxx
$//g' /opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmpxx 2>/dev/null > /opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl.tmp
2016-05-09 14:43:02.644 -0400 Error: ebl_verify_new_fetched_copy(pan_cfg_ebl.c:730): EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) No valid entries found.
2016-05-09 14:43:02.644 -0400 Error: ebl_update_local_file(pan_cfg_ebl.c:1001): EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) Unable to fetch external list. Using old copy for refresh.
2016-05-09 14:43:02.645 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) Unable to open EBL(/opt/pancfg/mgmt/devices/localhost.localdomain/vsys1_DBL_O365_IPv4.ebl)
2016-05-09 14:43:02.994 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) Build ips node(1)
2016-05-09 14:43:10.439 -0400 client device reported Phase 1 was SUCCESSFUL
2016-05-09 14:43:10.664 -0400 Error: ebl_refresh(pan_cfg_ebl.c:1329): EBL entry(0xe4cad90, 0x1ccebe00, 0x1b6c6c00 vsys1/DBL_O365_IPv4, 1, 1) EBLRefresh job failed. No valid IPs found in list
2016-05-09 14:43:10.990 -0400 EBL entry(0xe4cad90, 0x1ccebe00, 0x18869800 vsys1/DBL_O365_IPv4, 1, 1) timer init expires(0, Mon May 9 15:00:10 2016)
bilalnas@fw01(active-primary)>

[lmori]

Which PAN-OS version are you using ?

[nbilal]

PA5050's in HA Active/Active running 7.0.5-h2. No proxies or service routes configured.

[lmori]

Just double checked and it works with PAN-OS 7.0.

Could you check the file /opt/minemeld/log/minemeld-web.log on MineMeld for requests from PAN-OS ?

[nbilal]

Whenever the firewall goes to access the minemeld via "request system external-list url-test", I see this curl User-agent in my Minemeld weblog:

cstart: 0 start+num: 4294967295
interval: 100 desc: False
cstart: 100 start+num: 4294967295
interval: 100 desc: False
cstart: 200 start+num: 4294967295
interval: 100 desc: False
cstart: 300 start+num: 4294967295
interval: 100 desc: False
cstart: 400 start+num: 4294967295
interval: 100 desc: False
127.0.0.1 - - [09/May/2016:19:32:54 +0000] "GET /feeds/list_o365_ipv4 HTTP/1.0" 200 13769 "-" "curl/7.20.1 (i686-redhat-linux-gnu) libcurl/7.20.1 OpenSSL/1.0.1e zlib/1.2.3"
cstart: 0 start+num: 4294967295
interval: 100 desc: False
cstart: 100 start+num: 4294967295
interval: 100 desc: False
cstart: 200 start+num: 4294967295
interval: 100 desc: False
cstart: 300 start+num: 4294967295
interval: 100 desc: False
cstart: 400 start+num: 4294967295
interval: 100 desc: False
127.0.0.1 - - [09/May/2016:19:37:26 +0000] "GET /feeds/list_o365_ipv4 HTTP/1.0" 200 13769 "-" "curl/7.20.1 (i686-redhat-linux-gnu) libcurl/7.20.1 OpenSSL/1.0.1e zlib/1.2.3"
cstart: 0 start+num: 4294967295
interval: 100 desc: False
cstart: 100 start+num: 4294967295
interval: 100 desc: False
cstart: 200 start+num: 4294967295
interval: 100 desc: False
cstart: 300 start+num: 4294967295
interval: 100 desc: False
cstart: 400 start+num: 4294967295
interval: 100 desc: False
127.0.0.1 - - [09/May/2016:19:37:33 +0000] "GET /feeds/list_o365_ipv4 HTTP/1.0" 200 13769 "-" "curl/7.20.1 (i686-redhat-linux-gnu) libcurl/7.20.1 OpenSSL/1.0.1e zlib/1.2.3"

However, if I run "request system external-list refresh", I see no output on the minemeld. Note that the URL I'm putting in the "test-url" command is copied straight from the DBL object on the firewall.

[nbilal · Accepted solution]

Oh man! Silly mistake. I've been making configuration changes on Panorama and pushing to the firewall, however, it was overridden locally on the FW so was not taking the changes. I had been copying the URL from the object in the Panorama device-group, not from the local firewall configuration. When logged into the local firewall found that there was an older DBL configured there.

Sorry for wasting your time! It's working great now.

[lmori]

No worries ! I am glad everything is fine !

And this will also make some good reference for debugging the PAN-OS <-> MM exchange