Large Varying IP Pools for DNS (CB Defense dev-prod05.conferdeploy.net) And Firewall Rules[SOLUTION]

[BrianRa]

First I would like to say that we are pursuing this with CarbonBlack and we have worked with PAN support already to see what our options are.  This is as much an informative post as it is to see what other people think and are doing.
For the record PAN support suggested changing the DNS entry from a lookup to a FTP file check.  We would prefer to correct the actual problem rather than do this and use the three cron jobs I created (if I die/quit/etc they don't want to have to figure it out, I don't blame them).

On to post #2 and the topic!

There is a solution.  See below.

[BrianRa]

We are currently fighting a problem with the dev-prod05.conferdeploy.net domain name and our firewall rules.  When doing a DNS lookup on this domain devices return 8 IPs.  However this domain has at any given time 30 IP entries (based on the monitoring script I wrote that resets every 24 hours).  If you work with firewalls (or any device) you know that they only do a DNS lookup every so often then cache the results.  This is a problem when a PC/Server can do the same lookup and come up with 22 IPs the firewall does not know about.

We are seeing the firewall allowing and rejecting the same domain name because it does not know the IPs the client is trying to reach (see the excel document).  This would be fairly simple (though clunky) to either tell the firewall to check a FTP file with the list of IPs in it for that domain or just enter all the IPs into a firewall rule and ignore the DNS but over the last couple of months monitoring dev-prod05.conferdeploy.net the IPs have changed.  This is the reason I set the script to reset every 24 hours.

What are people doing to combat this problem?  Other sites do not present in this manner. 

Is this a security thing for CB Defense (so many IPs its hard to take down)?  Or a misconfiguration on the DNS that shows all the current IPs regardless of the region your IP is in?

This can be tested at any time by opening the command line and typing “nslookup dev-prod05.conferdeploy.net” (Windows) 4 or 5 times in a row.  Each time 8 IPs will present and each time there will be IPs that were not listed in the previous lookup.

Our specs:

Palo Alto Firewalls (8.x)

Windows 7/10 desktops

Windows 2012/2016 servers

Linux scripts:

RUN EVERY 5 MINUTES

                  host dev-prod05.conferdeploy.net | grep has | awk '{print $4}' >> dev-prod05.conferdeploy.net_BULK-IPs.txt

RUN EVERY 10 MINUTES

                  awk '!seen[$0]++' dev-prod05.conferdeploy.net_BULK-IPs.txt > dev-prod05.conferdeploy.net.txt

RUN ONCE A DAY:

                  rm /dev-prod05.conferdeploy.net_BULK-IPs.txt

[BrianRa]

Generate Time	Source	Destination	Destination address	Application	Action	URL
9/10/2018 10:00	Internal	External	18.214.112.236	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 10:00	Internal	External	34.198.54.80	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 10:00	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 10:00	Internal	External	34.199.106.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 10:00	Internal	External	34.192.44.112	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:59	Internal	External	34.198.88.190	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:59	Internal	External	34.198.88.190	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:59	Internal	External	34.199.227.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.237.240.4	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.237.240.4	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.237.240.4	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.237.240.4	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.237.240.4	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	35.173.197.210	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	35.173.197.210	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.234.104.211	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:56	Internal	External	34.198.54.80	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.192.44.112	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.194.166.1	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.198.88.190	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.192.210.120	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:55	Internal	External	34.199.106.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:54	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:54	Internal	External	52.207.81.137	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:54	Internal	External	52.207.81.137	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:54	Internal	External	34.197.159.199	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.199.227.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.198.54.80	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.192.44.112	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.192.44.112	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.199.227.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.194.166.1	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.194.166.1	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.192.210.120	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.199.227.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:53	Internal	External	34.197.244.92	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.197.244.92	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.192.44.112	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.194.166.1	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.199.106.239	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.194.166.1	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.199.106.239	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:52	Internal	External	34.192.44.112	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:51	Internal	External	34.199.106.239	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:51	Internal	External	34.199.106.239	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:51	Internal	External	34.199.227.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:51	Internal	External	34.197.244.92	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:51	Internal	External	52.207.81.137	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:50	Internal	External	52.207.81.137	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:50	Internal	External	52.207.81.137	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.199.227.239	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.192.210.120	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.192.210.120	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.198.54.80	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.198.54.80	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.192.210.120	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	18.213.132.157	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.197.159.199	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:49	Internal	External	34.199.106.239	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.194.191.180	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.238.4.33	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	35.173.197.210	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.238.4.33	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	35.173.197.210	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.238.4.33	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.206.73.176	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	18.214.112.236	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	18.214.112.236	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.194.191.180	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.238.4.33	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:48	Internal	External	34.225.200.240	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:47	Internal	External	34.237.240.4	ssl	alert	dev-prod05.conferdeploy.net
9/10/2018 9:47	Internal	External	18.214.112.236	ssl	block-url	dev-prod05.conferdeploy.net
9/10/2018 9:47	Internal	External	34.239.202.48	ssl	block-url	dev-prod05.conferdeploy.net

[Remo]

@BrianRa

I know your post was more informational than anything else, but ...

... just a though but as these connections are all "ssl", why don't you allow this traffic with a custom URL category that you specify directly in your security policy rule instead of using an FQDN object?

[rmfalconer]

I'm not great with Linux so I want to verify my understanding of what your scripts are doing.

The script that runs every 5 minutes seems to run a lookup on that particular host and output the results to a text file.  Is that text file appended or overwritten?

The script that runs every 10 minutes looks for duplicates and outputs the results to a text file. Is that text file appended or overwritten?

The PA does a FQDN refresh every 30 minutes and I think it can only reference 10 results per FQDN so that won't work with the frequency or quantity that you need.

If the second script is appending so that you have a list of permitted IP addresses in that text file, then you can reference that file with an external dynamic list and build that into a security policy. We use many EDLs, including one that we publish internally for blocking certain sites. Our internal list only changes a few times a week so it's not all that dynamic. But its not clunky and works fine with minimal maintenance.

Not a great solution but......you could push out a host file to all the endpoints with a bunch of the correct IP addresses and resolve them to dev-prod05.conferdeploy.net. That would control specifically which IPs the endpoints will try to access and not rely on DNS. 

[BrianRa]

vsys_remo thanks for the reply.

We currentlly have a rule for all "non internet" users/machines that has a URL Filter and we have added a "vendor whitelist" URL Category.  The domain is in this custom list.  This rule also onlly allows SSL and HTTP applications.

This is the rule that is sometimes allowing access and other times denying based on whether or not the PA knows about the IP being used.  I may be missing something you are adding but the custom URL Category list only allows me to put in domains.

[Remo · Accepted solution]

---

@BrianRa wrote:

This is the rule that is sometimes allowing access and other times denying based on whether or not the PA knows about the IP being used.

---

Then - as far as I know PaloAlto - there must be something else that prevents the access, because when you allow access based on a URL the firewall does not care about the IP behind the domainname/URL. In this case the firewall only checks the http host header or SNI extension / certificate CN in a TLS connection.

The logs that you posted: Are these logs all from the same security policy rule? Do you may be specify sourceaddresses in that rule that does not allow the access for all servers that need to connect to that URL? I am asking this because there are multiple IP addresses that are both allowed and blocked and so far I am not (and never was) aware of a bug that results in such a behaviour. Here are some IPs that are allowed and blocked in your log:

• 34.197.244.92
• 34.198.88.190
• 34.197.159.199

[BrianRa]

@rmfalconer I will break each one down so it makes more sense.  If you understand more than this and it is redundant I apologize but I want to make sure it is all clear.  All definitions are in reference to what I am doing with it.  A not on the ".txt" extension, this has nothing to do with linux but makes it easily Windows readable (auto opens in notepad++ for me).

• host = nslookup
• | breaks out for the new command based on the results of the previous command
• grep = search request based on the result of "host" command (because of the pipe) containing the word "has"
• awk = a programing language initiator for printing the 4th value in the string based on a space breakout
• >> is equl to an append at the end of file
• > is equal to a replace/create file
• rm = remove file

RUN EVERY 5 MINUTES (Done every 5 minutes to try to capture all the IPs that are available)

                  host dev-prod05.conferdeploy.net | grep has | awk '{print $4}' >> dev-prod05.conferdeploy.net_BULK-IPs.txt

dev-prod05.conferdeploy.net has address 52.45.174.75

dev-prod05.conferdeploy.net has address 52.2.229.136

Find all lines that contain "has" in the line

Now pull out the 4th variable based on a space delimiter from each line

52.45.174.75

52.2.229.136

Paste that value at the end of the current/defined txt file

RUN EVERY 10 MINUTES (Done every 10 minutes to try to capture new IPs but there is no reason to do it as often because the likely hood of a new IP after the first hour is low)

                  awk '!seen[$0]++' dev-prod05.conferdeploy.net_BULK-IPs.txt > dev-prod05.conferdeploy.net.txt

Search in the defined "BULK-IPs.txt" file for a unique value

Repeat this command for all lines in this file

Paste that unique values into a new/overwritten defined txt file

RUN ONCE A DAY: (I chose once daily because it is easy to troubleshoot and keeps the file down to the expected 30 IPs)

                  rm /dev-prod05.conferdeploy.net_BULK-IPs.txt

Remove the defined "BULK-IPs.txt" file

There is no reason to rm/delete the dev-prod05.conferdeploy.net.txt final product file because it is already overwritten every 10 minutes with a new file

Please let me know if any of this does not make sense and I will try to explain it.

[BrianRa]

@Remo I have included the rule.  We have tried with both Application and Service (better results for internet access in general with Services as it turns out).Yes all the log results are the results of that rule.  We removed all confidential/internal data from the excel sheet.  In that timeframe dozens of hosts had made requests to get to the site.  All the "Destination address" results are valid IPs for the dev-prod05.conferdeploy.net domain.  From what I understand the firewall checks the DNS of the requested domain/site and if they do not match blocks the request (DNS poisoning of a client can be prevented this way).

[EDIT]
I stand corrected.  We did not have that domain properly in the "Vendors Whitelist" like we should have.  We had a ruler higher up in the stack we were using that IS using FQDN for all CarbonBlack domains we need.  I added the dev-prod05.conferdeploy.net properly (spelling) into the custom URL Category and it is functioning properly. 

FQDN - does a DNS lookup to ensure the information is correct

Custom Objects -> URL Category - only checks the domain to ensure it is in the allowed/denied list (as this is SSL an invalid cert warning should popup [in a browser] or the application should [hopefully] fail due to the failed cert)

[/EDIT]

[Remo]

---

@BrianRa wrote:

Custom Objects -> URL Category - only checks the domain to ensure it is in the allowed/denied list (as this is SSL an invalid cert warning should popup [in a browser] or the application should [hopefully] fail due to the failed cert)

---

The cert is only changed by the firewall if you have TLS decryption enabled. Otherwise the connection is simply allowed/denied based on the configuration.

About the previous confusion with the DNS lookups. Maybe you mixed something with the HTTP and TLS evasion Anti-Spyware feature. Because you are right allowing only the access based on a URL actually opens a secuirty risk. So if you really need high security while still allowing specific access to the internet (really high security and internet are mutually exclusive - but thats just my opinion 😛   ) you should also add IP addresses to the destination address column and if you do it the way you did with the dns lookups scripts you also need to make sure that your network is secured for DNS poisoning attacks from the internet. Anyway I wrote already too much details for something that isn't relevant here where I just wanted to explain the risk with the access based on the URL category - so back to that. The risk here is that a client only uses the name "dev-prod05.conferdeploy.net" in the TLS handshake but does not connect to an IP that effectively behind this FQDN. So if the client uses this name but connects to an IP that is controlled by an attacked, the firewall will happily allow the connection with the rule in your screenshot. And this risk can be mitigated by the use of the HTTP and TLS Evasion anti spyware signatures. But for these signatures to work properly you need to activate the DNS proxy feature that ALL dns requests pass the firewall. This way the firewall is able to detect at least situations where the client does a dns request for an fqdn but then connects to another IP than it has received in the dns reply. If a malware connects to this domain using an IP that it has received in another way than dns or a hardcoded one then the evasion signatures will not help either...

At least your problem with the access to this fqdn is solved 😉

[BrianRa]

---

@Remo wrote: 

 So if the client uses this name but connects to an IP that is controlled by an attacked, the firewall will happily allow the connection with the rule in your screenshot.

---

As soon as I realized how that worked this was my first thought!

Fortunately our DNS is fairly secure (externally anyway).  However if a bad actor got in and started spoofing DNS internally then yes we would have problems.  Also a user that understands the vulnerability would be able to access sites that are blocked.  We do have all of the Security Profiles built and are using all but Data Filtering on this rule.

Unfortunately I don't know that DNS proxy would work internally as we are using split DNS internally for both domain and some site requests.  During our initial configuration Palo Alto implied this was not prefered due to the load it put on the firewall (this may not be correct but it has caused us to avoid it on most of our networks).

“Do not own a computer;
Do not power it on;
and do not use one.”

Morris’s Three Golden Rules of Computer Security