topic Re: SSL Inbound decryption woes in General Topics

SSL Inbound decryption woes

cryptochrome — Mon, 08 Jul 2013 19:10:48 GMT

Hi there,

we just configured our first SSL Inbound decryption, but we have some trouble and need help troubleshooting it.

Very simple setup:

Webserver in DMZ zone

Firewall policy: from:untrust to:dmz; src:any; dst:webserver; app:ssl,web-browsing; service:service-http(s); action:allow

Decryption policy: from:untrust to:dmz; src:any; dst:webserver; action:decrypt

The webserver's certifictate and key have been imported to the firewall.

Accessing the webserver from an external PC: Traffic gets decrypted perfectly.

Accessing the webserver from an iPhone and from and Android device: Traffic is *not* being decrypted.

In all cases the source IPs were completely random and are not subject to any firewall rule. This is reproducible and I tried to find out why it would decrypt in one case but not in another.

Any ideas? Is there a way I can troubleshoot this other than looking at the traffic logs, which don't contain any helpful information?

Thanks

Re: SSL Inbound decryption woes

UhMayYeah — Mon, 08 Jul 2013 19:37:17 GMT

Can you try different browsers on the mobile devices.This could be caused by Unsupported SSL cipher suite (algo) selected by Client.

Re: SSL Inbound decryption woes

cryptochrome — Mon, 08 Jul 2013 20:19:35 GMT

How can I see that on the firewall? Where are incompatibilities being logged?

Will try...

Re: SSL Inbound decryption woes

UhMayYeah — Mon, 08 Jul 2013 23:12:24 GMT

If the SSL version or cipher suite is not supported for decryption you should see a related entry in O/P of following command :

>show system setting ssl-decrypt exclude-cache

To reset this cache :

> debug dataplane reset ssl-decrypt exclude-cache

Ref :

Following Global counters could be helpful in troubleshoting :

>show counter global | match proxy - PAN-OS 3.0.0 and 3.1.0

proxy_process 1205 info proxy pktproc Number of flows go through proxy

proxy_no_process 453 info proxy pktproc Number of flows donot go through proxy

proxy_wqe_held 253 info proxy resource Number of wqe held by proxy for notify answer

proxy_excluded 78 info proxy pktproc Number of ssl sessions bypassed proxy because of exclusion

proxy_client_hello_failed 4 warn proxy pktproc Number of ssl sessions bypassed proxy because client hello can't be parsed

proxy_url_request_pkt_drop 24 info proxy pktproc The number of packets get dropped because of waiting for url category request in ssl proxy

proxy_url_category_unknown 23435 info proxy pktproc Number of sessions checked by proxy with unknown url category

url_session_not_in_ssl_wait 4 error url system The session is not waiting for url in ssl proxy

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 06:25:23 GMT

Hi Nadir,

thanks for the help. In fact, when I tried mobile Firefox the session was decrypted just fine. Using Google Chrome (Android) or mobile Safari (iPhone) the session would not get decrypted. Looks like an issue with Webkit based browsers. HOWEVER:

There is nothing related in the certificate list you get when issuing the 'show system setting ssl-decrypt exclude-cache' command. The webserver's certificate does not show up in that list.

What I also don't understand: If there is a problem with ciphers, why can I connect to the webserver using iPhone/Android just fine? The browser seems to be encrypting the session without a problem. It's the firewall that does not decrypt.

Re: SSL Inbound decryption woes

mikand — Tue, 09 Jul 2013 06:39:01 GMT

First of all I would try without the inbound inspection to verify that there isnt something else thats borked.

Then I would verify that I have the inbound inspection properly setup including:

Block sessions with unsupported version

Block sessions with unsupported cipher suites

Block sessions if resources not available

Verify then in monitor -> logs -> traffic that when you visit using browser (from computer and not iPhone/Android) that the traffic is really decrypted (should be a checkbox in the "Decrypted" part to the right of the popup - you get the popup if you click on the magnify-icon to the left of the traffic log row). Then do the same for the visit from your iPhone/Android device and compare.

Third I would take a tcpdump of the traffic which your iPhone and Android produces to look into which cipher suites are being requested and what is actually happening on the line (preferly compare this with the traffic generated when your browser from your computer (non iPhone/Android device) visits this https-site).

For example PA has problems with newer versions of TLS (not supported at all) and the inbound inspection wont work at all if DH (Diffie Hellman) is being used for obvious reasons (both would the tcpdump help you with to find out).

Re: SSL Inbound decryption woes

UhMayYeah — Tue, 09 Jul 2013 06:43:55 GMT

As far I understand , the unsupported modes and failures would be bypassed and not decrypted, unless you have configured a Decryption Profile and checked the options to Block sessions in case these checks fail.

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 06:58:16 GMT

Hi Mikand,

thanks for the detailed instructions. Actually, I already did all of that except for the tcpdumps. The decryption policy is definitely ok (else it wouldn't decrypt the non-iPhone sessions). We also have a decryption profile that specifically blocks unsupported versions/ciphers and if ressource is unavailable. This means these sessions are supposed to blocked, but they are not. They are just not being decrypted.

What bugs me the most is that there is simply no information on the box about what's going on. At least not obvious. The traffic log is not helpful at all, since all it tells me is whether a session was decrypted or not. It doesn't tell me why. There is nothing in the system logs either. The exclude-cache is empty. I love PAN, but in this area they need to improve. A lot.

I am going to ask the customer to send me the server's certificate so I can check whether it uses Diffie Hellman. That's about the only thing left I can think of as a culprit.

Re: SSL Inbound decryption woes

mikand — Tue, 09 Jul 2013 08:19:15 GMT

With some luck there should be some ssl related info in dp-log or the mp-log.

Try with "less dp-log" or "less mp-log" followed by a press on the tab-key to see which files are available.

Once located you can tail the file with:

tail follow yes dp-log <filename>

tail follow yes mp-log <filename>

(dp = dataplane, mp = managementplane)

You can also use grep to search for strings withing these logfiles.

Re: SSL Inbound decryption woes

ericgearhart — Tue, 09 Jul 2013 17:38:21 GMT

We just confirmed this same behavior... a colleague of mine at work was trying to get decrypt working, and I remembered this thread. Decrypt works fine and the URL filtering log populates when he browses using a regular laptop, but from an iPad SSL decrypt does NOT work. The App-ID stays 'ssl' instead of flipping over to web-browsing, and the URL filtering log doesn't populate when he browses from an iPad.

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 18:02:12 GMT

I guess it's time to open a case with PAN support.

Re: SSL Inbound decryption woes

ericgearhart — Tue, 09 Jul 2013 18:03:11 GMT

I've got two pcaps open right now, and it looks like the difference between the two of them is the TLS version that was negotiated.

The pcap I captured of a session where decrypt seems to work has TLSv1. The pcap of the session where decrypt did NOT appear to work negotiated TLSv1.2.

Can somebody else confirm what I'm seeing?

Re: SSL Inbound decryption woes

ericgearhart — Tue, 09 Jul 2013 18:04:03 GMT

We're opening a case too.. we can reference your case (and vice-versa) if you'd like, maybe it'll get some more traction that way.

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 18:05:38 GMT

Thanks. Will do and post the case number here.

Re: SSL Inbound decryption woes

mikand — Tue, 09 Jul 2013 19:10:12 GMT

https://live.paloaltonetworks.com/message/17280#17280

I wonder what happend to TLS1.2 support?

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 19:14:24 GMT

Interesting. Thanks Mikand. Let's see what PAN support has to say about this.

Re: SSL Inbound decryption woes

cryptochrome — Tue, 09 Jul 2013 19:27:08 GMT

MY PAN support case number: 00146841

Re: SSL Inbound decryption woes

ericgearhart — Tue, 09 Jul 2013 20:26:01 GMT

cryptochrome - our case number is 00146826

My colleague titled it "SSL Decrypt does not work from iOS" because that's essentially what the impact of not supporting TLS 1.2 is apparently.

Re: SSL Inbound decryption woes

cryptochrome — Wed, 10 Jul 2013 06:17:44 GMT

- it also affects Android's chrome browser, which is also based on WebKit, just like mobile Safari. Would be interesting to find out whether it also applies to desktop Safari.

Re: SSL Inbound decryption woes

ericgearhart — Fri, 12 Jul 2013 13:47:04 GMT

Support is asking for a freaking GoToMeeting session, which is honestly a waste of time at this point. I'm pushing back on them asking for a GTM session.