Custom App for CRL downloads

[dennisss]

Hi,

I am trying to create a custom app that will match CRL downloads, to allow them without any questions ask. Shouldn't be too hard : on a previous web security gateway, I would match a pattern like the following: "http://([^/:])*crl.*\.crl"

When translated to an app signature, I already know I am looking for two patterns, on the following contexts:

• http-req-host-header
• http-req-uri-path

Now, my issue is that my pattern are incredibly simple. I am not able to reach the 7 bytes limit with this. One other way would be to match the MIME type in the RSP header I guess... application/pkix-crl or application/x-pkcs7-crl are fine. The issue is with the occasional misconfigured root CAs that still reports text/plain.

So....  Is there anything I can do with this to simply allow CRLs without creating a huge custom category?

Thank you.

[mikand]

Perhaps creating an URL-filter instead?

[dennisss]

The URL-filter has severe limitations. Unless I maintain a huge custom category of all the common CRL distribution points on the Internet, I can't see how I would do this. I have two main issues with the URL-filter expressions.

1. matching hosts, the wildcard has to match a complete token. I can't use  *crl.*.* -> *crl is invalid because it is not the only character in the token. So, I can't restrict destination URLs based on the hostname.
2. The path is not predictable enough. All I know is that the file name is going to be .crl,, but it might a very long path (many subdirectories) or a short one. ex:  /*.crl, /*/*/*/*.crl ... Still, that's not so bad because I could create a dozen of URL filter expressions that would match the file up to 11 directories deep, as an exemple. And that would cover most case... But it still something that could be avoided with a custom appid, or less restrictive URL expressions.

Unless there is something I don't know yet about Pan-OS url-filter expressions...

[mikand]

Sounds like a case where you should contact the appid team and submit this as a request: http://researchcenter.paloaltonetworks.com/tools/

[dennisss · Accepted solution]

Well, I was able to create a working signature. I just needed to increase my "scope" ; I used the http-req-headers instead of http-req-uri-path. So, I can match on something like  ".*\.crl HTTP.*User\-Agent.*, and I looking into the http-rsp-headers for the transaction as well. Simple enough... I feel like it was a worthy exercise after all.

I attached the file for other people to use, and I submitted the request as well.

Thank you.

[craymond]

Thank you -

That was a very interesting work around, and will be useful.

[SRA]

You can try to match on the http-rsp-headers for the content type of crl : application/pkix-crl, application/x-pkcs7-crl, etc.

[fhu_omi]

Hello

I tested this custom APP ID but on PAN 9.1.1 and i think also on 9.0.7 it is not working. As i see, there is a problem in the http-rsp-headers match condition. Without the http-rsp-headers condition it is working. The other way around, only with the http-rsp-headers it does also not working, therefore it seems a problem in the http-rsp-headers condition. I tried to simplify this condition .*((pkix-crl)|(x-pkcs7-crl)|(plain)).* but it does not match. I also tried .*((pkix\-crl)|(x-pkcs7\-crl)|(plain)).* and only pkix\-crl , because my test  response is a pkix one, but nothing, the requests alwys recognized as web-browsing. Any ideas?

[AlexNC]

Hi Everyone

I'm new to the topic of custom App-IDs, i tried this custom App-ID. but when accessing these CRL urls then it never matches this rule. It is either incomplete or web-browsing. I am running 9.1.3 currently.

Has anyone this custom App-ID still running and working?

Thanks for any help and best regards

Alex

[AlexNC]

I've been testing the custom application and trying to figure out what is causing it to not work anymore.

My knowledge of custom App-ID's is very limited so if there is a mistake within the application please apologize and let me know.

For now I have adjusted the custom App-ID and got it to work (i was testing it out by blocking anything matching that App-ID) and checking if all the CRL URL's are being blocked now.

So far I got the following results:

http://crl.sca1b.amazontrust.com/sca1b.crl > working OK
http://crl.quovadisglobal.com/hydsslg2.crl > working OK
http://cdp.geotrust.com/GeoTrustRSACA2018.crl > working OK
http://cdp.geotrust.com/GeoTrustRSACA2018.crl > working OK
http://crl3.digicert.com/ssca-sha2-g6.crl > working OK
http://crl4.digicert.com/ssca-sha2-g6.crl > working OK
http://crl.pki.goog/GTS1O1core.crl > working OK
http://crl.godaddy.com/gdig2s1-1878.crl > working OK
http://crl3.digicert.com/sha2-ev-server-g2.crl > working OK
http://crl4.digicert.com/sha2-ev-server-g2.crl > working OK

http://mscrl.microsoft.com/pki/mscorp/crl/Microsoft%20IT%20TLS%20CA%205.crl > working OK > application/octet-stream

http://crl.swisssign.net/EEFD46CAF7275E91BC5AB6E787CD0AFA550A2642 > not working yet
http://crl.swisssign.net/E7F1E7FD2E53AD11E5811A57A4738F127D98C8AE > not working yet

Now I only need to know how to upload a file and i'll gladly share the app

[Sec101]

I've removed the rsp section, and only used the GET http-req-headers.   The match specified in the above article does not work.    It's not the rsp section that is the problem, it is the initial http-req-headers pattern match.  What did you do to get the first match working?  

In the packet captures taken on the firewall it appears there is a /r/n at the end of the request.   Does wireshark add these in, or is this something literally being read by the firewall?

Everything else has been removed except the 1st http-request header, method get:

.*\.crl HTTP.*\r\n  will match

.*\.crl HTTP.* does not match

.*\.crl\sHTTP.*  does not match

[AlexNC]

<application version="9.1.0">
  <entry name="APP_CRL_VAL">
    <signature>
      <entry name="GET CRL File">
        <and-condition>
          <entry name="And Condition 1">
            <or-condition>
              <entry name="Or Condition 1">
                <operator>
                  <pattern-match>
                    <qualifier>
                      <entry name="http-method">
                        <value>GET</value>
                      </entry>
                    </qualifier>
                    <pattern>.*\.crl HTTP.*User\-Agent.*</pattern>
                    <context>http-req-headers</context>
                  </pattern-match>
                </operator>
              </entry>
            </or-condition>
          </entry>
          <entry name="And Condition 2">
            <or-condition>
              <entry name="Or Condition 1">
                <operator>
                  <pattern-match>
                    <pattern>((application/pkix-crl)|(application/x-pkcs7-crl)|(text/plain)|(application/octet-stream)).*</pattern>
                    <context>http-rsp-headers</context>
                  </pattern-match>
                </operator>
              </entry>
            </or-condition>
          </entry>
        </and-condition>
        <scope>session</scope>
        <order-free>no</order-free>
      </entry>
    </signature>
    <subcategory>infrastructure</subcategory>
    <category>networking</category>
    <technology>client-server</technology>
    <description>matches CRL download</description>
    <risk>2</risk>
    <evasive-behavior>no</evasive-behavior>
    <consume-big-bandwidth>no</consume-big-bandwidth>
    <used-by-malware>no</used-by-malware>
    <able-to-transfer-file>no</able-to-transfer-file>
    <has-known-vulnerability>yes</has-known-vulnerability>
    <tunnel-other-application>no</tunnel-other-application>
    <tunnel-applications>no</tunnel-applications>
    <prone-to-misuse>no</prone-to-misuse>
    <pervasive-use>no</pervasive-use>
    <file-type-ident>no</file-type-ident>
    <virus-ident>no</virus-ident>
    <data-ident>no</data-ident>
    <default>
      <port>
        <member>tcp/80</member>
      </port>
    </default>
  </entry>
</application>

This is the app that I got to work and added a 3rd way that I found within the wireshark traces.

But again, this is my first attempt on getting anything working with a custom PAN app-id and I used the base application provided within this Post. Without that I wouldn't have even known where to begin.

Feel free to try it out and maybe/hopefully you find a flaw or something. I wouldn't be surprised if there is something odd in there.

Cheers

[Sec101]

I must have been doing something wrong, as I'm getting matches off of this now.   Thank you for the quick reply.   Appears to be working!

Still not sure why the  below will create a match.   I udnerstand the http.*    But the User\-Agent directly after it doesn't make sense.  Unless the firewall is taking "any" of these patterns and making a match against what it sees?   

HTTP.*User\-Agent.*

A request I see in a browsers appears to look like:

GET /pkiops/crl/MicCodSigPCA2011_2011-07-08.crl HTTP/1.1

there is only a User-Agent listed as a separate request header:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0
But it must check any of these fields?

[AlexNC]

Hi There

To be honest the User Agent part was already in there.
I just took the app, tried it..... tried to so what wireshark says/shows and searched some documents on how the string should look.

if I recall correctly there is only a difference was pattern matching and changing it to "session" instead of "protocol-data-unit" after that it worked and I added the case for "application/octet-stream" since I found some CRL Pages sending it like that.

I don't know or do much with custom App-ID so maybe someone else can weigh in on it.

Regards

Alex

[Sec101]

I'm not sure on this, but it looks like octet-stream, might be a fallback for an unknown.   BTW-Changing to session did the trick.   Makes sense as to why "transaction" wouldn't match this no matter what I was adding.   It almost looks as though the firewall wasn't reading that HTTP response.  Even though the below post seems to say otherwise:

https://live.paloaltonetworks.com/t5/general-topics/transaction-and-session-of-signature-scope/td-p/...