Tips & Tricks: App-ID Debugging

nikoolayy1 — Wed, 25 Jun 2025 14:53:04 GMT

This article is inspired from Tips & Tricks: Flow Basic Debugging written by @kiwi and I recommend reading that article first before reading this one.

Palo Alto Networks NGFWs use App-ID to detect the exact application inside a traffic stream but sometimes traffic will be first classified for example as App-ID "SSL" and after the decryption it can be classified for example as "web-browsing". This is called application shift and sometimes it can cause issues as when traffic App-ID changes it will go again through the Security Policy Match. If you have a rule that blocks "ssl" at the top and then a rule that allows "web-browsing" then this could be an issue 😉 Also always add the correct parent apps to your custom apps.

For more information see:

How to Prevent Application Shift - Knowledge Base - Palo Alto Networks

How can you see your traffic and the original application App-ID after that. What are the next App-ID's for traffic to see the application shift or if the wrong app-id is matched for some reason? How to see if the final app-id for some reason is not the correct one?

Well this is easy as you just need to enable packet debugging for app-id. For example I used "Example Domain" that is https and it first will be detected as "ssl" and after I configured decryption on the firewall it will be detected as "web-browsing" and you can use a self-generated CA cert as Trusted CA for the SSL emulation as shown in How to Generate a New Self-Signed SSL Certificate - Knowledge Base - Palo Alto Networks.

For testing I suggest using "route ADD" on windows that will cause traffic from your computer to go to the FW ip address and then use "curl" to start traffic. As www.example.com changes it's IP address as it is behind Akamai, use "--resolve" to not do DNS resolution and "-kv" to ignore the errors from the self-signed cert.

route ADD 88.0.0.0 MASK 255.0.0.0 192.168.211.138 curl https://www.example.com/ -kv --resolve www.example.com:443:88.221.211.16

Next in the CLI we enable debugging (I am using VM but note on VM devices you may need to disable session offload "set session offload no"):

debug dataplane packet-diag set filter match source 192.168.211.1 destination 88.221.211.16 destination-port 443 debug dataplane packet-diag set filter match source 88.221.211.16 destination 192.168.211.1 source-port 443 debug dataplane packet-diag set log feature appid all debug dataplane packet-diag set log on Packet log is enabled. WARNING: Enabling of debug commands could result in network outage. Not recommended if dataplane CPU is above 60%. admin@PA-VM> debug dataplane packet-diag show setting -------------------------------------------------------------------------------- Packet diagnosis setting: -------------------------------------------------------------------------------- Packet filter Enabled: yes Match pre-parsed packet: no Filter offload: yes Index 1: 192.168.211.1/32[0]->88.221.211.16/32[443], proto 0 ingress-interface any, egress-interface any, exclude non-IP Index 2: 88.221.211.16/32[443]->192.168.211.1/32[0], proto 0 ingress-interface any, egress-interface any, exclude non-IP -------------------------------------------------------------------------------- Logging Enabled: yes Log-throttle: no Sync-log-by-ticks: yes Features: appid : basic policy dfa agt Counters: Timeout duration: 60 seconds Buffer threshold: 80% CPU threshold: 80% -------------------------------------------------------------------------------- Packet capture Enabled: no Snaplen: 0 Username: --------------------------------------------------------------------------------

After running the curl command and stop loggin you can see the logs (for non VM platforms "less dp-log pan_packet_diag.log" is the command to see logs as they are not in the managment plane but seperate physical data plane):

debug dataplane packet-diag set log off debug dataplane packet-diag aggregate-logs less mp-log pan_packet_diag.log ............................... 025-06-19 12:08:02.646 +0300 debug: pan_appid_simple_cache_find(pan_appid_cache.c:518): [CACHE] found app 15 for daddr 0-10d3dd58ffff0000(443, 6) not valid time 5240(5659) count 9(16) 2025-06-19 12:08:02.646 +0300 debug: pan_appid_cache_find(pan_appid_cache.c:725): [CACHE] not found appid 0-1d3a8c0ffff0000(62808) -> 0-10d3dd58ffff0000(62808) 2025-06-19 12:08:02.646 +0300 debug: pan_policy_lookup(pan_policy.c:2302): [ACE] Trigger slow match for appid(0) uappid(0) 2025-06-19 12:08:02.646 +0300 debug: pan_policy_match_service(pan_policy.c:1576): match 0,0 for app 0 uapp 0 proto 6 sport 62808 dport 443 192.168.211.1[62808]-->88.221.211.16[443] 2025-06-19 12:08:02.656 +0300 pan_appid_process_lookup(pan_appid_proc.c:2884): work 0xe03675f100 2025-06-19 12:08:02.656 +0300 debug: pan_session_set_appid_session(pan_appid_priv.h:192): Set appid_session 0xe005fbc100 for session 139 2025-06-19 12:08:02.656 +0300 debug: pan_appid_simple_cache_add_i(pan_appid_cache.c:886): [CACHE] daddr 0-10d3dd58ffff0000(443) appid 15 match appid in cache count 9 threshold 16 2025-06-19 12:08:02.656 +0300 pan_appid_policy_lookup(pan_appid_proc.c:3175): SESSION-DSCP: set session DSCP: 0x00 pan_policy_lookup_fill_arg(normal) : src zone(1) dst zone(2) src(000000000000000000000000c0a8d301:192.168.211.1) src port(62808) dst(00000000000000000000000058ddd310:88.221.211.16) dst port(443) is_ssl(0) 2025-06-19 12:08:02.656 +0300 pan_cfg_app_policy_lookup(pan_cfg_policy.c:1165): Before lookup (short-category-vector): 192.168.211.1[62808]-->88.221.211.16[443] app 15 use app 15; do lookup 1, app fst 0, url fst 1, rule(1): rule-1 category: any, session_category(sp): any, any, any, any, any, any 2025-06-19 12:08:02.656 +0300 debug: pan_policy_lookup(pan_policy.c:2302): [ACE] Trigger slow match for appid(15) uappid(0) 2025-06-19 12:08:02.656 +0300 debug: pan_policy_match_service(pan_policy.c:1576): match 0,0 for app 15 uapp 0 proto 6 sport 62808 dport 443 2025-06-19 12:08:02.656 +0300 pan_cfg_set_session_rule_data(pan_cfg_policy.c:1482): SESSION-DSCP: Security policy match, setting session DSCP 0 2025-06-19 12:08:02.656 +0300 pan_cfg_app_policy_lookup(pan_cfg_policy.c:1193): After lookup: 192.168.211.1[62808]-->88.221.211.16[443] app 15 use app 15; do lookup 1, app fst 0, url fst 1, rule(1): rule-1 category: any, .............................. 192.168.211.1[62808]-->88.221.211.16[443] 2025-06-19 12:08:02.676 +0300 pan_appid_process_match(pan_appid_proc.c:1950): match data 759 2025-06-19 12:08:02.676 +0300 debug: pan_appsig_process_result(pan_app_sigs.c:669): Process app signature [web-browsing] rule [check-get-post-put], dir [cts] 2025-06-19 12:08:02.676 +0300 debug: pan_appsig_process_result(pan_app_sigs.c:695): slot is 1 2025-06-19 12:08:02.676 +0300 debug: pan_appid_check_string_match(pan_app_sigs.c:477): MATCH_ANY: app rule [check-get-post-put] match 2025-06-19 12:08:02.676 +0300 debug: pan_appsig_process_result(pan_app_sigs.c:743): Application found [web-browsing] 2025-06-19 12:08:02.676 +0300 debug: pan_appid_process_match(pan_appid_proc.c:1963): Do policy lookup for session 139 container app: 109 2025-06-19 12:08:02.676 +0300 pan_appid_policy_lookup(pan_appid_proc.c:3175): SESSION-DSCP: set session DSCP: 0x00 pan_policy_lookup_fill_arg(normal) : src zone(1) dst zone(2) s admin@PA-VM>

The most important parts are "found app 15" and "Do policy lookup for session 139 container app: 109" as this means traffic was first identified as app 15 and then switched to app 109 and thanks to @kiwi the below internal ID can be resolved with the commands below.

admin@PA-VM> debug device-server dump idmgr type shared-application id 15 ssl admin@PA-VM> debug device-server dump idmgr type shared-application id 109 web-browsing admin@PA-VM> debug device-server dump idmgr type shared-application name web-browsing 109 admin@PA-VM> debug device-server dump idmgr type shared-application name ssl 15

After that the Palo Alto Networks app wiki at Application Research Center can also be used. Also if there is a wrong App ID you can write your own using the Host header as I have shown in my previous article How to Write Palo Alto Networks Custom Vulnerability and Application Signatures with Examples | Palo Alto Networks.

NOTE: There could be more than one app shifts during a session as for example ssl > web-browsing > gmail-base so keep that in mind!

That's all folks!

Re: Tips & Tricks: App-ID Debugging

kiwi — Wed, 25 Jun 2025 15:07:08 GMT

Very cool stuff @nikoolayy1 !

Re: Tips & Tricks: App-ID Debugging

crasmussen — Thu, 26 Jun 2025 17:34:38 GMT

Thanks for sharing with the community @nikoolayy1 !

Re: Tips & Tricks: App-ID Debugging

jforsythe — Thu, 26 Jun 2025 18:01:54 GMT

This is an absolutely brilliant deep dive into App-ID debugging! The explanation of application shift and the practical CLI commands for packet debugging are incredibly helpful. The "route ADD" and "curl" examples made it super easy to follow along. Huge thanks for sharing these tips and tricks; this is exactly the kind of valuable insight that makes troubleshooting so much clearer! @nikoolayy1

Re: Tips & Tricks: App-ID Debugging

terryguerrero — Tue, 12 Aug 2025 06:20:17 GMT

thanks for sharing this helpful information in the community.

Re: Tips & Tricks: App-ID Debugging

wedlandshannon — Tue, 12 Aug 2025 06:28:39 GMT

good information @nikoolayy1

Re: Tips & Tricks: App-ID Debugging

Vinod_Ola — Mon, 18 Aug 2025 08:28:08 GMT

Thank you for sharing useful information.

article Tips & Tricks: App-ID Debugging in General Articles

Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging

Re: Tips & Tricks: App-ID Debugging