Random denial

[jdprovine]

What would cause traffic that is allowed through one rule to be denied by a rule below it. Shouldn't it just go through the rule allowing it and not even go any further.

Odder even still is that traffic is not always denied sometimes it does pass through the rule it is allowed.

[Lucky]

Hi Jprovine,

In my experience so far, such things will happen most often when there is a change to AppID or less often when there is an issue with UserID. Does your traffic suddenly change to other policy or it sometimes misses expected policy? Can you share details of what is evaluated in the policy? If your first policy that is missed sometimes is set for "any" app, could it be that you have UserID active and your users sometimes come from wired sometimes from wireless network, and one of them doesn't have UserID mapping?

If you'd describe policy that is missed (what are you evaluating) it'd be easier to guess...

Best regards

Luciano

p.s. sorry for coming off so wrong in the other thread, that was certainly not my intention, I didn't re-read what I wrote and adjust my tone once I typed out the response. Apologies for that.

[jdprovine]

Apology accepted, let me gather some details for you and some screenshots

[jdprovine]

I have attached the following

The first  pic shows that tcp8070 is being denying access fomr 192.152.126 to 136.175.237.201

The second pic show the rule that allows that same traffic

[Lucky]

Hi,

I think you uploaded the same allow pic twice, it is the same timestamps and session ID. Also, I would rather see the sec. policy that does not match sometimes, and just crop around it, no need for the rest of the screen. Maybe remove previous pics as they show some info on your network and we don't need them here for the troubleshooting 🙂

Rgds.

[jdprovine]

here is the pol that allow it

[Lucky]

Hi, thanks for the upload.

It does not make sense if traffic is steady from known IPs to known ports.There is no AppID in the rule nor userID...

What I would do now is get to the CLI and check the packet flow, read out the log of the packet flow and figure out where it goes wrong. Here is the link onto the detailed explanation how to do packet flow captures: https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Run-a-Packet-Capture/ta-p/62390

Here is the list of commands for your first IP address in the destination list (change accordingly)

debug dataplane packet-diag clear all

debug dataplane packet-diag set filter match source 194.239.152.126 destination 136.176.237.146

debug dataplane packet-diag set filter match destination 194.239.152.126 source 136.176.237.146

debug dataplane packet-diag set log feature flow basic

That so far set up your filters and debug collection. I would not set for ports because there might be a port change or something along the way, let's capture everything between hosts. Check if setup is fine by issuing

debug dataplane packet-diag show settings

if all ip addresses are ok, get everything ready to initiate connection and than: turn on filters, logging and captures by issuing

debug dataplane packet-diag set filter on

debug dataplane packet-diag set log on

Initiate the problematic connection (or reset it by killing the session, if you are troubleshooting when it appeared) and wait for it to fail, once it fails stop collection of information

debug dataplane packet-diag set filter off

debug dataplane packet-diag set log off

Wait for two minutes, and do

debug dataplane packet-diag aggregate-logs

less dp-log pan_packet_diag.log

search for first initiating packet (normal "less" behavior as in any linux) and follow the flow as it went through the firewall, from ingress to slowpath to fastpath and forward - you will see all the details and will be able to read out what happened to it.

If firewall has several dataplanes, than it will be less dp0-log or dp1-log or other dp, but any session will always initiate in dp0 and while it might be moved to other dataplanes if they exist, depending on their load, it will also say so in the logs from dp0.

If reading the file becomes too complicated, create tech support file, download it, unzip it, find the log in question and paste anonymously to some pastebin so I can read it? I am not bad at reading those flows.

If reading my instructions becomes too complicated, ... sorry 🙂 ask for clarification, rather than guessing.

DON'T forget to stop logging (do "show setting" to see if you disabled it) and don't do this without TAC assistance if you are unsure of health of the firewall (too busy or critical, etc.)... you can always blame them if it goes wrong 🙂

Otherwise, I am not sure what to tell you, I am thinking your traffic is having some issues, I have never seen such "straight-forward" rule fail.. sounds really strange that anything would miss such a simple rule, right?

Let me know of the results, I am intrigued now.

Regards

[jdprovine]

This is was operates our spinkler system so its not on all the time, looks like some great troubleshooting instructions though. I will let you know what I get, it is a very odd situation

[BenjAudy.MTL]

Hi,

Can you show the screenshot where the traffic is denied? Like Lucky said, you posted the same picture twice.

Benjamin

[Lucky]

Good call, Benjamin, perhaps I missed something really obvious.

Jprovine - if you ever have time to do this troubleshooting steps even for the known traffic, I find it as a valuable tool, just to follow up simple and known TCP three-way handshake through flow basic. It was very valuable for me because I learned steps firewall takes in evaluating and processing packets, and it shows lots of information (can be a bit intimidating until you get used to it, it "shows" how it thinks). With flow basic I can figure out probably 8 or 9 out of 10 problems I run into.

And as Benjamin said, let us see that failed session, maybe we can spot it from there.

Rgds

[jdprovine]

Sorry I thought I had in another post here it is

[Lucky · Accepted solution]

Hi,

denied one is UDP to 146 and also from port 8070 to port 8070 while allowed one is to different IP and from different port and it is TCP, last but not the least (why?) on the denied one there is no destination interface (are you missing the route for this IP address from the firewall?)

So, check your custom service8070 does it include both udp and tcp ports, and check if you can ping that IP address from the IP that is on the ethernet1/15.237 (ping source _ip_from_1/15.237_ host 136.176.237.146) or do "show routing route" to check routes to that ip....

Rgds

[BenjAudy.MTL]

When you see not-applicable as the application, it means the firewall didn't even bother analyze the content of the packet because the port or protocol does not match any Allow rule. You must have made a mistake with your service definition. The name you gave them implies it's just for TCP traffic.

The dropped packets on my firewall also don't have any destination interface, so it's probably not related.

Benjamin

[jdprovine]

Well the services are set to tcp14001 and tcp8070 how would I know if one of them was wrong?

[jdprovine]

I will check that out thanks