User's in session table hitting wrong NAT rule

[Ben-Price]

Hi All,

I have a client that has several NAT rule's (as per below). The have discovered in the session table 2 IP's from the 10.128.48.0/22 subnet seem to be hitting 'guest_nat' rule below when they should be hitting the 'users_nat' rule below. When testing the NAT policy match with the affected IPs they hit the correct NAT rule (users_nat).

They are currently migrating some of security policy rules to use group mappings, but I don't believe that would affect this in anyway, but could be wrong. 

Affected IPs:

10.128.48.10/22

10.128.48.11/22

admin@fw1(active)> test nat-policy-match protocol 6 from users to untrust source 10.128.48.11 destination 216.239.38.210 destination-port 443

Source-NAT: Rule matched: users_nat
10.128.48.11:0 => 203.100.5.6:51317 (6),

admin@fw1(active)> show running nat-policy <------ removed non relevant NAT policies
}

"users_nat; index: 16" {
nat-type ipv4;
from users;
source 10.128.48.0/22;
to untrust;
to-interface ;
destination any;
service 0:any/any/any;
translate-to "src: 203.100.5.6 (dynamic-ip-and-port) (pool idx: 1)";
terminal no;

}

"guest_nat; index: 18" {
nat-type ipv4;
from guest;
source 10.128.16.0/20;
to untrust;
to-interface ;
destination any;
service 0:any/any/any;
translate-to "src: 203.100.5.6 (dynamic-ip-and-port) (pool idx: 1)";
terminal no;
}

admin@fw1(active)> show running nat-rule-ippool rule users_nat

VSYS 1 Rule users_nat:
Rule: users_nat, Pool index: 1, memory usage: 36472
-----------------------------------------
Oversubscription Ratio: 4
Number of Allocates: 367534
Last Allocated Index: 39302
-----------------------------------------
------------ Private Pool ---------------
Number of Allocates: 0
Last Allocated Index: 0

admin@fw1(active)> show running nat-rule-ippool rule guest_nat

VSYS 1 Rule guest_nat:
Rule: guest_nat, Pool index: 1, memory usage: 36472
-----------------------------------------
Oversubscription Ratio: 4
Number of Allocates: 367552
Last Allocated Index: 54244
-----------------------------------------
------------ Private Pool ---------------
Number of Allocates: 0
Last Allocated Index: 0

admin@fw1(active)> show running global-ippool

Idx Type From To ToNum Ref. Cnt Mem Size Ratio Ready
---- --------------- -------------------------------- --------------- ----- -------- -------- ----- ------
1 Dynamic IP/Port 10.128.48.0-10.128.51.255 203.100.5.6 1 4 36472 4 k1 /p1
2 Dynamic IP/Port 10.128.12.0-10.128.13.255 203.100.5.10 1 2 36472 4 k1 /p1
3 Dynamic IP/Port 10.0.98.0-10.0.98.255 203.100.5.7 1 2 36472 4 k1 /p1
4 Dynamic IP/Port 10.40.0.7-10.40.0.7 203.100.4.140 1 1 36472 4 k1 /p1
5 Dynamic IP/Port 10.0.35.0-10.0.35.255 203.100.4.142 1 1 36472 4 k1 /p1

Usable NAT DIP/DIPP shared memory size: 61785060
Used NAT DIP/DIPP shared memory size: 182360(0.30%)
Dynamic IP NAT Pool: 0(0.00%)
Dynamic IP/Port NAT Pool: 5(0.30%)

admin@fw1(active)> show session info

target-dp: *.dp0
--------------------------------------------------------------------------------
Number of sessions supported: 2097150
Number of allocated sessions: 692
Number of active TCP sessions: 405
Number of active UDP sessions: 275
Number of active ICMP sessions: 0
Number of active GTPc sessions: 0
Number of active HTTP2-5gc sessions: 0
Number of active GTPu sessions: 0
Number of pending GTPu sessions: 0
Number of active BCAST sessions: 0
Number of active MCAST sessions: 0
Number of active predict sessions: 0
Number of active SCTP sessions: 0
Number of active SCTP associations: 0
Session table utilization: 0%
Number of sessions created since bootup: 6168769
Packet rate: 1764/s
Throughput: 11728 kbps
New connection establish rate: 6 cps
--------------------------------------------------------------------------------
Session timeout
TCP default timeout: 3600 secs
TCP session timeout before SYN-ACK received: 5 secs
TCP session timeout before 3-way handshaking: 10 secs
TCP half-closed session timeout: 120 secs
TCP session timeout in TIME_WAIT: 15 secs
TCP session delayed ack timeout: 250 millisecs
TCP session timeout for unverified RST: 30 secs
UDP default timeout: 30 secs
ICMP default timeout: 6 secs
SCTP default timeout: 3600 secs
SCTP timeout before INIT-ACK received: 5 secs
SCTP timeout before COOKIE received: 60 secs
SCTP timeout before SHUTDOWN received: 30 secs
5GC delete timeout: 15 secs
other IP default timeout: 30 secs
Captive Portal session timeout: 30 secs
Session timeout in discard state:
TCP: 90 secs, UDP: 60 secs, SCTP: 60 secs, other IP protocols: 60 secs
--------------------------------------------------------------------------------
Session accelerated aging: True
Accelerated aging threshold: 80% of utilization
Scaling factor: 2 X
--------------------------------------------------------------------------------
Session setup
TCP - reject non-SYN first packet: True
Hardware session offloading: True
Hardware UDP session offloading: True
Tunnel acceleration: True
IPv6 firewalling: False
Strict TCP/IP checksum: True
Strict TCP RST sequence: True
Reject TCP small initial window: False
Reject TCP SYN with different seq/options: True
ICMP Unreachable Packet Rate: 200 pps
--------------------------------------------------------------------------------
Application trickling scan parameters:
Timeout to determine application trickling: 10 secs
Resource utilization threshold to start scan: 80%
Scan scaling factor over regular aging: 8
--------------------------------------------------------------------------------
Session behavior when resource limit is reached: drop
--------------------------------------------------------------------------------
Pcap token bucket rate : 10485760
--------------------------------------------------------------------------------
Max pending queued mcast packets per session : 0
--------------------------------------------------------------------------------

Has anyone seen such an issue before?

Thanks in advance.

[aleksandar.astardzhiev · Accepted solution]

Hi @Ben-Price ,

I was surprized as well, I believe (not completete sure) it is "cosmetic bug" or should I say expected behaviour, due to the fact that both of your NAT rules are using same source translated address.

If you look closely to the output you provide you can see that both running nat-policy rules are saying they are using same pool idx: 1

[aleksandar.astardzhiev]

Hi again @Ben-Price ,

I tried to replicate your case with virtual test FW, but I am not getting the same result (wrong nat rule in the session details).

Note: My test was with VM running 9.0.11

admin@PA-VM> show running nat-policy

"users; index: 1" {
nat-type ipv4;
from user;
source 10.10.10.0/24;
to untrust;
to-interface ;
destination any;
service 0:any/any/any;
translate-to "src: 20.20.20.3 (dynamic-ip-and-port) (pool idx: 1)";
terminal no;
}

"guests; index: 2" {
nat-type ipv4;
from guests;
source 192.168.10.0/24;
to untrust;
to-interface ;
destination any;
service 0:any/any/any;
translate-to "src: 20.20.20.3 (dynamic-ip-and-port) (pool idx: 1)";
terminal no;
}

admin@PA-VM> show session id 132

Session 132

c2s flow:
source: 10.10.10.1 [user]
dst: 8.8.8.8
proto: 6
sport: 36651 dport: 22
state: INIT type: FLOW
src user: unknown
dst user: unknown

s2c flow:
source: 8.8.8.8 [untrust]
dst: 20.20.20.3
proto: 6
sport: 22 dport: 1484
state: INIT type: FLOW
src user: unknown
dst user: unknown

start time : Fri Aug 13 08:08:39 2021
timeout : 5 sec
total byte count(c2s) : 296
total byte count(s2c) : 0
layer7 packet count(c2s) : 4
layer7 packet count(s2c) : 0
vsys : vsys1
application : incomplete
rule : allow-all
service timeout override(index) : False
session to be logged at end : True
session in session ager : False
session updated by HA peer : False
address/port translation : source
nat-rule : users(vsys1)
layer7 processing : enabled
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : True
session traverses tunnel : False
session terminate tunnel : False
captive portal session : False
ingress interface : ethernet1/1
egress interface : ethernet1/2
session QoS rule : N/A (class 4)
tracker stage firewall : Aged out
end-reason : aged-out

Session 135

c2s flow:
source: 192.168.10.1 [guests]
dst: 8.8.8.8
proto: 6
sport: 49743 dport: 22
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

s2c flow:
source: 8.8.8.8 [untrust]
dst: 20.20.20.3
proto: 6
sport: 22 dport: 12242
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown

start time : Fri Aug 13 08:09:36 2021
timeout : 5 sec
time to live : 1 sec
total byte count(c2s) : 222
total byte count(s2c) : 0
layer7 packet count(c2s) : 3
layer7 packet count(s2c) : 0
vsys : vsys1
application : undecided
rule : allow-all
service timeout override(index) : False
application db : 0
app.id : c2s node (0, 0) s2s node (0, 0)
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
address/port translation : source
nat-rule : guests(vsys1)
layer7 processing : enabled
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : True
session traverses tunnel : False
session terminate tunnel : False
captive portal session : False
ingress interface : ethernet1/3
egress interface : ethernet1/2
session QoS rule : N/A (class 4)
end-reason : unknown

Unfortunately I don't have resources at the moment to have proper test (with established session), but I assume the result will be the same.

With above I would say:

- I still believe you see cosmetic bug. The correct NAT is used, but wrong nat name is logged. There is no way your traffic from users zone to hit rule with different source zone.

- What version are you running?

- Have you tried to set different source-translated address, just for a test?

[Ben-Price]

@aleksandar.astardzhiev Thanks for testing. I have also run some basic testing in my lab with a similar scenario e.g. 2 internal subnets using dynamic IP and port using the same translated source address. When I ping a host on the internet the PAN session browser identifies the correct NAT rule to use. 

The client is running PAN OS 9.0.4, I am running an older version 8.1.19.

I have asked them to try the below:

Kill the session for both of the affected IPs and then re-initiate it and see which NAT policy they hit (try this multiple times to see if they constantly hit the wrong rule)?

Disable the guest_nat policy and re-initiate host's connection and see if they can connect out and what the session browser indicates as per NAT policy?

Change the source translated address on the guest_nat policy and see if the issue goes away or continues?

It does seem to be looking as a cosmetic bug, as all other matching criteria match as per the outlined users_nat rule. I will revert once I have further info.

[Ben-Price]

@aleksandar.astardzhiev Thanks for your input here. 

It does look as though it was some sort of 'cosmetic bug', as you mentioned. We have not been able to replicate the issue, nor have seen any further incidents.

I am not sure if the pool idx number makes a difference, as you tested and I tested in my lab, I could not re-create the issue. 

We have closed this case internally for now and will keep an eye on it.

Thanks.