topic Re: Upgrade to 5.0.14-h3 stopped traffic in General Topics

Upgrade to 5.0.14-h3 stopped traffic

jambulo — Mon, 20 Oct 2014 15:25:47 GMT

We just attempted to upgrade some 5020's to 5.0.14-h3(mainly to patch the evasion vulnerability) and quickly found that the upgrade broke traffic traversing the firewall. During the short period of time it we were running on 5.0.14-h3, there were a whole lot of "incomplete" sessions for TCP and a lot of UDP sessions with zero packets received.

Does anyone else have experience with 5.0.14-h3?

Re: Upgrade to 5.0.14-h3 stopped traffic

hshah — Mon, 20 Oct 2014 15:40:18 GMT

I think you might have asymmetric traffic in network.

Is it among non-internet zone, if yes you might want to try following command.

show counter global | match syn

This will help us to determine potential asymmetric routing issue and fix. If values are high than apply following command.

> configure

# set deviceconfig setting session tcp-reject-non-syn no

# commit

Refer following document for more help.

SYN-ACK Issues with Asymmetric Routing

Regards,

Hardik Shah

Re: Upgrade to 5.0.14-h3 stopped traffic

dreputi — Mon, 20 Oct 2014 16:20:57 GMT

Hello Jambulo,

You can check if the configured interfaces are having proper arp entries on the PAN. Also, check the arp entries on the connected switches/routers. If the device is in HA mode, and if the connected devices didn't update the arp entries to the active device, you can try to clear the arp entries on those devices so that they learn the arp entries freshly. You can also try to run test gratuitous arp command to send out grat arps that will force connected devices to update their arp entries.

>test arp gratuitous ip <ip/netmask> interface <interface>

-Make sure the traffic is hitting the correct rules. For ex, if group-mapping is used in security rules, make sure that the users are properly identified so that they hit correct rules.

Regards,

Dileep

Re: Upgrade to 5.0.14-h3 stopped traffic

hshah — Mon, 20 Oct 2014 17:08:22 GMT

Hi Jabulo,

Following commands should help.

# set deviceconfig setting session tcp-reject-non-syn no |yes <------- asymmetric routing

# set deviceconfig setting tcp asymmetric-path bypass | drop <--------- asymmetric flow of packets

I am very positive its following issue.

6.0.5 h3 explanation

To verify same, provide us following output.

show counter global | match syn

Regards,

Hardik Shah

Re: Upgrade to 5.0.14-h3 stopped traffic

jambulo — Mon, 20 Oct 2014 20:27:46 GMT

Here is the output from "show counter global | match syn"

flow_inter_cpu_nat_mismatch 22592 1 info flow pktproc Inter-CPU NAT sync mismatch

ha_nat_policy_mismatch 104559 5 warn ha system HA NAT session sync: policy mismatch

ha_nat_pool_mismatch 814 0 warn ha system HA NAT session sync: IP/port pool state mismatch

Re: Upgrade to 5.0.14-h3 stopped traffic

hshah — Mon, 20 Oct 2014 20:30:53 GMT

Hi Jambulo,

This doesnt look like a asymmetric routing issue. Please provide us traffic log snapshot. Make sure its enlarged view.

Regards,

Hardik Shah

Re: Upgrade to 5.0.14-h3 stopped traffic

tshiv — Tue, 21 Oct 2014 03:09:01 GMT

Hello jambulo,

I have come across this issue with 5.0.14-h3 software code. It is currently being investigated. It would helpful to us if you can open a support ticket and provide the necessary data. This issue needs to be investigated to find the root cause.

Thanks