DNAT issues into servers with teamed nic's ?

[mpgioia]

DNAT issues into servers with teamed nic's ?

Anyone seen issues with this before ?

I literally can't DNAT into servers with teamed nic's..

I'm going to run a wireshark capture on the server to see what is going on..

[TranceforLife]

Are servers virtual or physical boxes? Can you access resources locally (without going through the Palo)?

[mpgioia]

Yes.. can hit the services on the servers fine inside...

Hyper-V.. quite sure.. don't have access to the underlying Hyper-V solution.. just the guest.. but can see Hyper-V NDIS bindings everywhere in the network settings so assume they are virtual and Hyper-V is the virt solution.

[mpgioia]

The NAT's are getting to the server...

I can see it in wireshark on the endpoint..

There's a mess of ip checksum mismatch's and ACK number mismatches..

Probably assymetry going on..

[mpgioia]

There.. Following a stream end to end.

[DPoppleton]

 Traffic is getting from 10.1.0.17 to 14.202.123.242 (SYN), and 14.202.123.242 is replying (SYN-ACK) but 10.1.0.17 isn't getting it. Is it getting blocked in the firewall?

[mpgioia]

Yes..  I was going to say that .. because client is sending SYN's again with seq=0 and never updating.

Checking...

[mpgioia]

Negative.  The inter-zone default block at the bottom is not popping up when over-ride and log is set.

And wireshark shows some streams getting to the SMTP EHLO.  So I know it gets through sometimes.

Also on PAN monitor it shows as tcp-fin for session end reason.. so I know it gets to the end of the tcp state-machine.

[mpgioia]

Not just SMTP.

Even RDP.

I notice in PAN logs it shows app as 'incomplete' and session end reason 'aged-out'.

But the successfull ones are app 'ms-rdp' and tcp states that are otherwise.

Like port '80' another DNAT, app parsed as 'activesync'... but.. also a ton of 'incompletes' .. which I can re-create by simply telnetting to port 80.. which would normally test my DNAT out.. but this deployment .. ages out.

[mpgioia]

I'm going to go on the basis the teamed setup on the NICs for the two servers i'm DNAT'ing to are creating session/state protocol hygiene disaster where PAN needs top notch hygiene (i.e. a cheap-ass NAT/router that was the routing/NAT point before the PAN port forwarded fine for these services).

Is there anything I can disable in terms of hygiene checks with assymetry for the TCP state machine on the PAN that will undeniably confirm/deny this is the issue ?

Also wireshark is spitting out ip checksum offload errors everywhere.. which may not be so much an issue as wireshark complains about this when packets have the whole hardware processing offload on the NIC for the checksum.  Guaranteed the servers with their broadcom software and NICs and team setup are doing this checksum offload.

But thought id just mention it. 

[mpgioia]

[mpgioia]

I did this.

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Set-the-Palo-Alto-Networks-Firewa...

Didn't make a difference. 😕

[TranceforLife]

Hi,

Once the first packet (SYN) traversed the firewall from the client side the session was created. Reply back from the server (SYN, ACK) will hit the same session on the firewall and should get delivered to the client. Can you please run a PCAP from the client side as well as on the Palo (creating filters so you only catching a relevant data). What we want to see is (SYN, ACK) hitting the client side as well as Palo sees ACK from the client. 

[mpgioia]

Agreed... shall get it tomorrow.

[mpgioia]

---

@TranceforLife wrote:

Are servers virtual or physical boxes? Can you access resources locally (without going through the Palo)?

---

Yes.. so strangely.. I see a correlation.

2 x servers being Hyper-V hosts.

They can't have anything DNAT'd to them.

Guests that lie ontop of them are fine.

I don't think it has anything to do with the teamed NICs.

I tore them down and still an issue.

Yes, can access resources from behind the firewall... all INSIDEis no issue.

Some sort of protocol/packet/path hygiene, I can think of, is the only issue.

Swap the PAN for a standard perimeter NAT/router.. like a cheapy/small thing.

And it works fine.  Routing/NAT'ing through the PAN.  Equals blockages again.