topic Re: DNAT issues into servers with teamed nic's ? in General Topics

DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 14:56:17 GMT

DNAT issues into servers with teamed nic's ?

Anyone seen issues with this before ?

I literally can't DNAT into servers with teamed nic's..

I'm going to run a wireshark capture on the server to see what is going on..

Re: DNAT issues into servers with teamed nic's ?

TranceforLife — Sun, 14 May 2017 19:07:25 GMT

Are servers virtual or physical boxes? Can you access resources locally (without going through the Palo)?

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 15:16:55 GMT

Yes.. can hit the services on the servers fine inside...

Hyper-V.. quite sure.. don't have access to the underlying Hyper-V solution.. just the guest.. but can see Hyper-V NDIS bindings everywhere in the network settings so assume they are virtual and Hyper-V is the virt solution.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 15:45:54 GMT

The NAT's are getting to the server...

I can see it in wireshark on the endpoint..

There's a mess of ip checksum mismatch's and ACK number mismatches..

Probably assymetry going on..

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 15:51:52 GMT

There.. Following a stream end to end.

Re: DNAT issues into servers with teamed nic's ?

DPoppleton — Fri, 12 May 2017 16:09:24 GMT

Traffic is getting from 10.1.0.17 to 14.202.123.242 (SYN), and 14.202.123.242 is replying (SYN-ACK) but 10.1.0.17 isn't getting it. Is it getting blocked in the firewall?

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 16:10:35 GMT

Yes.. I was going to say that .. because client is sending SYN's again with seq=0 and never updating.

Checking...

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 16:23:29 GMT

Negative. The inter-zone default block at the bottom is not popping up when over-ride and log is set.

And wireshark shows some streams getting to the SMTP EHLO. So I know it gets through sometimes.

Also on PAN monitor it shows as tcp-fin for session end reason.. so I know it gets to the end of the tcp state-machine.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 16:36:11 GMT

Not just SMTP.

Even RDP.

I notice in PAN logs it shows app as 'incomplete' and session end reason 'aged-out'.

But the successfull ones are app 'ms-rdp' and tcp states that are otherwise.

Like port '80' another DNAT, app parsed as 'activesync'... but.. also a ton of 'incompletes' .. which I can re-create by simply telnetting to port 80.. which would normally test my DNAT out.. but this deployment .. ages out.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 16:47:47 GMT

I'm going to go on the basis the teamed setup on the NICs for the two servers i'm DNAT'ing to are creating session/state protocol hygiene disaster where PAN needs top notch hygiene (i.e. a cheap-ass NAT/router that was the routing/NAT point before the PAN port forwarded fine for these services).

Is there anything I can disable in terms of hygiene checks with assymetry for the TCP state machine on the PAN that will undeniably confirm/deny this is the issue ?

Also wireshark is spitting out ip checksum offload errors everywhere.. which may not be so much an issue as wireshark complains about this when packets have the whole hardware processing offload on the NIC for the checksum. Guaranteed the servers with their broadcom software and NICs and team setup are doing this checksum offload.

But thought id just mention it.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 16:55:39 GMT

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 12 May 2017 17:50:39 GMT

I did this.

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Set-the-Palo-Alto-Networks-Firewall-to-Allow-non-Syn/ta-p/62868

Didn't make a difference. 😕

Re: DNAT issues into servers with teamed nic's ?

TranceforLife — Sun, 14 May 2017 09:37:07 GMT

Hi,

Once the first packet (SYN) traversed the firewall from the client side the session was created. Reply back from the server (SYN, ACK) will hit the same session on the firewall and should get delivered to the client. Can you please run a PCAP from the client side as well as on the Palo (creating filters so you only catching a relevant data). What we want to see is (SYN, ACK) hitting the client side as well as Palo sees ACK from the client.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Sun, 14 May 2017 09:39:57 GMT

Agreed... shall get it tomorrow.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Sun, 14 May 2017 23:38:49 GMT

@TranceforLife wrote:
Are servers virtual or physical boxes? Can you access resources locally (without going through the Palo)?

Yes.. so strangely.. I see a correlation.

2 x servers being Hyper-V hosts.

They can't have anything DNAT'd to them.

Guests that lie ontop of them are fine.

I don't think it has anything to do with the teamed NICs.

I tore them down and still an issue.

Yes, can access resources from behind the firewall... all INSIDEis no issue.

Some sort of protocol/packet/path hygiene, I can think of, is the only issue.

Swap the PAN for a standard perimeter NAT/router.. like a cheapy/small thing.

And it works fine. Routing/NAT'ing through the PAN. Equals blockages again.

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Mon, 15 May 2017 00:28:44 GMT

@MicGioia wrote:
@TranceforLife wrote:
Are servers virtual or physical boxes? Can you access resources locally (without going through the Palo)?

And interesting @TranceforLife you asked whether the destination was virtual or physical.

Because that seems to be a correlation now.. Why did you ask that ?

Re: DNAT issues into servers with teamed nic's ?

TranceforLife — Wed, 17 May 2017 20:44:02 GMT

Hey,

Can not remember why l did ask that. Maybe just a random thought 😄 In any case, DNAT based on destination ip. As long as your server truly holds that ip connection should work. Did you figure out in the end?

Re: DNAT issues into servers with teamed nic's ?

mpgioia — Fri, 26 May 2017 04:21:39 GMT

* facepalm * l3 switch/routing concentration point had two default routes.. One pivoted one way another pivoted to inside of PAN.. ('another way' was the next hop of a prior device that the PAN is intending to replace..)....

Everything fine and good.

Re: DNAT issues into servers with teamed nic's ?

TranceforLife — Fri, 26 May 2017 06:36:51 GMT

Well done!