topic Re: Gaming PCs and Consoles with DIPP NAT in General Topics

Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 07 Mar 2017 01:13:26 GMT

Hi all,

I've been researching this for a while now and I've also opened a case on the issue. Basically we just moved off of our Cisco ASA platform and on to our Palo Alto and I've run into a snag with gaming devices. We're a university with quite a few on-campus students and they understandably want to be able to use their entertainment devices on the network. This all seemed to work fine with the ASA... each building had a single public IP address for NAT and the ASA used Port Address Translation and the gaming devices showed they were behind a "Type 2" or "Moderate" NAT.

My new config is an Active/Active deployment and I've set up four public IP addresses for each building using DIPP. Gaming devices, as well as some PC game and voice applications, now show they're behing a "Type 3" or "Strict" NAT. Certain game and voice functionality will not work in this environment.

I know Palo Alto's first suggestion is Static NAT which is a bit problematic when we're talking about hundreds of devices. Sure, it's possible but it's a bit of a management headache. I'd love to just give them all public IPs and move away from NAT but we simply don't have enough IPv4.

I'm curious how others have resolved this?

Re: Gaming PCs and Consoles with DIPP NAT

reaper — Tue, 07 Mar 2017 08:39:37 GMT

you might want to give dynamic IP a shot (sans port)

this method maintains the original source port of the client, which comes in handy with gaming consoles

it may a bit more demanding on your pool, but if you have 4 IP's that should help (you can always oversubscribe a little AND there's a dipp backup) : Tutorial: Network Address Translation

can I ask for the reason Active/Active is being used? (it's only really good to 'fix' asymmetric routing, for all other scenarios it usually simply decreases capacity and increases complexity)

Re: Gaming PCs and Consoles with DIPP NAT

BPry — Tue, 07 Mar 2017 13:42:27 GMT

Having setup housing networks with PA devices I've always setup different floors with one public IP address a piece, obviously this would need to be modified to fit what you are allocated, and then setting them up with a dynamic-ip-and-port policy to match there network range. This would be even easier if you do things on a per building basis like you are, and I've never run into any issues with it at all.

I would be curios to know why an active active configuration was used as well; some people when moving from an existing ASA active active configuration that just keep the same basic setup, even when it doesn't really make sense from a Palo Alto perspective. I would investigate if you actually need to be doing this, more than likely you don't and would benefit the same from an active passive configuration.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 07 Mar 2017 16:27:12 GMT

Thanks for the reply so far guys.

The active/active deployment was originally chosen by our previous network admin and I continued on the deployment after they left. We have dual-homed 10gig connectivity from our provider and our traffic can take either connection going out depending on the route. We're not using even a single full 10gig connection yet so it probably doesn't really matter at this time but the idea was to have the capacity for the full 20gig use if needed down the line. Of course if we have a failed piece of hardware at some point it may result in degraded service but probably only if we're utilizing over 10gig of bandwidth at that time. So far everything seems OK with the deployment other than that the way I'm currently advertising our NAT IP addresses are just an OSPF export rule facing our edge router so each firewall sees a route to the other one. This hasn't seemed to effect incoming traffic though.

I've been testing with my PS4 and, so far, the only way I've gotten a Type 2 is to set it's NAT to a Dynamic IP pool. The DIPP pool gives it Type 3 Strict every time. The current rule I'm testing is a reserved DHCP address for the PS4 and it is NATed with a DIPP pool of a single public IP. Even with this I'm getting Type 3.

BPry, if you've gotten this to work with DIPP is the thought that it might be something to do with the Active/Active? My DIPP NAT rules are configured with the same 4 IP addresses for each PAN box for complete failover (i.e. no dropped incoming packets in case of a single firewall outage since both firewalls have the same NAT rules with all of the 4 IPs per building).

I've been thinking that our UDP session timeout on the ASA was modified to be much longer than the default on the PAN boxes (30 seconds) or that perhaps Cisco's implementation of PAT was a bit more sticky a far as what ports each device got each time it established a new session.

Re: Gaming PCs and Consoles with DIPP NAT

jvalentine — Tue, 07 Mar 2017 17:10:37 GMT

I don't believe this is a UDP timeout problem... this is likely an Application-Layer Gateway issue. In order to excape from the most restrictive NAT modes, most of gaming consoles require that the NAT device support UPNP or be configured with static NAT.

The main problem is that UPNP is an extremely insecure protocol and is not supported today by Palo Alto Networks firewalls. This is a good thing(tm), generally speaking, as the environment is more secure... but can be somewhat problematic for users in your type of environment.

Have you had this problem with XBox? My understanding is that Palo Alto Networks modified the "teredo" App-ID (which is required for the XBox-Live App-ID to function properly) and added an Application-Layer Gateway (ALG) with the goal of making XBox-Live compatible with DIPP NAT.

If XBox is working in moderate-NAT mode using standard DIPP NAT, then I'd open a feature request and/or Application request to have them do something similar for the Playstation-Network App-ID signature.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 07 Mar 2017 17:16:25 GMT

Hi jvalentine,

So far we've gotten complaints for XBox, PS4, and some gaming applications/platforms on PC. I'd have to look to see if we've gotten any tickets for Nintendo platforms.

Completely agree about UPNP but I'm pretty certain the Cisco ASA 5580 we previously had deployed didn't supported UPNP either (it would be very strange if it had). While I had not personally tested a gaming platform on campus beforehand, the complaints only started after we deployed the Palo Alto.

I'll make sure we get our hands on an XBox so we can test internally with that as well.

Re: Gaming PCs and Consoles with DIPP NAT

BPry — Tue, 07 Mar 2017 17:22:53 GMT

@jsalmans I've only ever done Active/Passive so I'm not certain if it's possible that your active/active config is messing things up at all. Have you monitored the sessions for the consoles IP at all to see if you are getting the info back, or if the session is time-out for some reason.

Re: Gaming PCs and Consoles with DIPP NAT

jvalentine — Tue, 07 Mar 2017 17:23:25 GMT

Check and see if the XBox complaints are due to "strict NAT" or "moderate NAT".

(I should really break out my XBox and try it on my own network... it's been a while).

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 07 Mar 2017 17:36:43 GMT

@BPry Yesterday when I was working on it (i.e. playing a game... sometimes I love my job) I had the session browser open on one of the firewalls and I saw active sessions for SIP and some other protocols. Since we just put this thing in place I'm still learning now that we have real traffic passing through and I'm relying on my training course I took a year ago.

@jvalentine I'm going to have someone look into this today. I know we've gotten complaints about Strict NAT on the XBox Live App but that is actually running on a PC. I just browsed the tickets I have and most of them for XBox are talking about disconnects during online games (which may or may not be related) and I haven't found one that specifically mentions the NAT type.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Wed, 08 Mar 2017 01:38:06 GMT

@jvalentine I received a few reports back today from Xbox One users and, apparently, their devices are actually showing Type 1 Open NAT which surprised me. If you're correct and there is an ALG in effect then the issue I've been describing may be related only to the XBox Live PC application, various PC games and game platforms that utilize voice chat, and the PS4 consoles.

Apparently the tickets I've been getting for XBox consoles are about getting disconnected from games and party chat... still might be a NAT problem or perhaps a session timeout. I'm trying to get ahold of an XBox One to continue testing and I'll be sure to try to find the session state information for the PS4 tomorrow.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Wed, 08 Mar 2017 19:24:45 GMT

@BPry I'm looking at an active PS4 session right now and I'm seeing mostly aged-out on UDP traffic. It looks like the UDP traffic is mostly unknown-udp, stun, and rtp-base. Most of the TCP traffic seems to be ssl and Playstation Network but those are getting tcp-fin, tcp-rst-from-server, and tcp-rst-from-client. I am seeing some aged-out on TCP but I believe that was while we were in the game menu. This is while playing Overwatch which seems to be working just fine, including recieving voice chat.

I think the gaming/voice issues must revolve around P2P games. The XBox is showing Type 1 Open NAT but gets disconnected after only a minute or two of play on a P2P game. The Teredo sessions are showing aged-out but the age varies... some show longer than 30 seconds.

I do see some drops coming in from one of the IPs the Teredo established to as they're hitting an incoming drop rule we have but presumably that would only be happening if the remote device was trying to initiate new connectivity to the console. We don't have incoming ports from the internet open to much of anything except external facing services nor did we on the ASA.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Thu, 09 Mar 2017 00:52:29 GMT

Another update.

As a temporary solution I've created an external list with the reserved private IP addresses for affected devices and created a NAT policy set to Dynamic IP with the source as the external list.

Doing this puts the PS4 devices on "Type 2 Moderate" NAT. The XBox One devices still say Type 1 Open NAT but do not seem to be getting disconnected after only a minute or two in a peer to peer game.

I consider this more of a workaround than a long term solution... I'd still like to find out why DIPP isn't working for this and if there is anything to be done about it. We may be doing a network maintenance on Friday for a connected device so I may try to suspend one of the active firewalls to try to determine if this has anything to do with the active/active configuration or how I have the shared NAT set up between them.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 11 Apr 2017 15:16:05 GMT

So my workaround apparently did not resolve this completely as many Ubisoft games are still reporting Strict NAT even though the actual console is saying Moderate. These Ubisoft games are also getting disconnected during multiplayer gameplay.

I've got a TAC case open that is currently analyzing some packet captures but haven't heard back from them yet. In the meantime:

I've intiated a Suspend on one of our active firewalls so that the network acts as if only one firewall was in place. This did not resolve the issue and we're still getting Strict NAT and disconnects after only a minute or two into multiplayer gameplay. This should help eliminate our Active/Active setup from the list of suspects.
We set up our PA-200 on a connection through our connection from a local ISP (our main connection goes through a regional education and research optical network). We've started with the PA-200 set up more as a SOHO device using DIPP NAT for the outside interface which is set to receive DHCP from the ISP. From there, we've started making the config look more and more like our production firewalls but, so far, no problems when playing multiplayer through this even though the consoles are reporting Strict NAT.

Since I don't have multiple IP addresses from the smaller ISP, the next test will probably be to set up a new network in our production environment and assign some of our public IP addresses to be used as a DIPP NAT pool for the PA-200. It is really interesting that the PA-200 hasn't encountered the disconnect issues although it also could be that our test environment doesn't have multiple Playstation or XBox consoles all playing the same games at once.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Mon, 03 Jul 2017 13:43:56 GMT

We've made some progress on this and so I thought I'd come and do an update to this thread.

We may have identified two separate issues here:

Disconnection with certain games during multiplayer
NAT type reported by consoles

Originally I thought these were all part of the same issue, and originally when we were having problems with a much wider selection of games/consoles perhaps it was, but changing the NAT type to Dynamic IP resolved MOST of issue #1 and issue #2 but not all. We were still having issues with about 4 games that we know of, all AAA titles and 3 of them were from Ubisoft. Some of these games also apparently utilize peer-to-peer for faster multiplayer game play (less latency than having to use the Ubisoft servers as a middle-man) which complicates things further.

After some research I determined I'm not the only one having issues with Ubisoft games... look at their forums and it doesn't take long to see people complaining about their network stack, even on home networks (which, as we all know, are usually much simpler than enterprise and these devices are designed to work at home). Packet captures ended up showing Ubisoft servers attempting to open new connections outside of the IP/ports that had existing NAT sessions. At this point we aren't really sure why, we didn't see the behavoir on our test PA-200, only on our production PA-5060 a/a deployment, and packet capture from the Playstation 4 and from the firewalls show no new session was generated using the NAT IP that the Ubisoft servers were trying to communicate with. I've opened a case with Ubisoft and escalated it with no response yet.

As for issue #2, the TAC engineer I was working with found out from the internal team that DIPP NAT uses a Symmetric NAT implementation. Apparently, this implementation does not play nice with certain applications such as STUN which some of the consoles use to determine their NATed IP and port information which they then communicate to the game servers. It is unclear to me at this point what NAT implementation our previous ASA used... the only documentation I've been referred to so far indicates that the ASA attempts to use the port the internal device originally requested if it is available and, if not, it moves to the next available in the sub-pool.

Re: Gaming PCs and Consoles with DIPP NAT

it-thomas — Mon, 03 Jul 2017 14:56:13 GMT

Something else to note. SInce you are running A/A and a pair of 10g lines.

I remember in the past that there were issues with the PAN devices not playing well with asym traffic and it required a CLI command to be run in order to resolve it. It was resulting in asym traffic being dropped by default. Worth tossing at your TAC engineer and see if that might be a related issue.

Interesting post to say the least. A lot of us in the corporate world dont get to struggle with gaming traffic going over our circuits.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Mon, 03 Jul 2017 16:33:15 GMT

@it-thomas I'll respond back to the case and ask about that. If I remember correctly, there is a GUI setting now to allow or disallow asymetric traffic but it would be interesting if this was a cause... my understanding is that the firewall should be identifying if it is the owner of the session and, if not, pushing it across the HA link to the other firewall for processing. As such, there shouldn't be any true asymetric traffic.

Also, my NAT rules are duplicated on both devices. Originally I had though to separate them so that each device had two unique NAT IP addresses for each network and to advertise those out OSPF to our edge router. This would essentially force return traffic to come back to the correct firewall. However, after talking to the engineers and researching I realized if a firewall goes down, the other will just drop all return traffic since it doesn't have a NAT rule matching the IP addresses.

What I ended up doing was to create two versions of each NAT rule, both with the exact same IP address pools, and applying one to each firewall. Then I advertised the entire range through OSPF out from each firewall. This means that if a firewall fails the other should be able to process the traffic but the downside is returning traffic could go to the wrong firewall (but should then be routed to the proper one over the HA link as mentioned above).

My TAC engineer did look at packet captures and determined he didn't see a session being set up for the IP addresses Ubisoft was trying to reach... we're completely baffled on where the server is getting the idea to send traffic to that

*edit*

Not to derail the subjet but another solution just occured to me for the Active/Active NAT scenario I outlined above. It might take more work and maintenance but I think it would both assist in enforcing return traffic to the correct firewall AND allow for a firewall to go down without simply dropping traffic.

Split the 4 IP range I have now for each NAT pool so each has 2 IPs.. one pool would belong to firewall 0, the other to firewall 1.
Continue to advertise the entire range through OSPF
Add each pool to the HA Virtual Addresses as floating IPs setting the firewall priority accordingly so that each pool lives with it's owner firewall until one goes down. Each NAT pool would now have a specific routes advertised out for the pools it owns and these would all migrate if a firewall goes down.
Create two NAT rules for each firewall. First rule is the one that firewall is designed to use for outgoing NAT. The second rule uses the NAT pool the other firewall will use, but, since it is second, it should never get hit for outgoing use. Instead it is just there to have a matching NAT rule for any return traffic still active if the other firewall goes down.

4 IP address pools is a bit overkill for DIPP with our traffic load... our ASA was doing Dynamic PAT with 1 address used... so 2 in each pool should handle all of the traffic if one firewall is down.

Re: Gaming PCs and Consoles with DIPP NAT

jsalmans — Tue, 25 Jul 2017 18:59:55 GMT

One of our Sales Engineers got back to us and let us know the feature request has been added and that priority is at least partially based on the number of similar requests so here is the information for anyone who wants to reach out to their team and add to it:

Feature Request:

FR ID: 7654

Requesting support of DIPP with non-strict recognition by devices.

*Edit* The feature request got resubmitted and got a new ID so I updated the ID here.