03-06-2017 05:13 PM
Hi all,
I've been researching this for a while now and I've also opened a case on the issue. Basically we just moved off of our Cisco ASA platform and on to our Palo Alto and I've run into a snag with gaming devices. We're a university with quite a few on-campus students and they understandably want to be able to use their entertainment devices on the network. This all seemed to work fine with the ASA... each building had a single public IP address for NAT and the ASA used Port Address Translation and the gaming devices showed they were behind a "Type 2" or "Moderate" NAT.
My new config is an Active/Active deployment and I've set up four public IP addresses for each building using DIPP. Gaming devices, as well as some PC game and voice applications, now show they're behing a "Type 3" or "Strict" NAT. Certain game and voice functionality will not work in this environment.
I know Palo Alto's first suggestion is Static NAT which is a bit problematic when we're talking about hundreds of devices. Sure, it's possible but it's a bit of a management headache. I'd love to just give them all public IPs and move away from NAT but we simply don't have enough IPv4.
I'm curious how others have resolved this?
03-07-2017 12:39 AM
you might want to give dynamic IP a shot (sans port)
this method maintains the original source port of the client, which comes in handy with gaming consoles
it may a bit more demanding on your pool, but if you have 4 IP's that should help (you can always oversubscribe a little AND there's a dipp backup) : Tutorial: Network Address Translation
can I ask for the reason Active/Active is being used? (it's only really good to 'fix' asymmetric routing, for all other scenarios it usually simply decreases capacity and increases complexity)
03-07-2017 05:42 AM
Having setup housing networks with PA devices I've always setup different floors with one public IP address a piece, obviously this would need to be modified to fit what you are allocated, and then setting them up with a dynamic-ip-and-port policy to match there network range. This would be even easier if you do things on a per building basis like you are, and I've never run into any issues with it at all.
I would be curios to know why an active active configuration was used as well; some people when moving from an existing ASA active active configuration that just keep the same basic setup, even when it doesn't really make sense from a Palo Alto perspective. I would investigate if you actually need to be doing this, more than likely you don't and would benefit the same from an active passive configuration.
03-07-2017 08:27 AM
Thanks for the reply so far guys.
The active/active deployment was originally chosen by our previous network admin and I continued on the deployment after they left. We have dual-homed 10gig connectivity from our provider and our traffic can take either connection going out depending on the route. We're not using even a single full 10gig connection yet so it probably doesn't really matter at this time but the idea was to have the capacity for the full 20gig use if needed down the line. Of course if we have a failed piece of hardware at some point it may result in degraded service but probably only if we're utilizing over 10gig of bandwidth at that time. So far everything seems OK with the deployment other than that the way I'm currently advertising our NAT IP addresses are just an OSPF export rule facing our edge router so each firewall sees a route to the other one. This hasn't seemed to effect incoming traffic though.
I've been testing with my PS4 and, so far, the only way I've gotten a Type 2 is to set it's NAT to a Dynamic IP pool. The DIPP pool gives it Type 3 Strict every time. The current rule I'm testing is a reserved DHCP address for the PS4 and it is NATed with a DIPP pool of a single public IP. Even with this I'm getting Type 3.
BPry, if you've gotten this to work with DIPP is the thought that it might be something to do with the Active/Active? My DIPP NAT rules are configured with the same 4 IP addresses for each PAN box for complete failover (i.e. no dropped incoming packets in case of a single firewall outage since both firewalls have the same NAT rules with all of the 4 IPs per building).
I've been thinking that our UDP session timeout on the ASA was modified to be much longer than the default on the PAN boxes (30 seconds) or that perhaps Cisco's implementation of PAT was a bit more sticky a far as what ports each device got each time it established a new session.
03-07-2017 09:10 AM
I don't believe this is a UDP timeout problem... this is likely an Application-Layer Gateway issue. In order to excape from the most restrictive NAT modes, most of gaming consoles require that the NAT device support UPNP or be configured with static NAT.
The main problem is that UPNP is an extremely insecure protocol and is not supported today by Palo Alto Networks firewalls. This is a good thing(tm), generally speaking, as the environment is more secure... but can be somewhat problematic for users in your type of environment.
Have you had this problem with XBox? My understanding is that Palo Alto Networks modified the "teredo" App-ID (which is required for the XBox-Live App-ID to function properly) and added an Application-Layer Gateway (ALG) with the goal of making XBox-Live compatible with DIPP NAT.
If XBox is working in moderate-NAT mode using standard DIPP NAT, then I'd open a feature request and/or Application request to have them do something similar for the Playstation-Network App-ID signature.
03-07-2017 09:16 AM
Hi jvalentine,
So far we've gotten complaints for XBox, PS4, and some gaming applications/platforms on PC. I'd have to look to see if we've gotten any tickets for Nintendo platforms.
Completely agree about UPNP but I'm pretty certain the Cisco ASA 5580 we previously had deployed didn't supported UPNP either (it would be very strange if it had). While I had not personally tested a gaming platform on campus beforehand, the complaints only started after we deployed the Palo Alto.
I'll make sure we get our hands on an XBox so we can test internally with that as well.
03-07-2017 09:22 AM
@jsalmans I've only ever done Active/Passive so I'm not certain if it's possible that your active/active config is messing things up at all. Have you monitored the sessions for the consoles IP at all to see if you are getting the info back, or if the session is time-out for some reason.
03-07-2017 09:23 AM
Check and see if the XBox complaints are due to "strict NAT" or "moderate NAT".
(I should really break out my XBox and try it on my own network... it's been a while).
03-07-2017 09:36 AM
@BPry Yesterday when I was working on it (i.e. playing a game... sometimes I love my job) I had the session browser open on one of the firewalls and I saw active sessions for SIP and some other protocols. Since we just put this thing in place I'm still learning now that we have real traffic passing through and I'm relying on my training course I took a year ago.
@jvalentine I'm going to have someone look into this today. I know we've gotten complaints about Strict NAT on the XBox Live App but that is actually running on a PC. I just browsed the tickets I have and most of them for XBox are talking about disconnects during online games (which may or may not be related) and I haven't found one that specifically mentions the NAT type.
03-07-2017 05:38 PM
@jvalentine I received a few reports back today from Xbox One users and, apparently, their devices are actually showing Type 1 Open NAT which surprised me. If you're correct and there is an ALG in effect then the issue I've been describing may be related only to the XBox Live PC application, various PC games and game platforms that utilize voice chat, and the PS4 consoles.
Apparently the tickets I've been getting for XBox consoles are about getting disconnected from games and party chat... still might be a NAT problem or perhaps a session timeout. I'm trying to get ahold of an XBox One to continue testing and I'll be sure to try to find the session state information for the PS4 tomorrow.
03-08-2017 11:24 AM
@BPry I'm looking at an active PS4 session right now and I'm seeing mostly aged-out on UDP traffic. It looks like the UDP traffic is mostly unknown-udp, stun, and rtp-base. Most of the TCP traffic seems to be ssl and Playstation Network but those are getting tcp-fin, tcp-rst-from-server, and tcp-rst-from-client. I am seeing some aged-out on TCP but I believe that was while we were in the game menu. This is while playing Overwatch which seems to be working just fine, including recieving voice chat.
I think the gaming/voice issues must revolve around P2P games. The XBox is showing Type 1 Open NAT but gets disconnected after only a minute or two of play on a P2P game. The Teredo sessions are showing aged-out but the age varies... some show longer than 30 seconds.
I do see some drops coming in from one of the IPs the Teredo established to as they're hitting an incoming drop rule we have but presumably that would only be happening if the remote device was trying to initiate new connectivity to the console. We don't have incoming ports from the internet open to much of anything except external facing services nor did we on the ASA.
03-08-2017 04:52 PM
Another update.
As a temporary solution I've created an external list with the reserved private IP addresses for affected devices and created a NAT policy set to Dynamic IP with the source as the external list.
Doing this puts the PS4 devices on "Type 2 Moderate" NAT. The XBox One devices still say Type 1 Open NAT but do not seem to be getting disconnected after only a minute or two in a peer to peer game.
I consider this more of a workaround than a long term solution... I'd still like to find out why DIPP isn't working for this and if there is anything to be done about it. We may be doing a network maintenance on Friday for a connected device so I may try to suspend one of the active firewalls to try to determine if this has anything to do with the active/active configuration or how I have the shared NAT set up between them.
04-11-2017 08:00 AM - edited 04-11-2017 08:16 AM
So my workaround apparently did not resolve this completely as many Ubisoft games are still reporting Strict NAT even though the actual console is saying Moderate. These Ubisoft games are also getting disconnected during multiplayer gameplay.
I've got a TAC case open that is currently analyzing some packet captures but haven't heard back from them yet. In the meantime:
Since I don't have multiple IP addresses from the smaller ISP, the next test will probably be to set up a new network in our production environment and assign some of our public IP addresses to be used as a DIPP NAT pool for the PA-200. It is really interesting that the PA-200 hasn't encountered the disconnect issues although it also could be that our test environment doesn't have multiple Playstation or XBox consoles all playing the same games at once.
07-03-2017 06:42 AM - edited 07-03-2017 06:43 AM
We've made some progress on this and so I thought I'd come and do an update to this thread.
We may have identified two separate issues here:
Originally I thought these were all part of the same issue, and originally when we were having problems with a much wider selection of games/consoles perhaps it was, but changing the NAT type to Dynamic IP resolved MOST of issue #1 and issue #2 but not all. We were still having issues with about 4 games that we know of, all AAA titles and 3 of them were from Ubisoft. Some of these games also apparently utilize peer-to-peer for faster multiplayer game play (less latency than having to use the Ubisoft servers as a middle-man) which complicates things further.
After some research I determined I'm not the only one having issues with Ubisoft games... look at their forums and it doesn't take long to see people complaining about their network stack, even on home networks (which, as we all know, are usually much simpler than enterprise and these devices are designed to work at home). Packet captures ended up showing Ubisoft servers attempting to open new connections outside of the IP/ports that had existing NAT sessions. At this point we aren't really sure why, we didn't see the behavoir on our test PA-200, only on our production PA-5060 a/a deployment, and packet capture from the Playstation 4 and from the firewalls show no new session was generated using the NAT IP that the Ubisoft servers were trying to communicate with. I've opened a case with Ubisoft and escalated it with no response yet.
As for issue #2, the TAC engineer I was working with found out from the internal team that DIPP NAT uses a Symmetric NAT implementation. Apparently, this implementation does not play nice with certain applications such as STUN which some of the consoles use to determine their NATed IP and port information which they then communicate to the game servers. It is unclear to me at this point what NAT implementation our previous ASA used... the only documentation I've been referred to so far indicates that the ASA attempts to use the port the internal device originally requested if it is available and, if not, it moves to the next available in the sub-pool.
More on NAT implementations here:
https://en.wikipedia.org/wiki/Network_address_translation#Methods_of_translation
I've reached out to our sales rep and put in a feature request for some additional options with DIPP NAT so that it can use a different NAT implementation, or perhaps a hybrid that is a little less strict than full Symmetric NAT. If anyone else is interested in this I'd encourage you to reach out to your sales team and add your voice for a feature to support this.
07-03-2017 07:56 AM
Something else to note. SInce you are running A/A and a pair of 10g lines.
I remember in the past that there were issues with the PAN devices not playing well with asym traffic and it required a CLI command to be run in order to resolve it. It was resulting in asym traffic being dropped by default. Worth tossing at your TAC engineer and see if that might be a related issue.
Interesting post to say the least. A lot of us in the corporate world dont get to struggle with gaming traffic going over our circuits.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
