Issues with credit card processing terminals

[andber]

We have about 560ish credit card terminals from Bank of America, the hardware is from PAX. We are constantly getting reports of transactions failing due to “communication” issues with BoA. It’s not just one terminal and it’s not every single transaction. It’s random terminals and it’s random transactions. BoA is telling us that the transactions fail due to the amount of time the communications take. But when we looked at a failed transaction in our splunk logs, we don’t even see the device reach out to BoA to make a transaction. To us, we are 100% positive this is a PAX hardware issue and not a Palo issue. The problem is, they insist this started happening right after we moved these networks to the Palo. So, you know how that is going. So far what I’ve done is created an application override for tcp/443 and applied it to a rule that has the FQDN’s of the url’s these devices talk to. Has anyone run into similar issues with credit card machines? 

[BPry]

@andber,

Do you log your interzone-default traffic hits or otherwise have a rule that would capture denied traffic from these devices? Do you have netflow configured so that you can actually review the traffic flow after the fact outside of just the logs?

In an instance like this, you are going to have to have a capture that actually proves that the PAX endpoints aren't sending the traffic. If they were all previously working without issue until the PAN hardware was installed, I do have to say that blaming PAN is a valid reaction from PAX. 

[andber]

Yes, we see denies in our logs or other traffic. Yes I've taken captures and can't find any problems during my captures for all 4 stages, tx/rx/firewall/drop. Problem is the randomness. I've sat here for days running captures on some of these devices and they never have an issue. I've tried to use the most problematic devices and let captures run for hours, only to never have an issue. My fear is we will bypass the Palo for a test and the device will never be cured. But the bypass includes a completely different network topology. So that doesn't tell me if the problem is/was actually the Palo. i don't see how it could be the palo if I'm using app override and zero inspections on the traffic. That should pass it through as fast as possible.