Possible Reason Why Palo Rejecting SMTP traffic containing attachments.

[ccarter]

Hi All,

I was . Over the past couple of days we have seen a increase in delayed or non delivered emails that contain attachments greater that approx. 3mb that are being rejected or non delivered to the mailbox transport server. It is now a permanent issue and we have identified that the firewall is the cause of the issue but I am unsure how to fix this.

Steps to recreate:

1. Emails sent from external domain to internal domain is received by our email security platform and then forwarded to our on-premise mail server.
2. When the email is forwarded our email server then delivers the mail to the recipients mailbox and this is confirmed as successful.

Issues:

1. A email with an attachment greater than approx. 3mb is hitting our email security platform and then queued for delivery.
2. When the attempt for delivery is made from the email gateway service, we get the following error repeatedly until the retry attempts expire. - Response 451 4.4.2 [internal] connection closed by remote host.

2020-08-06 05:32:57 PM

Response: 451 4.4.2 [internal] connection closed by remote host

2020-08-06 05:33:46 PM

Delivery attempt #2

2020-08-06 05:33:46 PM

Recipient server:

What has been attempted:

1. We have spoken with Microsoft Support and they have advised that once an email reaches the exchange transport service it is immediately passed to the recipient mailbox. Meaning no issues with configuration of transport rules and internal network.
2. It has been identified that these retry attempts of emails that have attachments never reach the exchange server.
3. The rejection error of 451 4.4.2 indicates that there is a network error between the security platform and the on premise exchange server that indicates the firewall as the cause.
4. Tried modifying the security policy by removing the IPS security profile and removing the service and application defaults of SMTP to be any/any with no success.

Help Required:

1. Really need some assistance troubleshooting the traffic coming inbound and why it is being rejected by the firewall when the email contains attachment <3mb.

Thanks in advance,

Chris.

[BPry]

@ccarter,

Are you seeing associated traffic logs indicating the traffic from the security provider at all? I'm always skeptical of security providers blaming the issue on the firewall when you've removed any associated security profile, removed application signatures, and can't see anything in the logs indicating an issue on your end.

Try and get a packet capture before the firewall (you can of course take a PCAP directly on the firewall, just be sure to capture all stages of traffic flow, especially drop) and see if you are even receiving the traffic. That will tell you if it's actually getting stopped at the firewall (maybe it's getting dropped, but from what you've described the security rule you tested with should have really eliminated any firewall concerns). 

[ccarter · Accepted solution]

Well, that was fun.

The cause was discovered at our WAN gateway. Diverting SMTP traffic into a different WAN interface, bang! Emails that were pending for delivery just bulk dumped into everyone's mailbox. 

Dealing with Telstra is an absolute nightmare so this was the only option. There was no way I was ever going to get a technical resource that has the knowledge to troubleshoot this issue. 

I was impressed with the methodical way that Palo Alto Support assisted from discovery through to analysis and packet capture, it was a breath of fresh air. For those that may find this interesting, these were the steps that they took to identify the issue was not with the Palo or internal network. Using the app override function to bypass Layer 7 inspection to rule this out was a very good thing to learn during this process. 

++ Pattern in both packet captures is same that is when layer7 inspection was going on and when we did app-override, ruling out issues with layer7.
++ I suspect network issue based on following observation:
> Merge receive1 and transmit1 packet captures.
> Filter packets with "tcp.port==39775 "
> In merged pcap, at one point starting from frame number: 5708, issue occurs. This packet is from 67.219.246.155(Symantec)-->xxx.xxx.xxx.xxx (Exchange server public IP).
> Next expected sequence number is 2431727156, but this packet never arrives on firewall and such will not reach exchange server.
> Exchange server (or any TCP based application will work based on dup ack to tell the client or server that they didn't receive an expected packet) starts sending dup ack for 2431727156 and this goes on until the end of the entire TCP stream but the missing packet is never seen again from the Symantec side as if it is not getting the "dup acks" from Exchange server. 
> Firewall has these dup acks in its transmit stage which means firewall is sending out towards Symantec side.
> No drops seen in drop1 packet capture for this port.

Next Action Plan:
=============
++ This now needs to be checked between Symantec and Firewall. A packet capture on intermediate devices can tell where did the missing packet go. Simultaneous packet captures can be done on intermediate device and PA-FW.

Next step, get rid of Telstra.