About ccarter

Well, that was fun.   The cause was discovered at our WAN gateway. Diverting SMTP traffic into a different WAN interface, bang! Emails that were pending for delivery just bulk dumped into everyone's mailbox.    Dealing with Telstra is an absolute nightmare so this was the only option. There was no way I was ever going to get a technical resource that has the knowledge to troubleshoot this issue.    I was impressed with the methodical way that Palo Alto Support assisted from discovery through to analysis and packet capture, it was a breath of fresh air. For those that may find this interesting, these were the steps that they took to identify the issue was not with the Palo or internal network. Using the app override function to bypass Layer 7 inspection to rule this out was a very good thing to learn during this process.    ++ Pattern in both packet captures is same that is when layer7 inspection was going on and when we did app-override, ruling out issues with layer7. ++ I suspect network issue based on following observation: > Merge receive1 and transmit1 packet captures. > Filter packets with "tcp.port==39775 " > In merged pcap, at one point starting from frame number: 5708, issue occurs. This packet is from 67.219.246.155(Symantec)-->xxx.xxx.xxx.xxx (Exchange server public IP). > Next expected sequence number is 2431727156, but this packet never arrives on firewall and such will not reach exchange server. > Exchange server (or any TCP based application will work based on dup ack to tell the client or server that they didn't receive an expected packet) starts sending dup ack for 2431727156 and this goes on until the end of the entire TCP stream but the missing packet is never seen again from the Symantec side as if it is not getting the "dup acks" from Exchange server.  > Firewall has these dup acks in its transmit stage which means firewall is sending out towards Symantec side. > No drops seen in drop1 packet capture for this port. Next Action Plan: ============= ++ This now needs to be checked between Symantec and Firewall. A packet capture on intermediate devices can tell where did the missing packet go. Simultaneous packet captures can be done on intermediate device and PA-FW.   Next step, get rid of Telstra.  ... View more

I mentioned that it was going to be painfully obvious and it was once someone pointed it out and I was correct.   You have no idea how much I have been trying to find it, I wish I could buy you a beer or something.  ... View more