Multiple ISPs to one internal server

[DavidFensch]

We're having trouble getting traffic from multiple ISPs to a single internal server working on anything except the ISP that matches our default route. We need to be able to do this because we are trying to migrate from the ISP networks to the /24 we acquired from ARIN. Due to the nature of the business, we have thousands of devices in the field which do not support DNS and must be manually touched to update the IP addresses they report to, so both IPs will need to be live at the same time during the migration which is expected to take about 18 months. We are running a pair of 3250 in active/passive on 9.1.4. 

ISP1 lands on vlan.101, ISP2 lands on vlan.102, ISP3 lands on vlan.998, and a /24 that we recently purchased and are attempting to migrate to lands on vlan.996. Each ISP connection has it's own VR. Each ISP is in its own zone. The internal connection also has a VR, there are routes in the internal connection to 0.0.0.0/0 destined to each of the ISPs with varying costs. There is also a route from the internal VR to the subnets that land on each connection. The lowest cost 0.0.0.0/0 route points at ISP1.

With a DNAT rule configured on a public address on each ISP network, we are able to reach our internal server (http as a test server) on ISP1 only. If we use PBF, we can reach it on whichever ISP PBF is forwarding traffic to. But we are unable to have the server accept connections on all ISPs simultaneously and send traffic back through the correct ISP. 

Running packet captures on the internal server and our 3250, I can verify that traffic does reach the internal server through all four DNAT rules, and the internal server does reply. Reply traffic reaches the 3250, but all replies for sessions that did not originate on ISP are dropped. We see the test client send a SYN, then retransmit the SYN twice. We see the test server send a SYN ACK, then retransmit the SYN ACK twice, then transmit a RST. The "Drop" stage of the 3250 captures the SYN ACKs correctly un-NATed from the public IP the test client tries to connect with to the test client's IP. The "Drop" stage also captures the RST but not un-NATed, with the source being our internal server's private IP and the destination being the test client's public IP. 

"show session id <sessionId>" in the CLI shows the c2s flow as the test client's public IP to the public IP the test client tries to connect to. It shows the s2c flow as our test server's internal IP to the test client's public IP. It shows the correct security and NAT policies being matched. It shows the correct ingress and egress interfaces. 

We have an open case with support which so far has been less than helpful. Their only suggestion so far was to run "set deviceconfig setting tcp asymmetric-path bypass" which did not change anything. 

We tried following the guide here, but this resulted in the firewall believing that the traffic from the test client was ingressing the firewall on the interface to the internal network. This had un-NATed traffic from the client's public IP destined to the our server's public IP ingressing on the inside interface which did not work. 

Does anyone know how we have to configure this to get it working? I'm sure someone has ran into this situation before. 

[OwenFuller]

I think I'm visualizing your setup, but it's a bit complicated to work out in text. Can you provide a network diagram which shows your networks, VLANs, and zones just to be sure?

[OwenFuller]

Have you considered using ECMP instead of PBF? If you weighted your default routes to your three ISPs equally, and used ECMP with the symmetric return option enabled, it might do what you're wanting.

[rmfalconer]

Like @OwenFuller  said, a diagram would be helpful. But if I understand what you've written, as an example, you can get traffic to ingress on ISP2 and hit the server. And from the description, it looks like return traffic would go back through ISP1 since it has the lowest metric from the internal VR to an external VR.

Based on this, the public ingress and egress of the server will be different. This would cause problems with TCP. Did you take any captures from the originating client side? 

[OwenFuller]

I agree with @rmfalconer on the route metric issue. That's why I'm thinking ECMP with the symmetrical return option might be your best bet. I've not tried this with multiple VRs in the setup you have described. You could try it though, and if necessary, move all three ISPs to the same VR.