kprakash you have been incredibly helpful with all of this so far, thank you. It seemed like it would be better to branch this to a new topic. Here is a new / updated image with some more information... I currently have a single default route for outbound traffic, it is going to ISP A. I have two public blocks from ISP A, both are /28s and I have various NAT statements and rules in the PAN to pass traffic from my LAN side out ISP A via those blocks. I'm now trying to migrate everything to ISP B and the new /24 that they have given me, and then also announce that /24 to ISP A for failover. I'm currently having problems with routing - outbound traffic is always using ISP A... which I guess is to be expected given my default route. However, I need to figure out a way to re-do the setup to keep some traffic leaving that circuit. What I need to do is make it so all traffic sourced from 203.0.113.0/24 leaves via ISP B, unless there is a better/more specific route to the target via ISP A (which will happen because I still have many remote sites on ISP A, I don't want their traffic to have to go out to the internet and reach ISP B if they can instead stay within ISP A's network). If any traffic comes *in* via ISP A (bound for 203.0.113.0/24), then I'd also like it to LEAVE via ISP A. I also need to insure that any traffic sourced from 192.0.2.32/28 and 192.0.2.128/28 exits via ISP A, no matter what. (Those blocks belong to ISP A and ISP B won't do anything with them.) I know this is going to involve Policy Based Forwarding, but I'm not exactly sure how to setup the rules. I tried a PBF rule similar to this: source zone: untrust source address: 203.0.113.0/24 Destination address: any Egress I/F: eth0/3.2000 next hop: 198.51.100.1 (ISP B router) After committing that rule, I tried a traceroute from 203.0.113.1 (firewall interface) and it was still leaving via ISP A. It seems like PBF doesn't affect traffic from the firewall itself? So then I tried a host attached to eth0/3.2100 (203.0.113.99, with the PAN as default gateway), and even that traffic still left via ISP A. I know that to insure traffic "sticks" to an interface I will need the Symmetric Return option as well, which is why I just updated to 5.0.6 last night. Eventually I will have all traffic migrated to 203.0.113.x/24, at which point I can get rid of 192.0.2.32/28 and 192.0.2.128/28, but until I migrate IPs in DNS and get all of the rules setup, I have to keep those blocks around. Even once I migrate to 203.0.113.x/24, I still want traffic from that block to leave via whichever ISP has a better route... since ISP A is giving me routes for all of their sources, I'm hoping that routing/BGP is smart enough to take that path even if I set the default route (with higher weight) to ISP B? ISP B isn't yet sending me any routes, but I'm trying to see if they can send me local/customer routes along with a default so I don't need to use any static routes.
... View more