kprakash you have been incredibly helpful with all of this so far, thank you. It seemed like it would be better to branch this to a new topic.
Here is a new / updated image with some more information...
I currently have a single default route for outbound traffic, it is going to ISP A. I have two public blocks from ISP A, both are /28s and I have various NAT statements and rules in the PAN to pass traffic from my LAN side out ISP A via those blocks.
I'm now trying to migrate everything to ISP B and the new /24 that they have given me, and then also announce that /24 to ISP A for failover.
I'm currently having problems with routing - outbound traffic is always using ISP A... which I guess is to be expected given my default route. However, I need to figure out a way to re-do the setup to keep some traffic leaving that circuit.
What I need to do is make it so all traffic sourced from 203.0.113.0/24 leaves via ISP B, unless there is a better/more specific route to the target via ISP A (which will happen because I still have many remote sites on ISP A, I don't want their traffic to have to go out to the internet and reach ISP B if they can instead stay within ISP A's network).
If any traffic comes *in* via ISP A (bound for 203.0.113.0/24), then I'd also like it to LEAVE via ISP A.
I also need to insure that any traffic sourced from 192.0.2.32/28 and 192.0.2.128/28 exits via ISP A, no matter what. (Those blocks belong to ISP A and ISP B won't do anything with them.)
I know this is going to involve Policy Based Forwarding, but I'm not exactly sure how to setup the rules. I tried a PBF rule similar to this:
source zone: untrust
source address: 203.0.113.0/24
Destination address: any
Egress I/F: eth0/3.2000
next hop: 198.51.100.1 (ISP B router)
After committing that rule, I tried a traceroute from 203.0.113.1 (firewall interface) and it was still leaving via ISP A. It seems like PBF doesn't affect traffic from the firewall itself? So then I tried a host attached to eth0/3.2100 (203.0.113.99, with the PAN as default gateway), and even that traffic still left via ISP A.
I know that to insure traffic "sticks" to an interface I will need the Symmetric Return option as well, which is why I just updated to 5.0.6 last night.
Eventually I will have all traffic migrated to 203.0.113.x/24, at which point I can get rid of 192.0.2.32/28 and 192.0.2.128/28, but until I migrate IPs in DNS and get all of the rules setup, I have to keep those blocks around. Even once I migrate to 203.0.113.x/24, I still want traffic from that block to leave via whichever ISP has a better route... since ISP A is giving me routes for all of their sources, I'm hoping that routing/BGP is smart enough to take that path even if I set the default route (with higher weight) to ISP B?
ISP B isn't yet sending me any routes, but I'm trying to see if they can send me local/customer routes along with a default so I don't need to use any static routes.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!