Help with understanding OSPF FIB decision

[brannentaylor]

Hi All,  this weekend I had to clean up an asynchronous routing issue, by disabling route advertisement on one side.  I have a WAN, which is an upside down triangle, with A and B at the top, and C at the bottom.  C was routing right, C->B and traffic was coming back through A, A->C.  I disabled advertisements from A, so traffic now flows C->B and B->C.

I'm looking at the OSPF database on C, and it has both routes.  My question is, why is it choosing to go to B (10.28.1.1)?  It appears to me the metrics are the same, it could just as easily choose to go to A (10.24.2.5).  I'd like to engineer it to go in a predictable direction, but need to understand why it's making the choice it is.

Is it as simple as which one it learned first?  Is it random?  We have ECMP turned off.

  1                 10.24.2.5       172.16.0.0/16      type-5 (External)    0x80000094 0x00007BC1    58       
            Options: [Demand Circuit]
            Mask 255.255.0.0, type 1, tos 0 metric: 1, forward 0.0.0.0, tag 0.0.9.116

  1                 10.28.1.1       172.16.0.0/16      type-5 (External)    0x8000010F 0x000085BD   244       
            Options: [Demand Circuit]
            Mask 255.255.0.0, type 1, tos 0 metric: 1, forward 0.0.0.0, tag 0.0.252.0

Thanks,

Brannen

[rmfalconer · Accepted solution]

With OSPF, there is a criteria followed when deciding how to forward to the next hop.

The prefix length is the same from both, they are both type 5 and the metrics are the same. On routers, when everything is a tie like this, OSPF does its own equal cost load balancing. I would assume its the same on the PA since OSPF is standards based, though I've never read through the RFC to verify.

OSPF ECMP would cause asymmetry on about half the traffic. Were you seeing the problem on all traffic or only some of it?

Since these are type 5, where are they coming from? You could probably tweak the redistibution profile to have the prefix from one of the next hops be type-2, which would make it less desirable.

[OtakarKlier]

Hello,

Another option you could try would be to make one path less desierable by using interface metrics. I do this since I have multiple paths so my p2p WAN links are perferred over my VPN backup links. I tend to use large numbers so there wont be any other thing scausing issues, i.e. 10000 metrics, etc.

Hope that helps.

Cheers!

[brannentaylor]

According to PA TAC, if all else is equal, and ICMP is not enabled, traffic will be forwarded out the first interface that learns about the route.

[rmfalconer]

If you want a predictable traffic flow, you'll want to make sure that there are preferred paths at all sites, not just C. Having the PA choose the first learned interface isn't predictable since it would change during a link failure, reboot, maintenance, etc.

How are A and B talking to each other? If there was asymmetry before, you'll probably need to tweak something at those sites to have higher preference on the direct paths.

[brannentaylor]

I agree.  PA's it seems do not operate the same as Cisco routers or switches when it comes to OSPF.  My general philosophy is that you can have routers that can be firewalls and firewalls that can route - but a router is not a firewall and vice versa.

As a temporary measure, we put in static routes with lower AD to be preferred in the RIB.

As a permanent measure, assuming the PA's will cooperate and be RFC compliant, we're going to prefer one path by redistributing in as a Type-1 metric, and a less preferred path redistributing in from that direction as a Type-2 metric.  OSPF at route C should choose the Type-1 over the Type-2.