OK, here's a basic diagram showing our network layout. Our DCs are actually based off a /8 netmask. (I guess it's easier now to stick with real values rather than the arbitrary /16 from before.) There are 24-bit subnets carved out for servers, DMZ, transient networks, etc. The sites that call that DC home are within that /8 as well, in their own 24-bit subnets. So, the supernet "DC1 /8" encapsulates the management, server, and DMZ of that datacenter, AND all the sites which use that DC as home. (This is indicated by DC1-based sites shown as red, and DC2 as blue.) The red lines show the tunnel interface to DC1; the blue to DC2. Right now, these are set without overlapping routes, so from a site's perspective, all DC1 /8 goes through the DC1 tunnel, and all DC2 /8 goes through the DC2 tunnel. This is why we're dealing with asymmetric routes when communicating between regions -- the return path is necessarily opposite. Again, Internet is all local (for now), so there is really no public IP space in our core network, except to follow the DGW out to the local ISP. Certainly not over any tunnel link. We currently have no load balancing, as Internet traffic is bound to the site, and applications that are popular enough to need distribution have native means (Exchange front-end servers, SQL clustering, etc.) or manual distribution (Exchange datastores distributed by region). It's important to realize, in our case, the datacenters are not mirrors of each other. Some applications are hosted out of one or the other, and used by everyone. Other services are duplicated between the two, but not mirrored. I.E., each DC has its own email, print, and file services, but they are unique and do not have a copy of the others' data or shares (except for DR purposes via backups.) All users go to DC1 for a subset of applications that are homed there, and likewise with DC2. Currently, all VPN access is via DC2. I don't relish the idea of removing the redundant site tunnels, as that means all western users will have to flow from DC1 to DC2 and back again to reach services on the other coast. While the inter-DC links are large and do not have transfer caps, they can still be saturated. On a whim, we tested MTU from coast-to-coast and found that we can get to either DC with 1500 byte packets, but not to a far site. This was calculated as 1428. I don't really understand why. All traffic between sites and DCs is over IPSec links, which should be splitting up and reassembling encapsulated packets as necessary to traverse whatever backbone is available, right?
... View more