About locampo

+1 on needing to know the shutdown vs alarm threshold. Does the shutdown action need to be enabled, or is it just always there? At what temperature does the system shut itself down vs just fire an alarm? ... View more

But if it shutdown, it would be shutdown. Right? Maybe not for a chassis system where it's possible you could be connected to the MPC and it could tell you that one of the NPC or DPC cards shutdown due to overheating. But if the chassis itself or MPC overheats...    I am wondering if Shutdown:False instead refers to the system being set to shutdown upon a temperature threshold. Not sure if this is configuration.    In either case though... it's not clear at what temp the shutdown would happen. I assume that it's not the same as the alarm threshold.  ... View more

@OtakarKlier Yes, I have also seen instances were stacking switches introduces an additional logical single point of failure. It's tricky, because stacking switches has many other advantages (like troubleshooting, reducing management overhead, and reducing the number of connections back to the core). But indeed, I've seen bugs where something in the management plane fails and then causes the dataplane to progressively fail, and in a stack scenario that failure is stack-wide.   I'm surprised that we are still managing switches the way we are in this day and age. In the wireless arena, everything is now centrally managed in some kind of controller or virtual controller based system. For a long time now, deploying a new access point has been plug it in, let it get a DHCP address and find the controller, then you provision it in the controller and you're pretty much done. Why this has not been the case for switching is beyond me.   In any case, standalone switches would still need to be physically swapped and the users still down in the mean time. I think the best strategy we're leaning toward is to deploy everystack with an N+1 switch that we just manually TREAT as a spare. We'll down all the ports so it can't be used, and in the event of a single switch physically failing we can SSH into the stack, make that member the member number that failed and let it pick up the config. Then we can tell someone on site to physically move cables 1:1 from the dead switch to the spare and at that point we're back up and able to wait until we can find time to come an properly remediate.   The wireless backup option for the endpoints is a good idea... but again, I think the additional wireless noise would kill the entire network. We'd need some way to keep the wireless NICs off and on standby until needed, but even then if the density is high the WLAN might not support all those wired clients in addition to the normal load of wireless clients. ... View more

@RobinClayton That's good advice. However, 802.1x and NAC wouldn't completely remove the need to occasionally have specific port configurations for things like, oh say, random medical devices that for some reason still can't do autonegotiation even with up to date firmware as of 2018 and so need static speed and duplex settings to come up. But it would eliminate 90+% of the port-specific configurations, making those edge cases a potentially more manageable problem.   Thanks for the thoughts. ... View more

Nobody has dictated it, per se, but when a switch fails it's a "drop everything else" class emergency. When this happens on the weekends or afterhours, there's no one here to address it and my employer is a hospital, so it's a a 24x7x365 operation where anytime something breaks the words "patient safety" quickly get used.   In our current strategy, one of the two network engineers working M-F, 9-5, supporting all (close to 200) the switches (not to mention 30+ firewalls and ~300 access points across 28 or so physical locations) has to be available to physically replace the failed unit and then console into the stack to run the commands necessary to join the new switch to the stack, then pray that the operation doesn't impact the rest of the stack, then confirm the switch came up as the correct member number with the right config, then manually reconnect cables (being cafeful that each patch cable goes into the same port in the new switch as the old, lest some device requiring special VLAN or port configurations to end up on a standard port).    I was wondering if someone hadn't built a stacking framework that allowed for running a warm spare switch in the stack that would automatically take over the place and config of a failed member in the stack. That would reduce the repair procedure to just the moving cables part... which could easily be explained to and completed by a non-network engineer, say at 3am on a Sunday. N+1 is a concept that is ubiquiteous in almost every other area of infrastructure, except for access layer switches (as far as I've seen). ... View more

@OtakarKlier Hello there,   Yes, this is essentially what we do now. We have a stock of spare switches for each make and model and when one fails we load up the matching firmware and then rejoin it to the stack (on Comware this does not require restoring a config, you simply add it to the stack in place of the missing member). Our environment at the moment is all over the place in terms of code levels (bad, I know, but not uncommon), so it would not be possible to maintain a spare for each code level until we normalize our codebase across the organization.   Also, aside from needing to load up code, our current strategy relies on a network engineer being available to come into the office and physically install a switch. I was hoping that others might have found creative solutions to that requirement.   Your point about wireless is interesting. The only issue I see there is that making each wired host also a wireless host would create a huge additional load to our wireless infrastructure and wireless capacity is already an issue in most places. It would be cool if there were a solution that could maintain a preconfigured and tested wireless NIC that would remain cold but automatically turn itself on and connect if the wired network failed (like if it were unable to ping some IP). ... View more

@LukeBullimore Spanning Tree will protect against loops and therefore allows multiple paths in a switch fabric, and therefore redundancy between the access layer and the core. It does not, however, allow for redundancy at the access layer itself... you know, what endpoints actually physically connect to. Nothing I can think of really does or could, except dual connections between each endpoint and multiple switches, but that would require somekind of link aggregation and thus double the number of drops/patches/switches, and would therefore be cost-prohibitive to all but those with the heftiest of coffers.   But, since this is the weakest link in the chain of the network's armor and the one that actually serves the users themselves, I was wondering if maybe some solutions out there that worked around this problem. ... View more

That's about where we are at if no one is able to offer up a better idea. It will be a real headache for helpdesk/support, but it may be we have no other option. ... View more

Thanks for the response, @pulukas.   The way I've done this in the past has been strictly using PBF, but I come from an SMB background. In those situations, I was using PBF to provide failover across 2-3 connections to a central office in a hub-and-spoke model. In that kind of a scenario, PBF makes sense (if you're not already comfortable and confident with using a dynamic protocol).    With this project requiring a full mesh topology, the sheer number of paths at each site and total number of routes and monitors seems like a practical limit for PBF. But I was eager to get feedback from others like you who've actually used OSP outside of a lab environment. As far as BGP, wouldn't that be overkill here? The only place I've encountered BGP in use is at a previous MSP I worked for where they needed BGP at their edge to provide seemless redundancy across their two ISP connections for the level of uptime they needed. From what I do know of BGP, it seems like the most complex way to go from routing in the organization.   What about other dynamic protocol options? EIGRP, for example? OSPF was my initial thought just because it seemed simpler, but I'm not very familiar with the realworld pros and cons of these various options. Could that be somewhere inbetween OSPF and BGP in terms of complexity (and required expertise) vs control and reliability?   EDIT: Sorry. I see now that PAN doesn't support EIGRP... so, I suppose my options are RIP (which is just a bad idea, right?), OSPF, or BGP.   As far as link costs, that is exactly my concern with using any protocol. Not having done this before, I'm not quite sure exactly how I would want to weigh my paths to prevent assymetric routing and/or other odd pathing issues. I don't want to save configuration time just to have to spend more time troubleshooting issues with the protocol, you know?   Thanks again for taking the time to help with this.   ... View more

Nobody? ... View more

Thanks.    So, I got a response from a new engineer who confirmed the route metric was not involved. It turns out that the prior engineer also, after changing the metric, cleared all existing DHCP sessions from the FW. It looks like that was what fixed the issue, not changing metrics.    After looking closer at logs, it appears we had our layer 2 links connectivity drops for our metro-ethernet connectionsat multiple site locations. This caused the static route to be removed from the forwarding table at that time (because the associated interface went offline). My understanding is that, when that happened, the existing sessions from the relay agents to the DHCP server were no longer valid and were rebuilt, which did a new route look up to determine the destination zone for the sessions. The traffic was then routed (and NAT'd) out our outside interface via the default route. Once the links came back up for the metro-ethernet interfaces, the route was re-added to the forwarding table, however this did not cause the existing sessions to get rebuilt. The packets began to get routed appropriately, but since the session zone information did not change, the sessions were incorrect (inside>outside) and this was causing the traffic to still get NAT'd. Essentially, the observations we saw that didn't make sense were due to the fact that the sessions were no longer valid for the actual path the traffic was taking. Clearing the sessions was what fixed the issue.   I understand this is how stateful firewalls function and is not a PANOS-specific problem, yet it does seem odd to me that traffic no longer matching its parent session would not trigger the FW to reassess or clear the session in some way. Stateful firewalls have been around a long time, but no one has come up with a good way to mitigate these kinds of issues? Wondering if anyone knows of any technologies and/or configurations that could have helped prevent this.      ... View more

That was my thought, too.   The only thing I don't understand is how Threats and/or Applications would apply to the basic routing engine of the FW. My understanding was that packet processing in PANOS is that route look up happens early in the process, but actual forwarding in the last step, so I guess it is possible that a bug somewhere in the middle could cause the issue. That may even explain the disconnect between what we're observing in the monitor (which says the traffic should get routed out the intended/correct interface, following the specific route) vs packet captures (which show traffic being forwarded out the wrong/unintended interface, following the default route). A route lookup from the CLI for the destination IP showed the traffic should still have been hitting the route that it used to be hitting, so the lookup process itself is working... something else is causing the actual forwarding decision to differ from the FIB lookup result. A bug in the application engine could also explain why this issue seems to only be affecting DHCP traffic and not other traffic. Pings, for example, were working and being routed normally at the same time the DHCP traffic was being mis-routed.   We have an open case with support, but they don't seem to think this is a bug or anything unusual. The tech was telling us that this is the expected behavior... (?!) ... and that if the default route has a lower metric it would win out over the more specific route with the higher metric. Despite the fact this was only affecting some (not all) traffic and the fact that it was working for years as configured. I'll have to try and get the case escalated.   I'd still like to see a formal document for how PANOS makes forwarding decisions. There's some very nice documentation on packet processing (https://live.paloaltonetworks.com/t5/Learning-Articles/Packet-Flow-Sequence-in-PAN-OS/ta-p/56081) but the part dealing with actual forwarding decisions / routing table lookups is glossed over quite non-specifically. Probably because basic TCP/IP routing is assumed to be standard and known information. ... View more