topic Re: Automate the monitoring and remediation of shifting traffic off a degraded link in Automation/API Discussions

Automate the monitoring and remediation of shifting traffic off a degraded link

stangri-la — Fri, 24 Feb 2023 18:04:14 GMT

Hi all, as the title suggests I'd like to be able to automate the monitoring and remediation of shifting traffic off a degraded link. In my environment, we have two corp DIA circuits for internet-bound traffic which we perform ECMP load balancing on. The problem we have is if one of the two links is degraded and suffering substantial packet loss but not a complete outage, traffic continues to flow across both links.

I had the idea of using Ansible via API call to send pings out of each link and if the packet loss exceeded a certain threshold, then raise the metric of the static route for the affected link so that traffic would only use the healthy link until the affected link returned to normal. However, that's when I realized there's no way to send a ping command via API call so there's really no way to automate the link monitoring for degredation. Am I missing something here or is there some other way to accomplish this?

Re: Automate the monitoring and remediation of shifting traffic off a degraded link

JimmyHolland — Mon, 27 Feb 2023 11:28:10 GMT

Other options include path monitoring, or for even more flexibility in monitoring and path selection the SD-WAN suite of features.

Re: Automate the monitoring and remediation of shifting traffic off a degraded link

stangri-la — Mon, 27 Feb 2023 16:15:42 GMT

Thanks, @JimmyHolland. We already have path monitoring in place but that only helps when there's a complete outage, not when the link is degraded and suffering < 100% packet loss. I realize SD-WAN is meant to address this shortcoming but I wish this was a native feature of the firewalls.

Re: Automate the monitoring and remediation of shifting traffic off a degraded link

nikoolayy1 — Wed, 22 Mar 2023 14:36:05 GMT

What about TCL Expect? As some things are not available through the Palo Alto API and because of this Ansible is not an option you can use my script to ssh to the device and run ping.

https://live.paloaltonetworks.com/t5/general-articles/automating-the-palo-alto-ngfw-s-process-deamon-restarts/ta-p/529328

You will need to play around as you can make bash script triggering the tcp expect script and then the bash script can trigger an Ansible playbook that will dissable an interface or whatever else you want.

https://stackoverflow.com/questions/42353148/expect-within-bash-script