Starlink Failover: Fast Download Almost NO Upload Speed

[pomologist]

I have a weird problem setting up Starlink as a failover ISP. 

Download speed is blazing fast, but upload speed through the NGF is almost non-existent, 0-1 mbps.  

When I connect to the Starlink router directly, I get download speeds of 50bmps so I know it's not the ISP's fault. 

A troubleshooting ping test from the PA NGF web GUI, yields 50-60 percent ping failure.

Where would you look to start troubleshooting?  

[Claw4609]

Hello,

To confirm information of your bolded statement, are you having issues with upload, download, or both?

Is your routing just a single static route to the Starlink gateway, i.e. is there any route failover on the Palo of any kind setup? Im assuming the WAN interface on the Palo is DHCP with starlink?

While we havent had this with any of our Starlink connections, its possible you are running into an MTU issue. To confirm this if you have a client behind the Palo, in Windows you can send ping packets of different sizes so you may want to play around with that to see if/where it breaks at and adjust the Palos WAN interface MTU accordingly.

Its also possible you are just running into a layer 1 issue. Have you replaced the cable running from the Starlink to the Palo, or possibly tried a different port on the Palo altogether. 

[pomologist]

Thanks.  

I have issues with upload speed only, and only when routing through the Palo.  

Correct, Starlink WAN interface on the Palo is set for DHCP.  

I'll check into the MTU - good idea, thanks!

Trying a different cable also isn't a bad idea at all, I should have done that prior to now!  I'll try that, and also try a different port on the Palo, and report back. 

Thank you!

[Claw4609]

Something I also forgot to add is you should check the global counters as well: How to check global counters for a specific source and destinat... - Knowledge Base - Palo Alto Netw...

[pomologist]

I've done some further testing, and I understand the problem better now. I figured out that it's ONLY when the system is in failover mode that the issue appears. 

If I set a security policy for a test zone, to only allow outbound traffic through the Starlink WAN port, I get perfect upload and download speed.  This works:

 

However if a security policy is set to allow outbound traffic on both the Primary ISP and Starlink interfaces, then as soon as failover occurs, I get inbound traffic but no outbound traffic.  Moreover, as soon as failover occurs, even zones with security policies only allowing traffic on the Starlink interface, cease to upload any traffic.  This does not work upon failover:

 

Here is my router setup.

 

 

Is there anything you'd suggest?

[pomologist]

I tried adjusting MTU size, no change. 

I tried a different cable, no change. 

I checked global counters, and do have one obvious error: "Packet's Dropped: No route for ip multicast"

Further research has revealed that if I connect a regular cellular LTE modem with an ethernet port (I used a Nightgear for testing) in place of the Starlink, using the same firewall interface port, and all the same settings untouched in the firewall, I get perfect upload and download speeds WHILE failed over!

So the issue with no upload while failed over on Starlink, is specific to running failed over on Starlink.  Recall that I can connect with an ethernet cable directly to the Starlink modem/router and get perfect upload speeds. 

Any ideas?

[Claw4609]

Do you see any drops are anything when checking the global counters? How to check global counters for a specific source and destinat... - Knowledge Base - Palo Alto Netw...

If not I would recommend next steps being opening up a TAC case. You're issue with adding and removing the Starlink zone from the security policy doesn't make much sense and if you are able to replicate/resolve the issue by adding and removing the zone from security rules either there are different SSL decryption rules/PFB rules/Security Profiles/Zone Protection profiles applied differently between the two zones or its possible you may be running into a bug of some kind.

In our environment at least we have various locations with Starlink as a failover and we have not noticed these issues, however our main WAN and Starlink WAN, while on different interfaces, we attach the same zone to each.

[pomologist]

Yes I do see some drops.  See the screenshot:

 

[Claw4609]

Closed via RST could be a variety of things, could even just be a security policy not allowing it. Are you seeing all the traffic allowed through in the traffic/url/threat logs? Also is your Starlink WAN interface set to DHCP? And if so do you have that automatically install a default route or do you manually define the default route?

[pomologist]

Yes that's the weird thing.  No traffic is showing up as not allowed in traffic/url/threat logs.  Besides if it were a security policy issue, why can I plug in an LTE modem/router to the very same NGFW port, changing nothing whatsoever, and have perfect upload and download speeds?

Starlink WAN interface is set to DHCP.

I allow it to automatically install a default route, I do not manually define that route.  Is that what you do?

[pomologist · Accepted solution]

So after weeks of going back and forth with Palo Alto tech support on this and getting nowhere, it was finally our old friend ChatGPT that solved it!   On the Starlink interface, AI suggested I change Link Speed from "auto" to "100", and Link Duplex from "auto" to "full".  Problem instantly solved.  Unbelievable. 

It's interesting that I don't have this issue on the 440, the default "auto" settings work just fine on that firewall.  This problem is specific to my 820 only.  If anyone else has this issue, I hope this solves it for you!