SSL-VPN Portal not showing

[gorka_sainz]

Hi,

we have an SSL-VPN portal that has been working for a long time. However, in the last 2-3 weeks, we have experienced the same problem twice. The Portal won't show on the browser, but traffic logs show the traffic being allowed by the corresponding rule. Access to the management IP is working fine as well as all other services.

I have tryed restarting Management and Device Servers, as well as SSL VPN Web Server without any success.

The only thing that solved this issue in both cases was to reboot the entire device.

PAN-OS version is 4.1.2 but we didn't see any relationship between the upgrade and this issue, since it appeared a long time after upgrading, not right after.

I opened a case and sent techsupport file, but meanwhile, I would like to know if anyone has experienced similar issues and how were they resolved.

Thanks in advance!

[sspringer]

There is a known sofware packet buffer leak that can cause similar behavior. When encountered the only resolution is to restart the dataplane or upgrade to 4.1.4.

To check if you are affected by the leak you can run the following command:

> debug dataplane pool statistics

Software Pools

[ 0] software packet buffer 0  :        1/16384    0x8000000022000680

[ 1] software packet buffer 1  :        1/8192     0x8000000022810700

[ 2] software packet buffer 2  :        1/16384    0x8000000023018780

[ 3] software packet buffer 3  :        1/4192     0x8000000025028800

[ 4] software packet buffer 4  :        1/304      0x800000002d538a00

Above, all of the pools are depleted. Here is a example of what a device not affected by the leak:

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x8000000022000680

[ 1] software packet buffer 1  :     8192/8192     0x8000000022810700

[ 2] software packet buffer 2  :    16384/16384    0x8000000023018780

[ 3] software packet buffer 3  :     4096/4096     0x8000000025028800

[ 4] software packet buffer 4  :      304/304      0x800000002d22c880

The software pool stats will also be written to the dp-monitor.log file every 10 minutes so you could find the time of the failure and search for "software packet buffer".

- Stefan

[gorka_sainz]

Thanks for the answer!

Since there is no HA on this envorinment and restarting the data-plane everytime is not viable, I will update the device to 4.1.4.

In case it happens again, I will try with that command to check the output.

Really appreciated the answer.

Best regards!

[mikand]

Please feedback if 4.1.4 solved your issues regarding this or not (not that I currently use SSL-VPN Portal but can be good to know in case similar questions turns up in future 🙂

[jseals]

Yes, or give us some feedback on if your buffers were depleted or not.

Thanks,

Jason Seals

[david_rivas1]

Hello,

I have experimented the same issue on PanOS 4.0.9 in a PAN 2020.

Hardware Pools

[ 0] Packet Buffers            :    57170/57344    0x8000000410000000

[ 1] Work Queue Entries        :   192131/229376   0x8000000417000000

[ 2] Output Buffers            :      999/1024     0x8000000418c00000

[ 3] DFA Result                :     2048/2048     0x8000000419100000    

      DFA Result                :

[ 4] Timer Buffers             :     4092/4096     0x8000000418d00000    

      Timer Buffers             :

[ 5] PAN_FPA_LWM_POOL          :     8192/8192     0x8000000419300000

[ 6] PAN_FPA_ZIP_POOL          :     1024/1024     0x8000000419500000

[ 7] PAN_FPA_BLAST_POOL        :       64/64       0x8000000419700000

Software Pools

[ 0] software packet buffer 0  :        1/16384    0x8000000021800680

[ 1] software packet buffer 1  :        1/8192     0x8000000022010700

[ 2] software packet buffer 2  :        1/8192     0x8000000022818780

[ 3] software packet buffer 3  :        1/4096     0x8000000023820800

[ 4] software packet buffer 4  :        1/256      0x800000002ba24880

[ 5] Pktlog logs               :    10000/10000    0x800000002ca514e0

[ 6] Pktlog threats            :     4999/5000     0x800000002cc6a720

[ 7] Pktlog packet             :     5000/5000     0x800000002cd77080

[sspringer]

Hi David,

The fix for the 4.0 branch will be included in 4.0.11 which is targeted for release late April / early May.

- Stefan

[ppatel]

Hi David,

Software Version 4-0-11 was out April 11, 2012.

If you get a chance please upgrade to that version if NOT already to get rid of the Software Pools depletion bug .

Regards,

Parth

[david_rivas1]

Hello Parth,

Thank you for your advice. We did update the device to PANOS 4.0.11 and there is not any problem with VPN portal until today.

Thank you again.

Regards,

[cs]

Hello,

I have experimented the same issue on PanOS 4.1.6 in a PAN 2050.

[gorka_sainz]

Hi again,

due to other circunstances, we haven't updated the firmware yet (4.1.2), and just yesterday we got the same issue again.

This time I got traces and you got it right, the buffers were depleted:

Software Pools
[ 0] software packet buffer 0  :        1/16384    0x8000000022000680
[ 1] software packet buffer 1  :        1/8192     0x8000000022810700
[ 2] software packet buffer 2  :        1/16384    0x8000000023018780
[ 3] software packet buffer 3  :        1/4096     0x8000000025028800
[ 4] software packet buffer 4  :        1/304      0x800000002d22c880

After a dataplane restart, the SSL-VPN portal started working again.

Software Pools
[ 0] software packet buffer 0  :    16381/16384    0x8000000022000680
[ 1] software packet buffer 1  :     8182/8192     0x8000000022810700
[ 2] software packet buffer 2  :    16384/16384    0x8000000023018780
[ 3] software packet buffer 3  :     4096/4096     0x8000000025028800
[ 4] software packet buffer 4  :      304/304      0x800000002d22c880

I've suggested them to update the software next week, any recommendation on which version to install? guess I will go for the last one, 4.1.6 but I want to be sure that buffer depletion bug is solved in this version, and that we won't encounter new issues (specially SSL-VPN or IPSec related, as we suffered in the past with several 4.x.x versions).

[gorka_sainz]

Hi again,

after upgrading to 4.1.6, the problem disappeared for some days, but since yesterday, the portal is very unstable. It doesn't load properly, you have to refresh the browser several times until the login screen appears. After logging in, the client download won't start, or will stop downloading after a few seconds.

Are there any known related bugs with this version? we are going to reboot the unit this afternoon, but is there any command I can use to get more information about the root cause?

Thanks!

[sspringer]

Hello,

You can check the MP resources with 'show system resources follow'. If the MP is heavy into swap that could cause some problems, there are many memory leak fixes included in 4.1.7.

Also you can check to see if any core files exist on the device 'show system files'.

I would recommend generating a tech support file and opening a case with your support team, provide the time frame that the portal was unstable so historical logs can be reviewed from this time.

- Stefan