topic Re: SSL-VPN Portal not showing in General Topics

SSL-VPN Portal not showing

gorka_sainz — Fri, 30 Mar 2012 10:55:56 GMT

Hi,

we have an SSL-VPN portal that has been working for a long time. However, in the last 2-3 weeks, we have experienced the same problem twice. The Portal won't show on the browser, but traffic logs show the traffic being allowed by the corresponding rule. Access to the management IP is working fine as well as all other services.

I have tryed restarting Management and Device Servers, as well as SSL VPN Web Server without any success.

The only thing that solved this issue in both cases was to reboot the entire device.

PAN-OS version is 4.1.2 but we didn't see any relationship between the upgrade and this issue, since it appeared a long time after upgrading, not right after.

I opened a case and sent techsupport file, but meanwhile, I would like to know if anyone has experienced similar issues and how were they resolved.

Thanks in advance!

Re: SSL-VPN Portal not showing

sspringer — Sat, 31 Mar 2012 01:12:41 GMT

There is a known sofware packet buffer leak that can cause similar behavior. When encountered the only resolution is to restart the dataplane or upgrade to 4.1.4.

To check if you are affected by the leak you can run the following command:

> debug dataplane pool statistics

Software Pools

[ 0] software packet buffer 0 : 1/16384 0x8000000022000680

[ 1] software packet buffer 1 : 1/8192 0x8000000022810700

[ 2] software packet buffer 2 : 1/16384 0x8000000023018780

[ 3] software packet buffer 3 : 1/4192 0x8000000025028800

[ 4] software packet buffer 4 : 1/304 0x800000002d538a00

Above, all of the pools are depleted. Here is a example of what a device not affected by the leak:

Software Pools

[ 0] software packet buffer 0 : 16383/16384 0x8000000022000680

[ 1] software packet buffer 1 : 8192/8192 0x8000000022810700

[ 2] software packet buffer 2 : 16384/16384 0x8000000023018780

[ 3] software packet buffer 3 : 4096/4096 0x8000000025028800

[ 4] software packet buffer 4 : 304/304 0x800000002d22c880

The software pool stats will also be written to the dp-monitor.log file every 10 minutes so you could find the time of the failure and search for "software packet buffer".

- Stefan

Re: SSL-VPN Portal not showing

gorka_sainz — Mon, 02 Apr 2012 07:26:17 GMT

Thanks for the answer!

Since there is no HA on this envorinment and restarting the data-plane everytime is not viable, I will update the device to 4.1.4.

In case it happens again, I will try with that command to check the output.

Really appreciated the answer.

Best regards!

Re: SSL-VPN Portal not showing

mikand — Mon, 02 Apr 2012 10:09:08 GMT

Please feedback if 4.1.4 solved your issues regarding this or not (not that I currently use SSL-VPN Portal but can be good to know in case similar questions turns up in future 🙂

Re: SSL-VPN Portal not showing

jseals — Mon, 02 Apr 2012 18:11:21 GMT

Yes, or give us some feedback on if your buffers were depleted or not.

Thanks,

Jason Seals

Re: SSL-VPN Portal not showing

david_rivas1 — Wed, 11 Apr 2012 15:13:41 GMT

Hello,

I have experimented the same issue on PanOS 4.0.9 in a PAN 2020.

Hardware Pools

[ 0] Packet Buffers : 57170/57344 0x8000000410000000

[ 1] Work Queue Entries : 192131/229376 0x8000000417000000

[ 2] Output Buffers : 999/1024 0x8000000418c00000

[ 3] DFA Result : 2048/2048 0x8000000419100000

DFA Result :

[ 4] Timer Buffers : 4092/4096 0x8000000418d00000

Timer Buffers :

[ 5] PAN_FPA_LWM_POOL : 8192/8192 0x8000000419300000

[ 6] PAN_FPA_ZIP_POOL : 1024/1024 0x8000000419500000

[ 7] PAN_FPA_BLAST_POOL : 64/64 0x8000000419700000

Software Pools

[ 0] software packet buffer 0 : 1/16384 0x8000000021800680

[ 1] software packet buffer 1 : 1/8192 0x8000000022010700

[ 2] software packet buffer 2 : 1/8192 0x8000000022818780

[ 3] software packet buffer 3 : 1/4096 0x8000000023820800

[ 4] software packet buffer 4 : 1/256 0x800000002ba24880

[ 5] Pktlog logs : 10000/10000 0x800000002ca514e0

[ 6] Pktlog threats : 4999/5000 0x800000002cc6a720

[ 7] Pktlog packet : 5000/5000 0x800000002cd77080

Re: SSL-VPN Portal not showing

sspringer — Wed, 11 Apr 2012 19:13:29 GMT

Hi David,

The fix for the 4.0 branch will be included in 4.0.11 which is targeted for release late April / early May.

- Stefan

Re: SSL-VPN Portal not showing

ppatel — Thu, 10 May 2012 00:07:31 GMT

Hi David,

Software Version 4-0-11 was out April 11, 2012.

If you get a chance please upgrade to that version if NOT already to get rid of the Software Pools depletion bug .

Regards,

Parth

Re: SSL-VPN Portal not showing

david_rivas1 — Mon, 14 May 2012 07:42:19 GMT

Hello Parth,

Thank you for your advice. We did update the device to PANOS 4.0.11 and there is not any problem with VPN portal until today.

Thank you again.

Regards,

Re: SSL-VPN Portal not showing

cs — Tue, 15 May 2012 08:47:30 GMT

Hello,

I have experimented the same issue on PanOS 4.1.6 in a PAN 2050.

Re: SSL-VPN Portal not showing

gorka_sainz — Tue, 12 Jun 2012 14:54:46 GMT

Hi again,

due to other circunstances, we haven't updated the firmware yet (4.1.2), and just yesterday we got the same issue again.

This time I got traces and you got it right, the buffers were depleted:

Software Pools
[ 0] software packet buffer 0 :        1/16384    0x8000000022000680
[ 1] software packet buffer 1 :        1/8192     0x8000000022810700
[ 2] software packet buffer 2 :        1/16384    0x8000000023018780
[ 3] software packet buffer 3 :        1/4096     0x8000000025028800
[ 4] software packet buffer 4 :        1/304      0x800000002d22c880

After a dataplane restart, the SSL-VPN portal started working again.

Software Pools
[ 0] software packet buffer 0 :    16381/16384    0x8000000022000680
[ 1] software packet buffer 1 :     8182/8192     0x8000000022810700
[ 2] software packet buffer 2 :    16384/16384    0x8000000023018780
[ 3] software packet buffer 3 :     4096/4096     0x8000000025028800
[ 4] software packet buffer 4 :      304/304      0x800000002d22c880

I've suggested them to update the software next week, any recommendation on which version to install? guess I will go for the last one, 4.1.6 but I want to be sure that buffer depletion bug is solved in this version, and that we won't encounter new issues (specially SSL-VPN or IPSec related, as we suffered in the past with several 4.x.x versions).

Re: SSL-VPN Portal not showing

gorka_sainz — Wed, 04 Jul 2012 07:43:32 GMT

Hi again,

after upgrading to 4.1.6, the problem disappeared for some days, but since yesterday, the portal is very unstable. It doesn't load properly, you have to refresh the browser several times until the login screen appears. After logging in, the client download won't start, or will stop downloading after a few seconds.

Are there any known related bugs with this version? we are going to reboot the unit this afternoon, but is there any command I can use to get more information about the root cause?

Thanks!

Re: SSL-VPN Portal not showing

sspringer — Wed, 18 Jul 2012 17:51:53 GMT

Hello,

You can check the MP resources with 'show system resources follow'. If the MP is heavy into swap that could cause some problems, there are many memory leak fixes included in 4.1.7.

Also you can check to see if any core files exist on the device 'show system files'.

I would recommend generating a tech support file and opening a case with your support team, provide the time frame that the portal was unstable so historical logs can be reviewed from this time.

- Stefan