- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
03-30-2012 03:55 AM
Hi,
we have an SSL-VPN portal that has been working for a long time. However, in the last 2-3 weeks, we have experienced the same problem twice. The Portal won't show on the browser, but traffic logs show the traffic being allowed by the corresponding rule. Access to the management IP is working fine as well as all other services.
I have tryed restarting Management and Device Servers, as well as SSL VPN Web Server without any success.
The only thing that solved this issue in both cases was to reboot the entire device.
PAN-OS version is 4.1.2 but we didn't see any relationship between the upgrade and this issue, since it appeared a long time after upgrading, not right after.
I opened a case and sent techsupport file, but meanwhile, I would like to know if anyone has experienced similar issues and how were they resolved.
Thanks in advance!
03-30-2012 06:12 PM
There is a known sofware packet buffer leak that can cause similar behavior. When encountered the only resolution is to restart the dataplane or upgrade to 4.1.4.
To check if you are affected by the leak you can run the following command:
> debug dataplane pool statistics
Software Pools
[ 0] software packet buffer 0 : 1/16384 0x8000000022000680
[ 1] software packet buffer 1 : 1/8192 0x8000000022810700
[ 2] software packet buffer 2 : 1/16384 0x8000000023018780
[ 3] software packet buffer 3 : 1/4192 0x8000000025028800
[ 4] software packet buffer 4 : 1/304 0x800000002d538a00
Above, all of the pools are depleted. Here is a example of what a device not affected by the leak:
Software Pools
[ 0] software packet buffer 0 : 16383/16384 0x8000000022000680
[ 1] software packet buffer 1 : 8192/8192 0x8000000022810700
[ 2] software packet buffer 2 : 16384/16384 0x8000000023018780
[ 3] software packet buffer 3 : 4096/4096 0x8000000025028800
[ 4] software packet buffer 4 : 304/304 0x800000002d22c880
The software pool stats will also be written to the dp-monitor.log file every 10 minutes so you could find the time of the failure and search for "software packet buffer".
- Stefan
04-02-2012 12:26 AM
Thanks for the answer!
Since there is no HA on this envorinment and restarting the data-plane everytime is not viable, I will update the device to 4.1.4.
In case it happens again, I will try with that command to check the output.
Really appreciated the answer.
Best regards!
04-02-2012 03:09 AM
Please feedback if 4.1.4 solved your issues regarding this or not (not that I currently use SSL-VPN Portal but can be good to know in case similar questions turns up in future 🙂
04-02-2012 11:11 AM
Yes, or give us some feedback on if your buffers were depleted or not.
Thanks,
Jason Seals
04-11-2012 08:13 AM
Hello,
I have experimented the same issue on PanOS 4.0.9 in a PAN 2020.
Hardware Pools
[ 0] Packet Buffers : 57170/57344 0x8000000410000000
[ 1] Work Queue Entries : 192131/229376 0x8000000417000000
[ 2] Output Buffers : 999/1024 0x8000000418c00000
[ 3] DFA Result : 2048/2048 0x8000000419100000
DFA Result :
[ 4] Timer Buffers : 4092/4096 0x8000000418d00000
Timer Buffers :
[ 5] PAN_FPA_LWM_POOL : 8192/8192 0x8000000419300000
[ 6] PAN_FPA_ZIP_POOL : 1024/1024 0x8000000419500000
[ 7] PAN_FPA_BLAST_POOL : 64/64 0x8000000419700000
Software Pools
[ 0] software packet buffer 0 : 1/16384 0x8000000021800680
[ 1] software packet buffer 1 : 1/8192 0x8000000022010700
[ 2] software packet buffer 2 : 1/8192 0x8000000022818780
[ 3] software packet buffer 3 : 1/4096 0x8000000023820800
[ 4] software packet buffer 4 : 1/256 0x800000002ba24880
[ 5] Pktlog logs : 10000/10000 0x800000002ca514e0
[ 6] Pktlog threats : 4999/5000 0x800000002cc6a720
[ 7] Pktlog packet : 5000/5000 0x800000002cd77080
04-11-2012 12:13 PM
Hi David,
The fix for the 4.0 branch will be included in 4.0.11 which is targeted for release late April / early May.
- Stefan
05-09-2012 05:07 PM
Hi David,
Software Version 4-0-11 was out April 11, 2012.
If you get a chance please upgrade to that version if NOT already to get rid of the Software Pools depletion bug .
Regards,
Parth
05-14-2012 12:42 AM
Hello Parth,
Thank you for your advice. We did update the device to PANOS 4.0.11 and there is not any problem with VPN portal until today.
Thank you again.
Regards,
05-15-2012 01:47 AM
Hello,
I have experimented the same issue on PanOS 4.1.6 in a PAN 2050.
06-12-2012 07:54 AM
Hi again,
due to other circunstances, we haven't updated the firmware yet (4.1.2), and just yesterday we got the same issue again.
This time I got traces and you got it right, the buffers were depleted:
Software Pools
[ 0] software packet buffer 0 : 1/16384 0x8000000022000680
[ 1] software packet buffer 1 : 1/8192 0x8000000022810700
[ 2] software packet buffer 2 : 1/16384 0x8000000023018780
[ 3] software packet buffer 3 : 1/4096 0x8000000025028800
[ 4] software packet buffer 4 : 1/304 0x800000002d22c880
After a dataplane restart, the SSL-VPN portal started working again.
Software Pools
[ 0] software packet buffer 0 : 16381/16384 0x8000000022000680
[ 1] software packet buffer 1 : 8182/8192 0x8000000022810700
[ 2] software packet buffer 2 : 16384/16384 0x8000000023018780
[ 3] software packet buffer 3 : 4096/4096 0x8000000025028800
[ 4] software packet buffer 4 : 304/304 0x800000002d22c880
I've suggested them to update the software next week, any recommendation on which version to install? guess I will go for the last one, 4.1.6 but I want to be sure that buffer depletion bug is solved in this version, and that we won't encounter new issues (specially SSL-VPN or IPSec related, as we suffered in the past with several 4.x.x versions).
07-04-2012 12:43 AM
Hi again,
after upgrading to 4.1.6, the problem disappeared for some days, but since yesterday, the portal is very unstable. It doesn't load properly, you have to refresh the browser several times until the login screen appears. After logging in, the client download won't start, or will stop downloading after a few seconds.
Are there any known related bugs with this version? we are going to reboot the unit this afternoon, but is there any command I can use to get more information about the root cause?
Thanks!
07-18-2012 10:51 AM
Hello,
You can check the MP resources with 'show system resources follow'. If the MP is heavy into swap that could cause some problems, there are many memory leak fixes included in 4.1.7.
Also you can check to see if any core files exist on the device 'show system files'.
I would recommend generating a tech support file and opening a case with your support team, provide the time frame that the portal was unstable so historical logs can be reviewed from this time.
- Stefan
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!