Internet service down

[Jafar_Hussain]

Dear All,

I was facing disconnection issue with our internet firewall, cannot ping to LAN interface, all services related to the LAN interface was down like internet connection, MPLS and DMZ Zone, once I reboot the firewall everything back normally and the Lan interface working fine, we need to investigate to avoid the issue in future.

I have checked the TECH support file and found some critical issue below is the snapshot:-

DateTypeSeverityComponentCategoryObject Issue IDFixed InKBLogValueCommentFeedback

2020-05-04 18:01:10

Issue

Critical

MP

System

(pan_task)

An Out of Memory condition occured and restarted the process

2020-05-04 18:01:10

Issue

Critical

MP

System

(pan_task)

An Out of Memory condition occured and restarted the process

2020-05-04 18:01:09

Issue

Critical

MP

System

800

Out of Memory killer was invoked by the process.

2020-05-04 18:01:10

Issue

Critical

MP

System

800

Out of Memory killer was invoked by the process.

2020-05-05 09:49:55

Issue

Critical

MP

System

ehmon

ehmon restarted unexpectedly however no known issue was identified. Please open a case if further assistance is required.

2020-05-04 18:01:02

Issue

Critical

MP

System

Memory

An Out of Memory condition occured and restarted the process

2020-04-27 13:00:26

Issue

Critical

MP

System

sslvpn

When I checked the system (Message logs) then i found the error:-

May 4 18:01:10 800 kernel: Out of memory: Kill process 3567 (pan_task) score 352 or sacrifice child
May 4 18:01:10 800 kernel: Killed process 3567 (pan_task) total-vm:54612kB, anon-rss:2468kB, file-rss:3780kB
May 5 04:02:01 800 logrotate: ALERT exited abnormally with [1]

Below is the error in ehmon logs:-

2020-05-04 18:01:02.362 +0300 Error: ksysd_handler(kernel_error.c:168): KERNEL_ERROR: oom.errors: 1
2020-05-04 18:01:02.362 +0300 Error: ksysd_handler(kernel_error.c:221): KERNEL_ERROR: Out of memory condition detected, kill process 1
2020-05-04 18:01:09.697 +0300 Error: sysd_pthread_register(sysd_pthread.c:660): sysd thread create failed
2020-05-04 18:01:09.697 +0300 Error: pan_priv_log_initialize(pan_priv_log.c:96): sysd_pthread_register failed
*

Note:- Firewall version is 9.1.1 model - PA-850

Can anyone help me to understand the logs?

[Jafar_Hussain]

[BPry]

@Jafar_Hussain,

Open up a support case and provide the technical support file to TAC. Without all of the log files the only thing anyone on here can do is guess at what the issue could potentially be, TAC needs to look at all of the logs and determine if they can pinpoint what caused pan_task to over-run. You could be running into PAN-133883 but this seems unrelated.

Just as an aside, when you are running early releases of a new major release, you should expect to run into issues like this. If you are running a new release early in its life cycle, I would highly recommend staying as current as possible once you've validated new releases in your Lab environment. Personally right now, I'd recommend 9.1.2 to anyone on the 9.1 branch. 

[MP18]

Let us know if you have open Tac case for this?

Curious to know cause for this issue

[Jafar_Hussain]

@BPry 

Thanks for your reply.

I will check the TAC support file.

[Jafar_Hussain]

@MP18 

I didn't the open case for TAC now.

I have checked all logs and found memory issues. always MP plane goes high it may be caused to affect the firewall performance and stuck some process.

The current version is 9.1.1. I have checked the known issues with data plane and memory if the issue is happening again i will downgrade my firewall up to 9.0.7 which is the recommended version or  I will open a case for TAC.