topic Re: HA queue full in General Topics

HA queue full

SOC_CSG — Mon, 11 Aug 2014 05:44:51 GMT

Hi, im receiving this snmp trap in my Palo Alto (PA-3020 PANOS 6.0.3). Checking the system logs i see each 15 mins this log message "HA-queue-full". Why is this happening?

Re: HA queue full

HULK — Mon, 11 Aug 2014 06:28:00 GMT

Hello COS,

This message usually indicates that the HA buffer is full due to a communication breakdown on the HA1 link. But, i had a previous experience with exactly the same messages due to a USER-ID group mapping problem in HA environment.

So, could you please confirm, if the PAN firewall is configured with group-mapping...? If so, please verify that a valid filter with groups by limiting groups in your include list.

The PAN firewall should configure with a valid include list.

Thanks

Re: HA queue full

SOC_CSG — Mon, 11 Aug 2014 07:04:21 GMT

Yes, we have group-mapping but nobody in ITSystems has changed anything in AD. how could i touch the buffer or something to fix it???

Re: HA queue full

HULK — Mon, 11 Aug 2014 07:38:16 GMT

Hello COS,

Could you please "show system resources" on this PAN firewall, just to ensure resources are available on MP.

Thanks

Re: HA queue full

SOC_CSG — Mon, 11 Aug 2014 07:45:59 GMT

i dont see any strange thing.....i can see the useridd at then end, and the CPU is not very used..

Re: HA queue full

SOC_CSG — Thu, 14 Aug 2014 07:39:56 GMT

any new idea??

Re: HA queue full

HULK — Thu, 14 Aug 2014 08:07:14 GMT

Could you please check mp\useridd.log ( less mp-log userid.log) during the same time, while you have received "HA queue is full" messages.

Re: HA queue full

SOC_CSG — Thu, 14 Aug 2014 08:22:12 GMT

the alarm was reported by SNMP at 10:12:23. dc3 is not active but i dont think this is connected with the problem.

This is the useridd.log

2014-08-14 10:11:38.531 +0200 Error: pan_user_id_win_sess_query(pan_user_id_win.c:1471): session query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()

] ERROR: Login to remote object.

2014-08-14 10:11:43.610 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:11:49.792 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:11:55.867 +0200 Error: pan_user_id_win_sess_query(pan_user_id_win.c:1471): session query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()

] ERROR: Login to remote object.

2014-08-14 10:12:00.956 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:12:06.825 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:12:16.311 +0200 Error: pan_user_id_win_sess_query(pan_user_id_win.c:1471): session query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()

] ERROR: Login to remote object.

2014-08-14 10:12:21.405 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:12:23.783 +0200 Error: pan_user_id_ha_pre_send(pan_user_id_ha.c:549): HA queue (0/50000) left

2014-08-14 10:12:27.400 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()] ERR

OR: Login to remote object.

2014-08-14 10:12:33.392 +0200 Error: pan_user_id_win_sess_query(pan_user_id_win.c:1471): session query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()

] ERROR: Login to remote object.

2014-08-14 10:12:38.473 +0200 Error: pan_user_id_win_log_query(pan_user_id_win.c:1323): log query for dc3.cuatrocruces.com failed: [wmi/wmic.c:200:main()]

tHANKS

Re: HA queue full

HULK — Thu, 14 Aug 2014 08:28:28 GMT

014-08-14 10:12:23.783 +0200 Error: pan_user_id_ha_pre_send(pan_user_id_ha.c:549): HA queue (0/50000) left >>>>>>>>>>>>> This is the problem.

Is there any core file on this firewall..?

Thanks

Re: HA queue full

HULK — Thu, 14 Aug 2014 08:31:57 GMT

Is this a HA Active/Active setup...?

Thanks

Re: HA queue full

SOC_CSG — Thu, 14 Aug 2014 08:39:13 GMT

No, its ACTIVE/PASSIVE. these are the core files but i dont know exactly when the problem started.

admin@FW1(active)> show system files

/opt/dpfs/var/cores/:

total 4.0K

drwxrwxrwx 2 root root 4.0K Aug 6 10:11 crashinfo

/opt/dpfs/var/cores/crashinfo:

total 0

/var/cores/:

total 11M

drwxrwxrwx 2 root root 4.0K Aug 6 23:55 crashinfo

-rw-r--r-- 1 root root 11M Aug 7 00:00 useridd_6.0.3_0.tar.gz

/var/cores/crashinfo:

total 60K

-rw-rw-rw- 1 root root 54K Aug 6 23:55 useridd_6.0.3_0.info

Re: HA queue full

HULK — Thu, 14 Aug 2014 08:42:42 GMT

Since this issue is related to the user-ID daemon, i would request you to open a case with PAN support. We need to take a look into the USER-ID core file.

Thanks

Re: HA queue full

SOC_CSG — Thu, 14 Aug 2014 08:47:00 GMT

I have checked the MONITOR SYSTEM LOG with the core-file date (Aug 6 23:55 crashinfo) and i have found this log.

useridd:exiting because missed too many heartbeats.......i think this was the origin of the error....what could i do????

Re: HA queue full

SOC_CSG — Thu, 14 Aug 2014 08:48:09 GMT

Apparently we arent feeling anything weird....just SNMP TRAPS abut the proble each 15minutes.