topic Re: MGMNT Slow and Serching logs slow and Syslog server issue. in General Topics

MGMNT Slow and Serching logs slow and Syslog server issue.

Mohammed_Yasin — Tue, 01 Sep 2020 06:12:27 GMT

Device Model: PA-5220 HA Mode Active-standby

PAN-OS 10.0.0

The questions below as I couldn't find anything on Palo Alto website.

Recently we have upgraded Palo Alto to v10.0.0.

1. Web management interface became very slow and searching logs takes very long time to load.

Kindly advise if there’s any solution for that. Can we disable services of some added unused features, like SDWAN or IoT? Or is there any work-around to make it faster?

2. Integration with ArcSight Syslog server is not working well as logs are not parsed correctly.

Seems the raw data format sent from Palo Alto changed in this version. Kindly advise how to fix this.

Can we change the format to be similar to 9.0.x or 9.1.x format?

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

BPry — Wed, 02 Sep 2020 01:04:59 GMT

@Mohammed_Yasin,

Is this on production equipment, because I really wouldn't be running PAN-OS 10.0 in a production environment. How long ago did you upgrade to 10.0, the background process can take a bit to settle down and things get stable again.

As for the second question, there was additional information put into the syslog messages that could be interfering with how you are extracting the data. You'll really need to look at how you have built the extractors and fields you are using to fix that one.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

Mohammed_Yasin — Wed, 02 Sep 2020 06:35:48 GMT

Thank you for the comment.

Yes Its production, the firewall was upgraded 5 days ago.

its only very slow in wildfire logs appearing in the monitor section rest of the logs are performing usually average.

I have checked the following links for the CEF for Arcsight. but doesn't help me with PAN 10.

https://docs.paloaltonetworks.com/resources/cef.html

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/custom-logevent-format.html

https://community.microfocus.com/t5/ArcSight-User-Discussions/Palo-Alto-Global-Protect-logs-CEF-format/m-p/2801217

https://community.microfocus.com/t5/ArcSight-User-Discussions/bd-p/arcsight-discussions

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

BPry — Thu, 03 Sep 2020 16:43:09 GMT

@Mohammed_Yasin,

I would open a TAC ticket and see if they can see what's going on. PAN-OS 10.0 is a brand-new release, so you'll likely run into things like this until it has more time to bake in the wild and bugs get worked through. I absolutely wouldn't be running 10.0 in a production environment unless you need a feature that has been added into PAN-OS 10.0.

As for the guides you referenced, none of them have been updated for 10.0 yet from a quick glance. Since you have additional fields you may very well have to manually build extractors that function correctly with the new fields for PAN-OS 10.0. This is one of the downsides of upgrading to a new release early.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

MichaelShelton — Tue, 21 Dec 2021 14:51:09 GMT

We are having similar issues trying to pull anything under the monitor tab with the new 10.1 PAN. All the logs take forever to pull. Was way better on 9.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

OtakarKlier — Tue, 21 Dec 2021 21:52:59 GMT

Hello,

I have also seen this slow logs loading/timing out issue. I have found that it loads faster/properly if I use a date filter such as ( receive_time geq '2021/12/15 00:00:00' ) . I suspect a bug in the code but havent searched the release notes to verify.

Also I hope you are not running the base 10.0.0 code as there are issues and you should upgrade to a recommended code version:

https://live.paloaltonetworks.com/t5/customer-resources/support-pan-os-software-release-guidance/ta-p/258304

Regards,

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

epeeler — Tue, 11 Jan 2022 14:51:35 GMT

We're running 10.0.8 release and also seeing this slowness in traffic log filtering which was not present in previous releases. Very frustrating to have to wait up to 5 minutes for filtering results to complete.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

Jose_Antonio — Mon, 17 Jan 2022 11:11:35 GMT

We are running 10.1.3 and facing same issue. Does anyone know how to get rid of this issue??

Thanks you

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

VPsecurities — Fri, 21 Jan 2022 14:55:13 GMT

We have also had this issue ever since upgrading to 10.1.2 - really annoing

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

VPsecurities — Fri, 21 Jan 2022 14:57:16 GMT

Side note: management CPU is aprox 8% and data CPU is less than 3%. So I don't think its a CPU issue. it seems they are missing a searcch index og the DB layout is made in a way that makes this slow. comming from a Checkpoint world makes me wish back to CP.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

SantoshPatelQnA — Wed, 26 Jan 2022 02:35:11 GMT

Same issue on multiple firewall. Did anyone opened TAC case? I do do not have good luck working with TAC on these kind of issues.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

adam.b — Thu, 24 Feb 2022 19:52:31 GMT

Same here. On 10.1.3 currently. Almost unusable slow at times after upgrading to v10.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

SpiroKU — Mon, 07 Mar 2022 14:54:14 GMT

Yeah, we have the same issue. Since moving to PAN OS 10.1 the monitor tab has been unbearably slow. This is on a pair of PA 5250's. Did not see anything on the new release notes for PAN 10.2 to suggest an improvement but I was told that Palo Alto changed the way they log on PAN OS 10.1, so not entirely sure. Some aspects are snappy, but most times its the "cog of death" staring at me on the top right corner! Its really hard to be certain how reliable the data is due to the duration it takes to load data. Page size 20 does not make much of a different either. Can only hope that Palo Alto is aware of the issue and is being tracked. Hope to be back at PAN OS 8.1/9.1 search speed again!

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

PaloSoSlow — Thu, 31 Mar 2022 21:24:25 GMT

Commenting to see if someone does find a fix for this. I have panorama 10 point something and it is really difficult to pull anything out of the traffic logs. What i found works is to keep refining your query, the inital vague query takes forever to render but as i add terms to narrow the log down it seems to get a bit faster each time. I have read the opposite as well, that says the more terms the slower it runs, but that doesn't make sense to me why that would be the case. The VM has 8 cores and 48gb of ram so i am not sure its a hardware issue either.

Still this is basically searching text files? on linux? i mean come on, it shouldn't take 10 minutes for me to run a filter like

(receive_time geq '2022/03/28 00:00:00') and (receive_time leq '2022/03/28 23:59:59') and (port.dst eq 25)

but currently it does.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

SantoshPatelQnA — Thu, 31 Mar 2022 23:53:09 GMT

I have been monitoring this for my multiple customers, no update yet. I do not even think they are working in this direction as 10.0.0 is almost end of life already.

I have still kept all my customers below this version. I will test future versions on small customers and if the performance fixed than recommend to others.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

SpiroKU — Fri, 01 Apr 2022 18:09:51 GMT

I was reading the 10.1.5 release notes that came out. Some mention around logging and possibly improvements but does not give too much specifics. I will possibly update to 10.1.5 and let you know.

PAN-186725 - "Fixed an issue where index creation failed when Elasticsearch attempted to create a new index with a duplicate index name"

PAN-186516 - "Fixed an issue where log queries that included WildFire submission logs returned more slowly than expected"

PAN-184076 - "Fixed an issue on the firewall web interface where logs were delayed when querying for logs."

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

dannon — Fri, 01 Apr 2022 20:36:56 GMT

I've seen the same slowness on the monitor tab since updating to 10.1.x

I plan on installing 10.1.5 on our 5220 this weekend and will update the thread if I see an improvement after.

dannon

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

dannon — Wed, 04 May 2022 18:03:07 GMT

Updating PanOS to 10.1.5 on our 5520 fixed the slowness on the monitor tab for us.

We are currently on 10.1.5-h2 and the slowness has not come back.

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

ChristophWolk — Mon, 09 May 2022 09:25:19 GMT

Hello together,

I have a customer where the problem occured with the upgrade from 10.1.4 to 10.1.5-h1, 10.1.5-h2 did not fix the issue here.. Changing the retentiondays also does not fix the issue for us..

Re: MGMNT Slow and Serching logs slow and Syslog server issue.

Maur73G — Sat, 02 Jul 2022 10:52:56 GMT

Hello, I have the same Monitor Log tab slowness on a Panorama VM after upgrading from PAN-OS 10.0.10 to PAN-OS 10.1.6. I haven't yet upgraded the managed devices (physical and VMs). The PAN-OS 10.1 is not minimally reliable at all and I'm very worried too because PAN-OS 10.0.x EOL it's so close. I don't know if the Bugs founded in the Release Notes by @SpiroKU were fixed just for PA devices and not for Panorama.

I've opened a case with TAC but after a week they haven't found a fix.