topic Re: Scheduled Log Export CSV File Size in General Topics

Scheduled Log Export CSV File Size

PeteS — Thu, 03 Nov 2011 15:18:39 GMT

I scheduled a log export and ended up with a ton of small (~30MB) CSV files on my ftp server. Is there any way to specify the size of each exported log file?

I already tried setting the "Max Rows in CSV Export" (Device>Setup>Management) to the max value (1048576) but that did not appear to change the results of the scheduled log export job, each CSV file still contains just 65536 rows.

Re: Scheduled Log Export CSV File Size

gswcowboy — Thu, 03 Nov 2011 17:19:27 GMT

There are application limitations to view above the max rows via excel and notepad. Download pspad which should allow you to view the number of rows specified.

Re: Scheduled Log Export CSV File Size

PeteS — Thu, 03 Nov 2011 17:24:54 GMT

Actually the Excel limitation is now 1 million (since Office 2007) but I wasn't talking about any application limits - In PAN-OS when I schedule a log export job the log files that are ftp'd to my FTP server are all 65536 rows long, since I have several GB of log files per day this size is far too small to be useful (e.g. I will end up with hundreds of files per log per day). If I could get the entire log file ftp'd at once that would be great but I will settle for multiple files as long as they are reasonable (e.g. a million rows each is fine)

In other words I want the log files FTP'd by the export log file job to be bigger, not smaller.

Re: Scheduled Log Export CSV File Size

gswcowboy — Thu, 03 Nov 2011 17:30:22 GMT

I've tested this before but I will test again. What PANOS are you currently running?

Re: Scheduled Log Export CSV File Size

PeteS — Thu, 03 Nov 2011 17:31:11 GMT

4.0.4 - thanks

Here's what ends up my FTP server:

11/03/2011 11:01 AM        24,822,399 PaloAlto001_traffic_2011_11_03_last_calendar_day_0.csv
11/03/2011 11:02 AM        24,815,421 PaloAlto001_traffic_2011_11_03_last_calendar_day_1.csv
11/03/2011 11:04 AM        24,895,218 PaloAlto001_traffic_2011_11_03_last_calendar_day_3.csv
11/03/2011 11:06 AM        24,878,185 PaloAlto001_traffic_2011_11_03_last_calendar_day_4.csv
11/03/2011 11:07 AM        24,822,060 PaloAlto001_traffic_2011_11_03_last_calendar_day_5.csv
11/03/2011 11:08 AM        24,823,101 PaloAlto001_traffic_2011_11_03_last_calendar_day_6.csv
....

Re: Scheduled Log Export CSV File Size

skrall — Wed, 16 Nov 2011 17:49:15 GMT

Please open a case with support so we can get somebody assigned to investigate this.

Thanks,

Steve Krall

Re: Scheduled Log Export CSV File Size

markjx — Thu, 22 Dec 2011 11:53:18 GMT

Hey Steve!

Can you confirm which versions of the product this applies to?

I haven't seen this behavior in 3.0 or 3.1. I think I may have seen it in 4.0. What about 4.1?

If this did in fact change between 3.1 and 4.0, please point out page & paragraph in the Release Notes where this is documented. This (new?) behavior will break our log processing process. To dedicate hours to adjusting said process, I need to be able to document the vendor's change.

Thanks,

Re: Scheduled Log Export CSV File Size

PeteS — Thu, 22 Dec 2011 12:37:48 GMT

Hi Mark,

Just to let you know where I stand on this, PA confirmed that the max config rows settings does not affect scheduled export log jobs - those jobs will always use 65k max lines per file. Not great but I can work around that limitation but I found that the scheduled export jobs would transfer 14 files (with 65k lines each) and then stop each time the job ran. It should have transferred about 400 files to export my log (based on the volume of log data I have each day)

Palo Alto was able to reproduce this problem in their lab and engineering is currently working on it.

Just FYI...

Case Number 00055856

Re: Scheduled Log Export CSV File Size

markjx — Thu, 22 Dec 2011 14:51:44 GMT

Wow! That'd be catastrophic in our environment!!! Thanks for the info.

Can you let me know what version(s) you're seeing this in?

Thanks,

Re: Scheduled Log Export CSV File Size

PeteS — Thu, 22 Dec 2011 14:55:07 GMT

sw-version: 4.0.4