Scheduled Log Export CSV File Size

Reply
Highlighted
L1 Bithead

Scheduled Log Export CSV File Size

I scheduled a log export and ended up with a ton of small (~30MB) CSV files on my ftp server.  Is there any way to specify the size of each exported log file?

I already tried setting the "Max Rows in CSV Export" (Device>Setup>Management) to the max value (1048576) but that did not appear to change the results of the scheduled log export job, each CSV file still contains just 65536 rows.

Highlighted
L6 Presenter

Re: Scheduled Log Export CSV File Size

There are application limitations to view above the max rows via excel and notepad. Download pspad which should allow you to view the number of rows specified.    

Highlighted
L1 Bithead

Re: Scheduled Log Export CSV File Size

Actually the Excel limitation is now 1 million (since Office 2007) but I wasn't talking about any application limits - In PAN-OS when I schedule a log export job the log files that are ftp'd to my FTP server are all 65536 rows long, since I have several GB of log files per day this size is far too small to be useful (e.g. I will end up with hundreds of files per log per day).   If I could get the entire log file ftp'd at once that would be great but I will settle for multiple files as long as they are reasonable (e.g. a million rows each is fine)

In other words I want the log files FTP'd by the export log file job to be bigger, not smaller.

Highlighted
L6 Presenter

Re: Scheduled Log Export CSV File Size

I've tested this before but I will test again. What PANOS are you currently running?    

L1 Bithead

Re: Scheduled Log Export CSV File Size

4.0.4 - thanks

Here's what ends up my FTP server:

11/03/2011  11:01 AM        24,822,399 PaloAlto001_traffic_2011_11_03_last_calendar_day_0.csv
11/03/2011  11:02 AM        24,815,421 PaloAlto001_traffic_2011_11_03_last_calendar_day_1.csv
11/03/2011  11:04 AM        24,895,218 PaloAlto001_traffic_2011_11_03_last_calendar_day_3.csv
11/03/2011  11:06 AM        24,878,185 PaloAlto001_traffic_2011_11_03_last_calendar_day_4.csv
11/03/2011  11:07 AM        24,822,060 PaloAlto001_traffic_2011_11_03_last_calendar_day_5.csv
11/03/2011  11:08 AM        24,823,101 PaloAlto001_traffic_2011_11_03_last_calendar_day_6.csv

....

Highlighted
L4 Transporter

Re: Scheduled Log Export CSV File Size

Please open a case with support so we can get somebody assigned to investigate this.

Thanks,

Steve Krall

Highlighted
Not applicable

Re: Scheduled Log Export CSV File Size

Hey Steve!

Can you confirm which versions of the product this applies to?

I haven't seen this behavior in 3.0 or 3.1. I think I may have seen it in 4.0. What about 4.1?

If this did in fact change between 3.1 and 4.0, please point out page & paragraph in the Release Notes where this is documented. This (new?) behavior will break our log processing process. To dedicate hours to adjusting said process, I need to be able to document the vendor's change.


Thanks,

MJ

Highlighted
L1 Bithead

Re: Scheduled Log Export CSV File Size

Hi Mark,

     Just to let you know where I stand on this, PA confirmed that the max config rows settings does not affect scheduled export log jobs - those jobs will always use 65k max lines per file.   Not great but I can work around that limitation but I found that the scheduled export jobs would transfer 14 files (with 65k lines each) and then stop each time the job ran.  It should have transferred about 400 files to export my log (based on the volume of log data I have each day)

Palo Alto was able to reproduce this problem in their lab and engineering is currently working on it.

Just FYI...

Case Number   00055856

Highlighted
Not applicable

Re: Scheduled Log Export CSV File Size

Wow! That'd be catastrophic in our environment!!! Thanks for the info.

Can you let me know what version(s) you're seeing this in?

Thanks,

MJ

Highlighted
L1 Bithead

Re: Scheduled Log Export CSV File Size

sw-version: 4.0.4

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!