topic Re: big disparity between Detailed and Summary logs in General Topics

big disparity between Detailed and Summary logs

LCMember2860 — Tue, 23 Sep 2014 14:09:40 GMT

I ran 2 Panorama reports, using the detailed and summary databases, on application usage over the last 24 hours (simple reports, just top Applications ranked by bytes, no filters)

the results were completely different e.g the figures for web-browsing:

Summary: 720G

Detailed: 3.3T

The detailed figure looks correct, why is the summary figure so out of line?

Obviously we have quite a lot of traffic and logs and running reports using the detailed database takes forever, I'd prefer to use the summary database but the figures are completely wrong.

Re: big disparity between Detailed and Summary logs

hshah — Tue, 23 Sep 2014 14:16:39 GMT

Hi NOC,

Summary and Detailed logs are totally different parameters.

The entries under the detailed traffic logs are purged at a faster rate than the summary traffic logs. The hourly, daily, and weekly summaries are roll ups of 15 minute summaries on an hourly basis and a roll up of the hourly summaries on a daily basis as well as a roll up of the daily summaries on a weekly basis. So as we continue to roll up data the results can become summarized even further. This can lead to greater discrepancies between summarized databases and non-summarized databases.

You can also refer following threads for more details.

Traffic summary databese

Custom reports for Summary vs Detailed logs database

Let me know if this helps.

Regards,

Hardik Shah

Re: big disparity between Detailed and Summary logs

Retired Member — Tue, 23 Sep 2014 14:30:25 GMT

Also you may read

Logging and Reporting Settings Definitions

Re: big disparity between Detailed and Summary logs

LCMember2860 — Tue, 23 Sep 2014 14:52:20 GMT

Assuming the detailed logs haven't been purged I would expect the figure for total bytes to be approximately the same in both databases (for the same query) - we have 8T of log storage with 60% allocated to the detailed logs so this shouldn't be an issue on a query only looking at the last 24 hours.

i.e. if there is a steady 1G of web-browsing in a 15 minute period, this should get rolled up to be 4G for the 1 hour summary, 96G for the daily summary etc - or am I misunderstanding how the summary works?

Even if some of the detailed logs had been purged I would then expect the figure for total bytes to be higher from the Summary database, not lower as we are seeing.

Re: big disparity between Detailed and Summary logs

hshah — Tue, 23 Sep 2014 16:54:49 GMT

Hi NOC,

Can you provide us disk utilization differences on your firewall for sumVsDetailed. That will be more clear.

Regards,

Hardik Shah

Re: big disparity between Detailed and Summary logs

LCMember2860 — Wed, 24 Sep 2014 11:48:18 GMT

Re: big disparity between Detailed and Summary logs

hshah — Wed, 24 Sep 2014 13:12:47 GMT

Hi NOC,

Thanks alot for response, this explains allocation.

I would like to see data for comparison " Even if some of the detailed logs had been purged I would then expect the figure for total bytes to be higher from the Summary database, not lower as we are seeing."

Regards,

Hardik Shah

Re: big disparity between Detailed and Summary logs

LCMember2860 — Wed, 24 Sep 2014 16:12:02 GMT

which data would you like to see?