topic Re: log forwarding to CDL is generating high traffic volume in General Topics

log forwarding to CDL is generating high traffic volume

Carracido — Tue, 06 Jul 2021 22:27:58 GMT

Dear community!

We are sending logs to cortex data lake and we noticed high traffic volume for the sessions concerning log forwarding, with peaks up to 200GB of data sent.

Do you know if this volume of traffic can be normal?

Also, is there any documentation on how logs are being sent to CDL or how would you troubleshoot this issue?

Thank you in advance!

Re: log forwarding to CDL is generating high traffic volume

OtakarKlier — Wed, 07 Jul 2021 16:20:44 GMT

Hello,

Check your log sources for what they are sending. If its the Firewalls, make sure you are only logging at session end. Also you can filter what is sent to the data lake, if you wish to limit the data, but its a data lake so I say the more the merrier.

Regards,

Re: log forwarding to CDL is generating high traffic volume

LAYER_8 — Wed, 07 Jul 2021 17:28:39 GMT

If you're a Cortex Pro customer as well, the usage by Data Lake will be high. Just QoS the app on the box, give it a set bandwidth if it's causing troubles. (docs)

Re: log forwarding to CDL is generating high traffic volume

Remo — Wed, 07 Jul 2021 19:45:44 GMT

@Bearden

Sorry for this question, but could you explain thia again? I don't understand what you are trying to tell us.

Re: log forwarding to CDL is generating high traffic volume

Remo — Wed, 07 Jul 2021 19:46:03 GMT

Hi @Carracido

Over what time do you see these 200 GB peaks or also where? Is this from the ACC tab on the firewall? In that case you might see the end of a session which was open for days or even weeks and these 200 GB were the result of this very long session.

Re: log forwarding to CDL is generating high traffic volume

Carracido — Thu, 08 Jul 2021 22:58:49 GMT

Yes, those are sessions that stay alive for over a week.

In some sessions I see also that bytes received from CDL are almost the same as bytes sent from the firewall. Is this because of log acknowledgment? If yes, how possible is so big the ack?

Thank you!