topic Re: Expedition Panorama Log Collector Forwarding - ML-Learning in Expedition Discussions

Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Wed, 11 May 2022 11:43:09 GMT

Hello,

I have setup expedition as an rsyslog server and configured the Panorama Log collector to forward traffic logs to the expedtion server.

(for now i am filtering on panaorma to filter on only the device logs I am currently interested in)

The logs are all collected inside a single file under the folder

/palogs/panoramip/daily_log.csv.

I have imported panorama inside Expedition. My firewalls have global defined policies for all devices inside the device group an specific firewall rules depending on location.

The global firewall policy has rules that have an any in the source and I would like to create more specific rules for the location.

Since all traffic logs for all devices are in a single file right now I do not know if this is possible.

Some question I have, where do i have to enable the M-Learning on the panorama device or on the firewall device itself?

Will expedition be able to make difference between logs comming from Location1 and Location2, both have the same rule but source traffic will be different.

Is there a way to create seperate log files per device, I think you will have to do this in the rsyslog file.

So I want a different log analysis per device group for the same rule.

Hoping this explains a little bit what I want to achieve if not don't hesitate to aks for more info, screenshot, logs, ...

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

lychiang — Wed, 11 May 2022 16:47:18 GMT

@zGomez To answer your questions:

Where do i have to enable the M-Learning on the panorama device or on the firewall device itself?

You will retrieve all devices from Panorama device tab ,and when you click on right upper corner icon "show all devices" , you will see all firewall devices thats managed by the panorama, and go to the specific firewall device , click on "M. Learning" tab , and process the logs there.

Will expedition be able to make difference between logs coming from Location1 and Location2, both have the same rule but source traffic will be different.

ML will analyze the logs based on serial number of the device , so if you only want to analyze specific device, you can define that in the "log connector" that only select the specific firewall under specific Device Group.

Is there a way to create separate log files per device, I think you will have to do this in the rsyslog file.

So I want a different log analysis per device group for the same rule.

When you make the Expedition as syslog server, it should automatically create separate folder based on firewall IP to contain logs that came from separate firewall devices.

For any questions related ML Process, I would suggest you to review the tutorial video here:

https://www.youtube.com/playlist?list=PLD6FJ8WNiIqXAfspousboWn6AllrOWVMi

Module 4 will show you how to import logs from firewall includes making Expedition as syslog server

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Thu, 12 May 2022 08:58:02 GMT

Hi Lychiang,

Thank you for your feedback.

I have setup expedition as syslog server but i am fowarding traffic logs from panorama not with a log profile on the firewall's itself. Since all log are already forwarded to Panorama I am using this approach.

So I have a single log file with all traffic logs from all devices. But indeed I can see the log files includes the serial number of the device so maybe not an issue that I have a single file.

I will have to play with the log collector and see the outcome of this.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Tue, 17 May 2022 12:06:46 GMT

I have my syslog setup to receive traffic logs from panorama device that forwarding me all firewall logs of my managed devices.
The objective is to create a seperate log file based on the firewall hostname include in the log files( important not the same as the sending hostname, ip of the logs)

Example log file:

2022-05-16T14:26:01+00:00 PANORAMA 1,2022/05/16 14:26:01,012001050280,TRAFFIC,end,2049,2022/05/16 14:25:57,10.51.10.8,10.52.2.1,0.0.0.0,0.0.0.0,MPLS ,,,snmpv1,vsys1,VPN,trust,tunnel.2001,ethernet1/2.900,LogToPanorama,2022/05/16 14:25:57,76592,1,58824,161,0,0,0x4019,udp,allow,16 90,785,905,18,2022/05/16 14:25:25,1,any,0,578843949,0x8000000000000000,10.0 .0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,9,9,aged-out,29,3090,3092,0,,FIREWALLNAME,from-policy,,,0,,0,,N/A,0,0,0,0,,0,0,,,,,,,

This is currenlty in my syslog file.

$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%HOSTNAME%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_ca lendar_day.csv"
*.* -?DynaTrafficLog

Anybody that can help on configuring syslog for this?

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

lychiang — Tue, 17 May 2022 16:26:38 GMT

Are you not seeing the log saved in /palogs/{yourpanoramaIP}/ folder ?

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Fri, 20 May 2022 07:55:27 GMT

Yes i receive the logs that is not the issue.

I have my syslog setup to receive traffic logs from panorama device that forwarding me all firewall logs of my managed devices.
The objective is to create a seperate log file based on the firewall hostname include in the log files( important not the same as the sending hostname, ip of the logs)

Example log file:
2022-05-16T14:26:01+00:00 PANORAMA 1,2022/05/16 14:26:01,012001050280,TRAFFIC,end,2049,2022/05/16 14:25:57,10.51.10.8,10.52.2.1,0.0.0.0,0.0.0.0,MPLS ,,,snmpv1,vsys1,VPN,trust,tunnel.2001,ethernet1/2.900,LogToPanorama,2022/05/16 14:25:57,76592,1,58824,161,0,0,0x4019,udp,allow,16 90,785,905,18,2022/05/16 14:25:25,1,any,0,578843949,0x8000000000000000,10.0 .0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,9,9,aged-out,29,3090,3092,0,,FIREWALLNAME,from-policy,,,0,,0,,N/A,0,0,0,0,,0,0,,,,,,,

This is currenlty in my syslog file.

$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%HOSTNAME%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_ca lendar_day.csv"
*.* -?DynaTrafficLog

This stores the files in /palogs/IP-address/panorama_date_.csv

All traffic logs will be in the same file. I want seperate logs based on FIREWALLNAME (column 53 in the log line) this can be different values.
I could do this using if msg contains "FIREWALLNAME" but this is not really dynamic and hard to keep up with.
Can i do this using template?

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

lychiang — Fri, 20 May 2022 15:32:42 GMT

@zGomez , What is the purpose to separate logs based on the firewall IP, if you want to only do ML on specific firewall log, you can define log connector in the project that only select that specific firewall, then expedition will only do ML on the device that's defined in the log connector . For example , you can only select specific firewall in DG1 like below screenshot

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Mon, 23 May 2022 15:37:52 GMT

Hi Lychiang,

The issue I am facing is that under the device config M.learning i am not seeing all the files that contain that specific firewall.

I only see 1 file the first one created.

there are several files in the directory with the same rights as other file, containg serial and firewallname. i should see 11 files in the above screenshot.

The log is currently storing log files for 4 different firewalls.

If i take another device also in the log file i can see all of the file in /palogs.

When i manual copy the log files by filtering on LInux containing the first firewall logs i can see the file.

so it is before I create a project log-collector.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

lychiang — Mon, 23 May 2022 16:08:38 GMT

The first screenshot , was it in the panorama or in the specific firewall ? In Device-> Panorama, you will click on the retrieved connected device first , so it will retrieve all the firewall that's managed by the panorama, 2nd, you click on the right upper corner "show all devices" to show all devices that's managed by panorama, and go into the specific firewall device you want to analyze for the logs.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Mon, 23 May 2022 16:42:22 GMT

Hello the first screenshot was panorama --> show all devices --> then i went into the specific device firewall.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

lychiang — Mon, 23 May 2022 16:46:51 GMT

Then it's correct, if you have serial# of the device matching the serial # in the log , it will show the logs in the device for you to process.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Mon, 30 May 2022 12:31:45 GMT

Hi Liam,

No it is not correct, it should display more files 11 to be correct, I have in total 11 files that contain that firewall and serial number.

Output directory: ls -alh
total 45G
drwxr-xr-x 2 www-data expedition 4.0K May 23 15:21 .
drwxrwxr-x 6 www-data expedition 4.0K May 10 13:47 ..
-rw-r--r-- 1 www-data expedition 3.0G May 13 23:59 panorama_traffic_2022_05_13_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.4G May 14 23:59 panorama_traffic_2022_05_14_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.3G May 15 23:59 panorama_traffic_2022_05_15_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.0G May 16 23:59 panorama_traffic_2022_05_16_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.1G May 17 23:59 panorama_traffic_2022_05_17_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.2G May 18 23:59 panorama_traffic_2022_05_18_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.2G May 19 23:59 panorama_traffic_2022_05_19_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.0G May 20 23:59 panorama_traffic_2022_05_20_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.5G May 21 23:59 panorama_traffic_2022_05_21_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.3G May 22 23:59 panorama_traffic_2022_05_22_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.2G May 23 16:55 panorama_traffic_2022_05_23_last_calendar_day.csv

When i grep for hostname through the files and display only unique files names you can see it has more files.

lab:/palogs/10.255.125.50$ grep -H -r "MX-KIO1-FW0001" /palogs/10.255.125.50/ | cut -d: -f1 | sort -u

/palogs/10.255.125.50/panorama_traffic_2022_05_13_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_14_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_15_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_16_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_17_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_18_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_19_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_20_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_21_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_22_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_23_last_calendar_day.csv

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Wed, 01 Jun 2022 14:57:51 GMT

Hi,

Just wanted to give an update. I figured out that the M.learning under device read the first line of the log and if this contains the serial/firewallname will display the file name.

Since i have a single file for all traffic logs from different firewalls every day the first line can be different. So best approach is to have really seperate files per firewall device. I have not found how to do this dynamically do this with rsyslog filters but you can use a script to split the single file into multiple files based on the value off a column. (awk in linux)

So my question is more related to rsyslog how do i send the logs to different files if they come from a single source.

I would like rsyslog to create seperate files based on the $53 column in the traffic log (this is the firewallname).

If i figure out how to do this in rsyslog i will post this config.

Re: Expedition Panorama Log Collector Forwarding - ML-Learning

zGomez — Tue, 07 Jun 2022 07:47:41 GMT

Hi,

I asked asked a friend an he was so kind to provide rsyslog config.

This you can use if you forward all logs from panorama and do not use individual log profile on firewall.

set $!srcfw = field($msg, ",", 52);
$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%$!srcfw%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_calendar_day.csv"
*.* -?DynaTrafficLog