Expedition Panorama Log Collector Forwarding - ML-Learning

[zGomez]

Hello,

I have setup expedition as an rsyslog server and configured the Panorama Log collector to forward traffic logs to the expedtion server.

(for now i am filtering on panaorma to filter on only the device logs I am currently interested in)

The logs are all collected inside a single file under the folder

/palogs/panoramip/daily_log.csv.  

I have imported panorama inside Expedition.  My firewalls have global defined policies for all devices inside the device group an specific firewall rules depending on location. 

The global firewall policy has rules that have an any in the source and I would like to create more specific rules for the location.

Since all traffic logs for all devices are in a single file right now I do not know if this is possible.

Some question I have,  where do i have to enable the M-Learning on the panorama device or on the firewall device itself?

Will expedition be able to make difference between logs comming from Location1 and Location2, both have the same rule but source traffic will be different.

Is there a way to create seperate log files per device, I think you will have to do this in the rsyslog file.

So I want a different log analysis per device group for the same rule.

Hoping this explains a little bit what I want to achieve if not don't hesitate to aks for more info, screenshot, logs, ...

[lychiang]

@zGomez To answer your questions:

Where do i have to enable the M-Learning on the panorama device or on the firewall device itself?

You will retrieve all devices from Panorama device tab ,and when you click on right upper corner icon "show all devices" , you will see all firewall devices thats managed by the panorama, and go to the specific firewall device , click on "M. Learning" tab , and process the logs there.

Will expedition be able to make difference between logs coming from Location1 and Location2, both have the same rule but source traffic will be different.

ML will analyze the logs based on serial number of the device , so if you only want to analyze specific device, you can define that in the "log connector" that only select the specific firewall under specific Device Group. 

Is there a way to create separate log files per device, I think you will have to do this in the rsyslog file.

So I want a different log analysis per device group for the same rule.

When you make the Expedition as syslog server, it should automatically create separate folder based on firewall IP to contain logs that came from separate firewall devices. 

For any questions related ML Process, I would suggest you to review the tutorial video here:

https://www.youtube.com/playlist?list=PLD6FJ8WNiIqXAfspousboWn6AllrOWVMi

Module 4 will show you how to import logs from firewall includes making Expedition as syslog server

[zGomez]

Hi Lychiang,

Thank you for your feedback.

I have setup expedition as syslog server but i am fowarding traffic logs from panorama not with a log profile on the firewall's itself.  Since all log are already forwarded to Panorama I am using this approach.  

So I have a single log file with all traffic logs from all devices.  But indeed I can see the log files includes the serial number of the device so maybe not an issue that I have a single file.

I will have to play with the log collector and see the outcome of this.

[zGomez]

I have my syslog setup to receive traffic logs from panorama device that forwarding me all firewall logs of my managed devices.
The objective is to create a seperate log file based on the firewall hostname include in the log files( important not the same as the sending hostname, ip of the logs)

Example log file:

2022-05-16T14:26:01+00:00 PANORAMA 1,2022/05/16 14:26:01,012001050280,TRAFFIC,end,2049,2022/05/16 14:25:57,10.51.10.8,10.52.2.1,0.0.0.0,0.0.0.0,MPLS ,,,snmpv1,vsys1,VPN,trust,tunnel.2001,ethernet1/2.900,LogToPanorama,2022/05/16 14:25:57,76592,1,58824,161,0,0,0x4019,udp,allow,16 90,785,905,18,2022/05/16 14:25:25,1,any,0,578843949,0x8000000000000000,10.0 .0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,9,9,aged-out,29,3090,3092,0,,FIREWALLNAME,from-policy,,,0,,0,,N/A,0,0,0,0,,0,0,,,,,,,

This is currenlty in my syslog file.

$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%HOSTNAME%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_ca lendar_day.csv"
*.* -?DynaTrafficLog

Anybody that can help on configuring syslog for this?

[lychiang]

Are you not seeing the log saved in /palogs/{yourpanoramaIP}/ folder ?

[zGomez]

Yes i receive the logs that is not the issue.

I have my syslog setup to receive traffic logs from panorama device that forwarding me all firewall logs of my managed devices.
The objective is to create a seperate log file based on the firewall hostname include in the log files( important not the same as the sending hostname, ip of the logs)

Example log file:
2022-05-16T14:26:01+00:00 PANORAMA 1,2022/05/16 14:26:01,012001050280,TRAFFIC,end,2049,2022/05/16 14:25:57,10.51.10.8,10.52.2.1,0.0.0.0,0.0.0.0,MPLS ,,,snmpv1,vsys1,VPN,trust,tunnel.2001,ethernet1/2.900,LogToPanorama,2022/05/16 14:25:57,76592,1,58824,161,0,0,0x4019,udp,allow,16 90,785,905,18,2022/05/16 14:25:25,1,any,0,578843949,0x8000000000000000,10.0 .0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,9,9,aged-out,29,3090,3092,0,,FIREWALLNAME,from-policy,,,0,,0,,N/A,0,0,0,0,,0,0,,,,,,,

This is currenlty in my syslog file.

$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%HOSTNAME%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_ca lendar_day.csv"
*.* -?DynaTrafficLog



This stores the files in /palogs/IP-address/panorama_date_.csv

All traffic logs will be in the same file. I want seperate logs based on FIREWALLNAME (column 53 in the log line) this can be different values.
I could do this using if msg contains "FIREWALLNAME" but this is not really dynamic and hard to keep up with.
Can i do this using template?

[lychiang]

@zGomez , What is the purpose to separate logs based on the firewall IP,  if you want to only do ML on specific firewall log, you can define log connector in the project that only select that specific firewall, then expedition will only do ML on the device that's defined in the log connector .  For example , you can only select specific firewall in DG1 like below screenshot 

[zGomez]

Hi Lychiang,

The issue I am facing is that under the device config M.learning i am not seeing all the files that contain that specific firewall.

I only see 1 file the first one created.  

there are several files in the directory with the same rights as other file, containg serial and firewallname.  i should see 11 files in the above screenshot.

The log is currently storing log files for 4 different firewalls.

If i take  another device also in the log file i can see all of the file in /palogs.

When i manual copy the log files by filtering on LInux containing the first firewall logs i can see the file.

so it is before I create a project log-collector.

[lychiang]

The first screenshot , was it in the panorama or in the specific firewall ?  In Device-> Panorama, you will click on the retrieved connected device first , so it will retrieve all the firewall that's managed by the panorama, 2nd, you click on the right upper corner "show all devices" to show all devices that's managed by panorama, and go into the specific firewall device you want to analyze for the logs. 

[zGomez]

Hello the first screenshot was panorama --> show all devices --> then i went into the specific device firewall.

[lychiang]

Then it's correct, if you have serial# of the device matching the serial # in the log , it will show the logs in the device for you to process. 

[zGomez]

Hi Liam,

No it is not correct, it should display more files 11 to be correct,  I have in total 11 files that contain that firewall and serial number.

Output directory:   ls -alh
total 45G
drwxr-xr-x 2 www-data expedition 4.0K May 23 15:21 .
drwxrwxr-x 6 www-data expedition 4.0K May 10 13:47 ..
-rw-r--r-- 1 www-data expedition 3.0G May 13 23:59 panorama_traffic_2022_05_13_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.4G May 14 23:59 panorama_traffic_2022_05_14_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.3G May 15 23:59 panorama_traffic_2022_05_15_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.0G May 16 23:59 panorama_traffic_2022_05_16_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.1G May 17 23:59 panorama_traffic_2022_05_17_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.2G May 18 23:59 panorama_traffic_2022_05_18_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.2G May 19 23:59 panorama_traffic_2022_05_19_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 5.0G May 20 23:59 panorama_traffic_2022_05_20_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.5G May 21 23:59 panorama_traffic_2022_05_21_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.3G May 22 23:59 panorama_traffic_2022_05_22_last_calendar_day.csv
-rw-r--r-- 1 www-data expedition 3.2G May 23 16:55 panorama_traffic_2022_05_23_last_calendar_day.csv

When i grep for hostname through the files and display only unique files names you can see it has more files.

lab:/palogs/10.255.125.50$ grep -H -r "MX-KIO1-FW0001" /palogs/10.255.125.50/ | cut -d: -f1 | sort -u

/palogs/10.255.125.50/panorama_traffic_2022_05_13_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_14_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_15_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_16_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_17_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_18_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_19_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_20_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_21_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_22_last_calendar_day.csv
/palogs/10.255.125.50/panorama_traffic_2022_05_23_last_calendar_day.csv

[zGomez]

Hi,

Just wanted to give an update. I figured out that the M.learning under device read the first line of the log and if this contains the serial/firewallname will display the file name.

Since i have a single file for all traffic logs from different firewalls every day the first line can be different.  So best approach is to have really seperate files per firewall device.  I have not found how to do this dynamically do this with rsyslog filters but you can use a script to split the single file into multiple files based on the value off a column. (awk in linux)

So my question is more related to rsyslog how do i send the logs to different files if they come from a single source.

I would like rsyslog to create seperate files based on the $53 column in the traffic log (this is the firewallname).

If i figure out how to do this in rsyslog i will post this config.

[zGomez · Accepted solution]

Hi,

I asked asked a friend an he was so kind to provide rsyslog config.

This you can use if you forward all logs from panorama and do not use individual log profile on firewall.

set $!srcfw = field($msg, ",", 52);
$template DynaTrafficLog,"/palogs/%FROMHOST-IP%/%$!srcfw%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_calendar_day.csv"
*.* -?DynaTrafficLog