Expedition as syslog server, change logs directory

[LukeBullimore]

Hi Team!

We wish to use the Expedition tool for some logs coming from a PA-7000 series. Since the scheduled log export option is not feasible we wish to export logs via syslog. A few questions with this.

1. Where is the default location that the syslogs will go to?

2. Can we change the default log location, if so how? (We will be adding new virtual disk mounted to /PALogs and want to make the logs go there)

3. What will the permissions need to be on the new logs folder, does it need to be owned by "syslog" or "expedition", both? How?

I have a funny feeling its to do with the rsyslog.confg file but could do with some assistance.

Cheers,

Luke.

[dgildelaig · Accepted solution]

Hi Luke,

Yes, Expedition can get the logs as a syslog server as well.

In the /etc/rsyslog.conf you would define how the syslog server (RSyslog in Expedition case) would process the received entries:

• where to listen from (notice that the example defined the 10514 port, which it is not the default one. Probably you would prefer setting it up to 514)
• which IPs to allow reporting
• where to store the log entries

As we would like to help you speeding up the process of setting up the syslog server, we have provided three useful examples that you can simply take for your Expedition instance.

You can find them inside the folder 

/var/www/html/OS/rsyslog/

Inside, you will find a config to listen only for UDP connections, for TCP connections or both.

Take the one you prefer, edit it to add the IPs inside the AllowedSender parameteryou would allow to send syslog entries (tipically the IPs for the reporting interfaces in the PA7000) (by default shows:)

$AllowedSender TCP, 127.0.0.1, 10.11.29.0/24, 172.16.26.0/24, *.paloaltonetworks.com

and save it into /etc with the name rsyslog.conf

Remember to restart the syslog service by

service rsyslog restart

We have designed the config to save the logs into a /data folder that should exist (create it if it doesn't) and are stored by reporting IP and day (we try to emulate the structure that a firewall would create with the log export).

Regarding the permits, thing that the "rsyslog" group will store the logs there, but that www-data needs to be able to read them as well to generate the parquets and perform the ML activities.

$template DynaTrafficLog,"/data/%FROMHOST-IP%/%HOSTNAME%_traffic_%$YEAR%_%$MONTH%_%$DAY%_last_calendar_day.csv"

Finally, make sure that your Expedition instance is allowed to listen the desired port. Expedition has firewalld service that you may need to edit it. For instance:

sudo /usr/bin/firewall-cmd --permanent --add-port=514/udp

Let's start with this.

UPDATE: We have noticed that in some situations, it may be required to restart the machine to ensure that rsyslog captures the arriving traffic and process it.

[LukeBullimore]

Many thanks, @dgildelaig!

I will let you know how I get on with this.

[LukeBullimore]

Hi @dgildelaig

We attempted this today with unfortunately no luck.

We modified the rsyslog-default-udp config file and then copied it to /etc/  as rsyslog.conf (we changed the rsyslog.conf that already existed there as rsyslog.conf.bak) 

Unfortunately there are no logs coming into the folder we specified. FW rule was created and syslog has permissions to the folder, restarted syslog service.

We then copy pasted the snippet from the default-udp file into the already existing rsyslog.conf file in the /etc/ folder, restarted syslog service. We now have the "last calandar day.csv" file created in our log folder but it only contains rsyslog log files rather than FW log files. Please help! 🙂

[Laimonas]

Do i need to do anything else for Expedition to automatically parse those files?

Config works.

root@Expedition:/data/10.127.2.XX# ls -al

total 28496

drwx------ 2 root   root     4096 Apr  5 00:37 .

drwxr-xr-x 7 syslog root     4096 Apr  5 00:37 ..

-rw-r--r-- 1 root   root 29170806 Apr  5 00:58 xxx.xxxxxxx.xxx_traffic_2019_04_05_last_calendar_day.csv

[dgildelaig]

For anyone reading this thread in the future, we have found that in some cases, to see logs getting into the desired folder, it is ALSO necessary to restart the VM. Restarting the service did not seem to be enough, and as I am not a guru to identify which other services are involved in this process, what we found out is that restarting the VM helped to start seeing the logs arriving into the desired folder.

We spent hours with a client to see whether the config was wrong (which look perfect), sniffing traffic to see that was hitting the Expedition VM from the Firewalls that were forwarding the syslog entries, and at the end, we just reboot the VM and voilá!

I hope this helps others.

[hamadahFarajallah]

Hi,

I have face same issue, how can automatically parse these file i can not see it in the expedition GUI, when you go to devices --> m.Lerning noting is there?

any thing that i need to be done?

thanks in advance

Hamadah 

[hamadahFarajallah]

Hi,

I have an issue i can see the csv file in the folder but how can automatically parse these file,  I can not see it in the expedition GUI, when you go to devices --> m.Lerning noting is there?

any thing that i need to be done?

thanks in advance

Hamadah 

[lychiang]

@hamadahFarajallah Please make sure your expedition is up to date on v 1.1.105 since there was a fixed for the issue you running to in the earlier version.  

[azuniga]

Hello,

We had an issue on 1.1.104 with machine learning so we pushed a fix out on 1.1.105 so if you are currently using that version I would suggest you upgrade to the latest. If this isn't the case just make sure that the folder permissions are there for the PALogs folder and the sub folder within PALogs.

[hamadahFarajallah]

Thank you for your reply, but my issue is the file format :

not working format that when i exported as syslog:

2021-08-24T11:51:13+03:00 MOE-HQ-PA-FW-01.moe.local 1,2021/08/24 11:51:12,010108010441,TRAFFIC,start,2305,2021/08/24 11:51:12,10.0.70.54,192.168.6.215,0.0.0.0,0.0.0.0,ANY-to-ANY_HTTPs,,,ms-sms,vsys1,outside,DB-V-13,ae2.63,ae2.2,Panorama-log,2021/08/24 11:51:12,71962469,1,52271,80,0,0,0x4000,tcp,allow,4472,4402,70,7,2021/08/24 11:51:11,0,private-ip-addresses,0,6982130509256638010,0x8000000000000000,10.0.0.0-10.255.255.255,192.168.0.0-192.168.255.255,0,6,1,n/a,922,0,0,0,HQ-DC,MOE-HQ-PA-FW-01,from-policy,,,0,,0,,N/A,0,0,0,0,47a0c081-9c8c-40c7-91b8-d047d0d54e6b,0,0,,,,,,,

working format, and this format was export normaly from the firewall as (scheduled log export)
1,2021/08/22 00:00:00,013201026585,TRAFFIC,start,2305,2021/08/22 00:00:00,10.25.127.56,10.0.10.204,,,EDs-To-Call-Manager,,,web-browsing,vsys2,RYNC-IPVPN-OUTSIDE,RYNC-IPVPN-INSIDE,ae6.601,ae5.600,Log_Forwarding_WAN,2021/08/22 00:00:00,514109,1,46033,6970,0,0,0x100000,tcp,allow,373,295,78,4,2021/08/22 00:00:01,0,any,0,6971380400278891820,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,3,1,n/a,0,0,0,0,NC-WAN-VSYS2,RYNC-PA-FW-01,from-policy,,,0,,0,,N/A,0,0,0,0,49a0caee-7b03-4ee0-aa85-cc902ed09c8a,0,0,,,,,,,

How I can change the non working format?

any help?

Hamadah

[lychiang]

I think you have the correct format , have you already updated your expedition to v1.1.105?  If you already did , have you seen the log file in the GUI? If you still not seeing the file in GUI, please make sure you can ssh to the expedition and see the file is already in the folder you trying to search when you trying to process the log.