Logs import for ML

Reply
Highlighted
L2 Linker

Logs import for ML

We are following Expedition-LogAnalysisGuide_v1.0.2.pdf to start using ML  

Expedition is v1.1.12, BP v3.6.3, Task Manager is running and there are no internal checks to remediate

The firewalls are VMs running 9.0.1

In page 6 in the above doc, IP address 192.168.44.131 is used for Expedition. However, in page 7, a different IP is used in section Hostname: 10.30.11.50

Please confirm if the IP to configure in Schedule Log Report-Hostname should be the IP for Expedition.

We worked under this assumption, but we get the following error when clicking on Test SCP server connection: Error response from server: bash: /home/expedition/logs/ssh-export-test.txt: No such file or directory

What is causing this error and how can it be fixed?

Until the above is sorted, we would like to upload the logs manually, but the pdf does not include instructions on how to accomplish this:

MANUALLY EXPORT LOGS FROM MONITOR

You can always go to any firewall from Palo Alto Networks and from the Monitor tab export the logs in CSV format and upload that CSV file to Expedition for processing.

What is the process to manually upload the firewall logs onto Expedition?

Thanks.

Ho

Highlighted
L2 Linker

Anyone?? Thanks

 

Ho

Highlighted
L5 Sessionator

Make sure that you are exporting the logs to a folder that exists within Expedition.

 

Personally, I would recommend to use the path /PALogs, which you would have to create giving the following commands:

 

mkdir /PALogs
chown expedition:www-data /PALogs

chmod 750 /PALogs

 

With those, "expedition", the user that you use to send logs via sftp, will be able to write into the folder, and www-data, who reads the logs for generating the connections.parquet (CSV preprocessing) and RE and ML will have read rights.

 

Uploading the files manually only requires you to establish a SCP connection and place them in the path that you desire.

The path should be reacheable by www-data (meaning that the folders towards the complete path should be executable by www-data) and file should be visible/readeable by www-data.

Again, I would suggest placing them into /PALogs. You may desire to create subfolders for each FW, but this step is not necessary as we inspect the CSV content to identify the device's serial number.

Highlighted
L2 Linker

Thanks, we've followed the instructions provided and created folder PALogs and transferred the log there. We've also added the firewall as per instructions in page 16 of Expedition.

 

We are unable to find tab Plugins to continue with page 17. Has this tab been removed from v1.1.12?

 

How can be benefit from the ML analysis of these logs done by Expedition? An Excel doc has been created in Traffic tab including:

 

Device Vsys Date Rule Name Typology Hits

 

Thanks.

Ho

 

 

Highlighted
L2 Linker

Hi, wondering if anyone could assist with the last question?

 

Thanks.

Ho

Highlighted
L5 Sessionator

I guess that you are mixing two steps from the documentation.

 

1) Plugins are used within a Project to define the sources from where do you want to apply the learning.

 

2) Another prior step is to prepare the source, which it is to load the CSV logs into the Expedition folder, and preprocess them to make them ready for ML.

Highlighted
L0 Member

Hi,

I need help on this item as well. I have tried everything and the logs doesn't seem to be exported from Palo Alto to Expedition. The test file gets to the path but no exports happen at the scheduled time.
How can I just import manually logs into Expedition?

Highlighted
L5 Sessionator

If you have tested that the path is correct (I assume we are talking about the "Test SCP server connection" option that the NGFW have), this means that you have the permissions correct in your Expedition to receive the logs.

If you do not receive logs, my idea would be that maybe you still need to Commit in the Firewall the change regarding performing the Schedule log export.

 

Another option is to download the logs via the "Export to CSV" in the Monitor tab and upload them to the desired folder in Expedition via SCP.

This is not a desired way, as it requires manual upload each time you want to check logs, but if this a puntual task that you want to do, it may be fine.

 

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!