Logs import for ML

Reply
L2 Linker

Logs import for ML

We are following Expedition-LogAnalysisGuide_v1.0.2.pdf to start using ML  

Expedition is v1.1.12, BP v3.6.3, Task Manager is running and there are no internal checks to remediate

The firewalls are VMs running 9.0.1

In page 6 in the above doc, IP address 192.168.44.131 is used for Expedition. However, in page 7, a different IP is used in section Hostname: 10.30.11.50

Please confirm if the IP to configure in Schedule Log Report-Hostname should be the IP for Expedition.

We worked under this assumption, but we get the following error when clicking on Test SCP server connection: Error response from server: bash: /home/expedition/logs/ssh-export-test.txt: No such file or directory

What is causing this error and how can it be fixed?

Until the above is sorted, we would like to upload the logs manually, but the pdf does not include instructions on how to accomplish this:

MANUALLY EXPORT LOGS FROM MONITOR

You can always go to any firewall from Palo Alto Networks and from the Monitor tab export the logs in CSV format and upload that CSV file to Expedition for processing.

What is the process to manually upload the firewall logs onto Expedition?

Thanks.

Ho

L2 Linker

Re: Logs import for ML

Anyone?? Thanks

 

Ho

L4 Transporter

Re: Logs import for ML

Make sure that you are exporting the logs to a folder that exists within Expedition.

 

Personally, I would recommend to use the path /PALogs, which you would have to create giving the following commands:

 

mkdir /PALogs
chown expedition:www-data /PALogs

chmod 750 /PALogs

 

With those, "expedition", the user that you use to send logs via sftp, will be able to write into the folder, and www-data, who reads the logs for generating the connections.parquet (CSV preprocessing) and RE and ML will have read rights.

 

Uploading the files manually only requires you to establish a SCP connection and place them in the path that you desire.

The path should be reacheable by www-data (meaning that the folders towards the complete path should be executable by www-data) and file should be visible/readeable by www-data.

Again, I would suggest placing them into /PALogs. You may desire to create subfolders for each FW, but this step is not necessary as we inspect the CSV content to identify the device's serial number.

L2 Linker

Re: Logs import for ML

Thanks, we've followed the instructions provided and created folder PALogs and transferred the log there. We've also added the firewall as per instructions in page 16 of Expedition.

 

We are unable to find tab Plugins to continue with page 17. Has this tab been removed from v1.1.12?

 

How can be benefit from the ML analysis of these logs done by Expedition? An Excel doc has been created in Traffic tab including:

 

Device Vsys Date Rule Name Typology Hits

 

Thanks.

Ho

 

 

L2 Linker

Re: Logs import for ML

Hi, wondering if anyone could assist with the last question?

 

Thanks.

Ho

L4 Transporter

Re: Logs import for ML

I guess that you are mixing two steps from the documentation.

 

1) Plugins are used within a Project to define the sources from where do you want to apply the learning.

 

2) Another prior step is to prepare the source, which it is to load the CSV logs into the Expedition folder, and preprocess them to make them ready for ML.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!