Logs import for ML

[ash83]

We are following Expedition-LogAnalysisGuide_v1.0.2.pdf to start using ML  

Expedition is v1.1.12, BP v3.6.3, Task Manager is running and there are no internal checks to remediate

The firewalls are VMs running 9.0.1

In page 6 in the above doc, IP address 192.168.44.131 is used for Expedition. However, in page 7, a different IP is used in section Hostname: 10.30.11.50

Please confirm if the IP to configure in Schedule Log Report-Hostname should be the IP for Expedition.

We worked under this assumption, but we get the following error when clicking on Test SCP server connection: Error response from server: bash: /home/expedition/logs/ssh-export-test.txt: No such file or directory

What is causing this error and how can it be fixed?

Until the above is sorted, we would like to upload the logs manually, but the pdf does not include instructions on how to accomplish this:

MANUALLY EXPORT LOGS FROM MONITOR

You can always go to any firewall from Palo Alto Networks and from the Monitor tab export the logs in CSV format and upload that CSV file to Expedition for processing.

What is the process to manually upload the firewall logs onto Expedition?

Thanks.

Ho

[ash83]

Anyone?? Thanks

Ho

[dgildelaig]

Make sure that you are exporting the logs to a folder that exists within Expedition.

Personally, I would recommend to use the path /PALogs, which you would have to create giving the following commands:

mkdir /PALogs
chown expedition:www-data /PALogs

chmod 750 /PALogs

With those, "expedition", the user that you use to send logs via sftp, will be able to write into the folder, and www-data, who reads the logs for generating the connections.parquet (CSV preprocessing) and RE and ML will have read rights.

Uploading the files manually only requires you to establish a SCP connection and place them in the path that you desire.

The path should be reacheable by www-data (meaning that the folders towards the complete path should be executable by www-data) and file should be visible/readeable by www-data.

Again, I would suggest placing them into /PALogs. You may desire to create subfolders for each FW, but this step is not necessary as we inspect the CSV content to identify the device's serial number.

[ash83]

Thanks, we've followed the instructions provided and created folder PALogs and transferred the log there. We've also added the firewall as per instructions in page 16 of Expedition.

We are unable to find tab Plugins to continue with page 17. Has this tab been removed from v1.1.12?

How can be benefit from the ML analysis of these logs done by Expedition? An Excel doc has been created in Traffic tab including:

Device

Vsys

Date

Rule Name

Typology

Hits

Thanks.

Ho

[ash83]

Hi, wondering if anyone could assist with the last question?

Thanks.

Ho

[dgildelaig]

I guess that you are mixing two steps from the documentation.

1) Plugins are used within a Project to define the sources from where do you want to apply the learning.

2) Another prior step is to prepare the source, which it is to load the CSV logs into the Expedition folder, and preprocess them to make them ready for ML.

[LauraZ]

Hi,

I need help on this item as well. I have tried everything and the logs doesn't seem to be exported from Palo Alto to Expedition. The test file gets to the path but no exports happen at the scheduled time.
How can I just import manually logs into Expedition?

[dgildelaig]

If you have tested that the path is correct (I assume we are talking about the "Test SCP server connection" option that the NGFW have), this means that you have the permissions correct in your Expedition to receive the logs.

If you do not receive logs, my idea would be that maybe you still need to Commit in the Firewall the change regarding performing the Schedule log export.

Another option is to download the logs via the "Export to CSV" in the Monitor tab and upload them to the desired folder in Expedition via SCP.

This is not a desired way, as it requires manual upload each time you want to check logs, but if this a puntual task that you want to do, it may be fine.

[KartikaUtami123]

Hi Dgildelaig,

I've managed to import my logs to expedition to /PAlogs folder manually and would like to use expedition to help create firewall rulese from these log. 

I don't have a connection to a live firewall.

Can this be done with expedition? Can you please advise how?

Thanks

[dpuigdomenec]

Hi @KartikaUtami123 

Thanks for reaching out.

You can execute ML and RE using Expedition1 even if Expedition1 does not have access to the FW.

You can create the device and upload manually the xml configuration instead of requesting Expedition1 to download it from the device itself. Make sure when creating the device that you specify the proper serial number, as this field will be the used to map the logs with the device.

Let me share here some information to execute ML and RE using Expedition1:

• Youtube set of videos: https://www.youtube.com/playlist?list=PLD6FJ8WNiIqXAfspousboWn6AllrOWVMi
• "Log Analysis Feature Guide" here: https://live.paloaltonetworks.com/t5/expedition-articles/expedition-documentation/ta-p/215619

Hope this helps,

Best regards