- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
Enhanced Security Measures in Place: To ensure a safer experience, we’ve implemented additional, temporary security measures for all users.
01-11-2021 06:14 AM
Expedition 1.1.89
running machine learning on the traffic logs.
I've given www-data full permissions and chmod 777 to the file /data and all .csv files
When I try to "Process Files" under Palo Alto Network Devices>M.learning all i see in the comments is "Today's file in progress"
01-11-2021 06:28 AM - edited 01-11-2021 06:36 AM
According on how you have defined the file (Log files come from Syslog), today's file will not be processed.
This is because today's file is not over yet, new traffic logs are still coming and will remain coming until 23:59:59.999.
So, Expedition will prevent you from processing today's file so you do not miss, in your processing, data that would be later valuable for doing Rule Enrichment or Machine Learning.
As I see, you have defined in Expedition that you want this Firewall's logs to be automatically processed at 5:57:30, so, tomorrow at this time, this log (that by tomorrow it will be yesterday's log) will be processed.
One thing to mention regarding the file permissions. If you can see the file listed in the grid, it means that you have, at least, correct reading permissions to the file. This would be enough, unless you also want to be able to delete/compress after processing. In such case, you also want writing rights on the file.
Giving 777 permissions on files may be excessive. Normally your file belongs to the user "expedition" and the group "expedition". www-data if I am not mistaken, is part of the "expedition" user group. Therefore, with a 660 should be enough.
Remember that this translates to
rw-rw----
meaning:
rw- "the user expedition can read and write, but not execute (however, we do not execute CSV files)"
rw- "the group expedition can read and write the file"
--- the rest of users do not have access to the file
Notice also that we need to be able to reach the file itself. In your case this is within the /data folder. Therefore, the expedition and www-data users should be able to enter (execute) that folder.
So, for the folder we may want to provide a 770 to www-data:expedition. If we would have the files in further nested folders, we should provide access to those folders as well.
01-11-2021 06:28 AM - edited 01-11-2021 06:36 AM
According on how you have defined the file (Log files come from Syslog), today's file will not be processed.
This is because today's file is not over yet, new traffic logs are still coming and will remain coming until 23:59:59.999.
So, Expedition will prevent you from processing today's file so you do not miss, in your processing, data that would be later valuable for doing Rule Enrichment or Machine Learning.
As I see, you have defined in Expedition that you want this Firewall's logs to be automatically processed at 5:57:30, so, tomorrow at this time, this log (that by tomorrow it will be yesterday's log) will be processed.
One thing to mention regarding the file permissions. If you can see the file listed in the grid, it means that you have, at least, correct reading permissions to the file. This would be enough, unless you also want to be able to delete/compress after processing. In such case, you also want writing rights on the file.
Giving 777 permissions on files may be excessive. Normally your file belongs to the user "expedition" and the group "expedition". www-data if I am not mistaken, is part of the "expedition" user group. Therefore, with a 660 should be enough.
Remember that this translates to
rw-rw----
meaning:
rw- "the user expedition can read and write, but not execute (however, we do not execute CSV files)"
rw- "the group expedition can read and write the file"
--- the rest of users do not have access to the file
Notice also that we need to be able to reach the file itself. In your case this is within the /data folder. Therefore, the expedition and www-data users should be able to enter (execute) that folder.
So, for the folder we may want to provide a 770 to www-data:expedition. If we would have the files in further nested folders, we should provide access to those folders as well.
01-11-2021 09:32 PM
thank you for the explanation. The scheduled traffic logs did come in later and was processed.
Thank you for the alternative to chmod 777
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!