topic Re: Spark Temporary Data Structure - Path not provided in Expedition Discussions

Spark Temporary Data Structure - Path not provided

Sam.Howells — Tue, 07 Sep 2021 10:29:56 GMT

I have configured a new instance of Expedition as our previous VM was not able to contact the internet to obtain updates (and there is no offline updater that works for ML).

After upgrading to the latest release (1.1.108), I am trying to complete the Spark Temp Data Structure path in the settings section, however the information is never saved. I have confirmed the permissions on the folder I'm looking to use. I have also followed the following advice:

>sudo apt-get update

>sudo apt-get --reinstall install expedition-beta

I have also rebooted several times. Why am I unable to change the path (I am intending to use /PADataStruct as the folder??

Re: Spark Temporary Data Structure - Path not provided

Bernd — Mon, 13 Sep 2021 11:05:53 GMT

we have the same issue. We have created a directory "/datastore" with rights "drwxrwxrwx", owner "www-data" and group "expedition" (exactly like /PALogs). When we enter "/datastore" as "Temp. Data Structs:" the system responds with "Data saved", but it isn't saved. When we visit the page again the filed is empty and in the dashboard we still have the message " The ML Temporary Data Structures Path is incorrectly defined."

What are we doing wrong? Any ideas?

Re: Spark Temporary Data Structure - Path not provided

lychiang — Mon, 13 Sep 2021 15:46:19 GMT

After you changed the temp data structures to the new path , please click on the red button "Delete Connections.parquet files" and then click on "Save". It should save the new path. I verified it's working on v1.1.109.

Re: Spark Temporary Data Structure - Path not provided

Sam.Howells — Tue, 14 Sep 2021 07:35:11 GMT

This doesn't work for me. When I click "Delete Connection.parquet files" it shows a message "/PALogs/connections.parquet does not exist. Nothing to delete". I then click "Save", but the issue is still the same - when I come back to the page the data struct field is blank and the check fails on the home page. Can you help further?

Re: Spark Temporary Data Structure - Path not provided

dgildelaig — Tue, 14 Sep 2021 08:43:24 GMT

Deleting the Connections.parquet should not be necessary, and it should be done only if we want to remove all the already processed data.

Looking at the figure we have in Expedition M.Learning settings, our CSV traffic logs would be stored in a path (individually defined for each firewall) and converter into Connections.parquet to allow performing the data analytics processes in ML and RE.

During the preprocessing, a folder can be defined to store temporary structures.

If you have admin rights within Expedition, the tool should allow you to modify these settings. If you cannot, please contact me to fwmigrate at paloaltonetworks.com and we can do a session to see what may be preventing the storage of the changes.

Re: Spark Temporary Data Structure - Path not provided

DanaHawkins — Thu, 07 Oct 2021 19:35:34 GMT

I'm still having this issue. I have created a folder and applied the same ownership properties as /PALogs. When I enter the new folder name and click the "save" button it appears to be fine. But if I navigate away from the setting page and come back there is no setting. Is there a configuration file where I can set the manually under the hood?

Was there ever a resolution?

Re: Spark Temporary Data Structure - Path not provided

dgildelaig — Fri, 08 Oct 2021 08:45:31 GMT

Please, check if the latest 1.1.113 version corrects this issue for you.

Best,