topic Re: Boosting performance in Expedition Discussions

Boosting performance

FilipElsen — Thu, 26 Sep 2019 14:38:01 GMT

Hi guys,

I've followed your article to install Expedition on a Ubuntu 16.04 LTS server in order to benefit from a full sever capacity (DL-DL380-G9) - it worked great.

> 24 cores, 256GB RAM, 6TB hdd.

The aim is to use it for complex migrations, but also for Rule Enrichment/App-id adoption.

Are there settings to be updated so that Expedition can utilize the full capabilities of the server?

Thinking about fi: /etc/php/7.0/apache2/php.ini

Thanks a lot in advance,

Filip Elsen

Re: Boosting performance

dgildelaig — Thu, 26 Sep 2019 14:58:20 GMT

The Cores and al this extra RAM will be useful for the Rule Enrichment process.

Regarding the php.ini, you won't need to modify it unless you need to change the upload_max values. If you need to import a file larger than your current limit, you can increase that value. However, it would be rather strange that you would need more than 100MB to upload a config.

For most of the processes related to PHP, we only need 1 or 2 cores if the execution is being done in the background. The current limitations defined are in /home/userSpace/userParamenters.php, and define the amount of RAM and the execution time that can be consumed by a PHP execution.

Unless you would get errors due processing too large configurations, you won't need to modify them either.

What you may want to check is your file /home/userSpace/environmentParameters.php, that defines the CPUs and RAM that the Spark (machine learning, rule enrichment) can consume during the log processing activities.

Re: Boosting performance

FilipElsen — Fri, 27 Sep 2019 09:42:15 GMT

Hi Didac,

Thanks for the feedback.

-The files were modified - it' mandatory for our migrations.

- Can the cores, used by PHP be modified/increased? When performing auto-zone assign, the interface freezes for 3hours.

within the file: /home/userSpace/userParamenters.php I've increased the RAM from the default 1G to 10G.

-The file: /home/userSpace/environmentParameters.php contains correct information with regards our system:

define ('NumCPUs',19);
define ('TotalCPUs',24);
define ('SparkRAM','206271m');

The system is being used for:

-Migrations (complex Checkpoint migrations): auto-zone assign makes the system freeze for hours.

-Machine learning/rule enrichment/appid adoption. (+100G logfiles/day)

I would like to make sure, that all modules can benefit for this powerfull platform.

Thanks a lot,

Filip

Re: Boosting performance

dgildelaig — Mon, 30 Sep 2019 08:11:31 GMT

The PHP processes will not get improved by adding extra CPUs. We can have one CPU used for the UI PHP processes and a second one for the Task Manager Agent.

However, adding more CPUs won't make the PHP performance to run faster.

You will experience improvements when using them for ML, RE and other Spark processes (related to traffic log processing).

I feel a bit suspicious about your configuration related to RAM:

206271m

Do you really have 200GB of RAM to offer to your Expedition?

Re: Boosting performance

FilipElsen — Mon, 30 Sep 2019 10:50:52 GMT

Hi Didac,

Ok, then there's no more performance tuning I can perform, as I understand.

Yes, correct - 200G RAM is available.

Best regards,

Filip

Re: Boosting performance

FilipElsen — Mon, 30 Sep 2019 11:05:16 GMT

Looking at ML when processing files - all resources available can be used: