Boosting performance

[FilipElsen]

Hi guys, 

I've followed your article to install Expedition on a Ubuntu 16.04 LTS server in order to benefit from a full sever capacity (DL-DL380-G9) - it worked great.

> 24 cores, 256GB RAM, 6TB hdd.

The aim is to use it for complex migrations, but also for Rule Enrichment/App-id adoption.

Are there settings to be updated so that Expedition can utilize the full capabilities of the server?

Thinking about fi: /etc/php/7.0/apache2/php.ini

Thanks a lot in advance,

Filip Elsen

[dgildelaig]

The Cores and al this extra RAM will be useful for the Rule Enrichment process.

Regarding the php.ini, you won't need to modify it unless you need to change the upload_max values. If you need to import a file larger than your current limit, you can increase that value. However, it would be rather strange that you would need more than 100MB to upload a config.

For most of the processes related to PHP, we only need 1 or 2 cores if the execution is being done in the background. The current limitations defined are in /home/userSpace/userParamenters.php, and define the amount of RAM and the execution time that can be consumed by a PHP execution.

Unless you would get errors due processing too large configurations, you won't need to modify them either.

What you may want to check is your file /home/userSpace/environmentParameters.php, that defines the CPUs and RAM that the Spark (machine learning, rule enrichment) can consume during the log processing activities.

[FilipElsen]

Hi Didac,

Thanks for the feedback.

-The files were modified - it' mandatory for our migrations.

- Can the cores, used by PHP be modified/increased? When performing auto-zone assign, the interface freezes for 3hours.

within the file: /home/userSpace/userParamenters.php I've increased the RAM from the default 1G to 10G.

-The file: /home/userSpace/environmentParameters.php contains correct information with regards our system:

define ('NumCPUs',19);
define ('TotalCPUs',24);
define ('SparkRAM','206271m');

The system is being used for:

-Migrations (complex Checkpoint migrations): auto-zone assign makes the system freeze for hours.

-Machine learning/rule enrichment/appid adoption. (+100G logfiles/day)

I would like to make sure, that all modules can benefit for this powerfull platform.

Thanks a lot,

Filip

[dgildelaig]

The PHP processes will not get improved by adding extra CPUs. We can have one CPU used for the UI PHP processes and a second one for the Task Manager Agent.

However, adding more CPUs won't make the PHP performance to run faster.

You will experience improvements when using them for ML, RE and other Spark processes (related to traffic log processing).

I feel a bit suspicious about your configuration related to RAM:

206271m

Do you really have 200GB of RAM to offer to your Expedition?

[FilipElsen]

Hi Didac,

Ok, then there's no more performance tuning I can perform, as I understand.

Yes, correct - 200G RAM is available.

Best regards,

Filip

[FilipElsen]

Looking at ML when processing files - all resources available can be used: