Boosting performance

Reply
L1 Bithead

Boosting performance

Hi guys, 

I've followed your article to install Expedition on a Ubuntu 16.04 LTS server in order to benefit from a full sever capacity (DL-DL380-G9) - it worked great.

> 24 cores, 256GB RAM, 6TB hdd.

The aim is to use it for complex migrations, but also for Rule Enrichment/App-id adoption.

Are there settings to be updated so that Expedition can utilize the full capabilities of the server?

Thinking about fi: /etc/php/7.0/apache2/php.ini

 

Thanks a lot in advance,

Filip Elsen

L4 Transporter

Re: Boosting performance

The Cores and al this extra RAM will be useful for the Rule Enrichment process.

Regarding the php.ini, you won't need to modify it unless you need to change the upload_max values. If you need to import a file larger than your current limit, you can increase that value. However, it would be rather strange that you would need more than 100MB to upload a config.

 

For most of the processes related to PHP, we only need 1 or 2 cores if the execution is being done in the background. The current limitations defined are in /home/userSpace/userParamenters.php, and define the amount of RAM and the execution time that can be consumed by a PHP execution.

Unless you would get errors due processing too large configurations, you won't need to modify them either.

 

What you may want to check is your file /home/userSpace/environmentParameters.php, that defines the CPUs and RAM that the Spark (machine learning, rule enrichment) can consume during the log processing activities.

L1 Bithead

Re: Boosting performance

Hi Didac,

 

Thanks for the feedback.

 

-The files were modified - it' mandatory for our migrations.

- Can the cores, used by PHP be modified/increased? When performing auto-zone assign, the interface freezes for 3hours.

within the file: /home/userSpace/userParamenters.php I've increased the RAM from the default 1G to 10G.

-The file: /home/userSpace/environmentParameters.php contains correct information with regards our system:

define ('NumCPUs',19);
define ('TotalCPUs',24);
define ('SparkRAM','206271m');

 

The system is being used for:

-Migrations (complex Checkpoint migrations): auto-zone assign makes the system freeze for hours.

-Machine learning/rule enrichment/appid adoption. (+100G logfiles/day)

I would like to make sure, that all modules can benefit for this powerfull platform.

 

Thanks a lot,

Filip

 

 

L4 Transporter

Re: Boosting performance

The PHP processes will not get improved by adding extra CPUs. We can have one CPU used for the UI PHP processes and a second one for the Task Manager Agent.

However, adding more CPUs won't make the PHP performance to run faster.

You will experience improvements when using them for ML, RE and other Spark processes (related to traffic log processing).

 

I feel a bit suspicious about your configuration related to RAM:

206271m

Do you really have 200GB of RAM to offer to your Expedition?

L1 Bithead

Re: Boosting performance

Hi Didac,

Ok, then there's no more performance tuning I can perform, as I understand.

Yes, correct - 200G RAM is available.

Best regards,

Filip

 

L1 Bithead

Re: Boosting performance

 

Looking at ML when processing files - all resources available can be used:

ML.PNG

 
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!