Demisto Service Failed

[Pavendhan_K]

Hi Everyone ,

We are running XSOAR instance with NFR license. 

Recently for unknown reasons, the demisto service stopped and is never coming up when we try to start. 

[root@localhost ~]# systemctl status demisto.service
● demisto.service - Demisto Server Service
Loaded: loaded (/etc/systemd/system/demisto.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2023-05-17 19:43:28 AEST; 45s ago
Process: 2122 ExecStart=/usr/local/demisto/server (code=exited, status=203/EXEC)
Main PID: 2122 (code=exited, status=203/EXEC).

Can you help in fixing what is wrong here. Thank you. 

Cortex XSOAR  

[cpayne26]

There's not much that exit code  screenshot provides on its own and you should probably open a ticket with support.   In the interim, Assuming a standard installation you can probably see some more hints in /var/log/demisto.service.log  or in the journal logs (journalctl --since "<##>  hour ago" > <jfilename>   (example collecting 12 hours of journalctl logs:  journalctl --since "12 hour ago" > journalctl_12hr.txt)  ) .  Off the top of my head, make sure there's enough space in the filesystems the relevant directories (see link for relevant directories) and demisto has access.  Make sure you're system meets the minimum requirements.   also confirm the docker/podman  service is up and healthy

  https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.11/Cortex-XSOAR-Administrator-Guide/Instal...  (contains relevant filesystem structure and sizing recommendations) 
https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.11/Cortex-XSOAR-Administrator-Guide/System... (system requirements)

[MBeauchamp2]

Check the /var/log/demisto/server.log to see if the reason presents itself.

You can also check the journalctl logs on restart, problem may be there as well. 

[atullo]

Pavendhan_K Have you been using the NFR license for more than 30 days?

Start Your 30-Day Free Trial of Cortex XSOAR ... 

[dinopc]

Saw the below error for same issue in our Cortex XOAR installation after looking at server.log . How can we cleaup the disk space? which all files should be cleaned up?

2023-12-05 07:09:02.0424 error Failed getting docker working dir with err: [mkdir /var/lib/demisto/temp/script421178974: no space left on device] (source: /builds/GOPATH/src/gitlab.xdr.pan.local/xdr/xsoar/server/services/automation/container.go:195)

[cpayne26]

Dinopc 

A little more information is needed like demisto application version, what type of demisto architecture you're using, etc.   

This response assumes you're using  demisto version 6.12, single server deployment (bolt instead of elastic for the db), the default installation process was followed (so all demisto components were installed in /var/lib/demisto) and you're not using an advanced config like live backup or distributed db.  (https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.12/Cortex-XSOAR-Administrator-Guide/Deploy...) 

To archive data for demisto  You would use 
https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.9/Cortex-XSOAR-Administrator-Guide/Free-up...
To archive the db data and https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.12/Cortex-XSOAR-Administrator-Guide/Archiv... to archive artifacts and attachment.    Sometimes people forget to clean up artifacts and attachments when they clean up demisto.  
 
However before I archive I like to confirm that archiving will help with the filesystem being full (its possible something else could be filling up the filesystem for example a badly configured syslog where the unix server isn't using filesystems to help manage space...ie all space is allocated to /)  to do this you can log into your unix server and run df -h to look at the filesystem layout and du -sh to look at how much space under /var/lib/demisto/data is being used by each directory.   

Another helpful hint is you can look in /var/lib/demisto/data/partitionsData to see how much data your demisto app has to archive.  There's one that represents each calendar month of data that you have on the server (see example below)

/var/lib/demisto/data/partitionsData:
total 325G
drwxr-xr-x 2 demisto demisto 4.0K Nov 1 00:00 .
drwxr-xr-x 4 demisto demisto 4.0K Feb 4 2023 ..
-rw------- 1 demisto demisto 10G Nov 27 23:41 demisto_052022.db  (May 2022)
-rw------- 1 demisto demisto 15G Nov 27 23:41 demisto_062022.db  (June 2022)
etc...

One more thing to check depending on how your filesystem are configured, if you're doing database backups through the app https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.12/Cortex-XSOAR-Administrator-Guide/Back-u...  assuming default configs the data is stored in /var/lib/demisto/backup.   can configure your system to use a different directory on a different filesystem and move the backup files to that fileystem as well.  This can also help with space issues.

Last bit of advice is to make sure you have appropriate backups/system snapshots because the database backup doesn't backup artifacts and attachments (see detail in step 3 https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.12/Cortex-XSOAR-Administrator-Guide/Back-u...) and you'll want to be able to have a good recovery strategy if something goes wrong.

I hope this helps.