Expedition root directory keeps growing

[stevenjwilliams83]

I have expanded the root directory like 2 times already and it keeps filling up. So now the expedition gui will not load. 

So what is taking all the space?  

expedition@pan-expedition:/hdd/PaloLogs$ df -h

Filesystem                       Size  Used Avail Use% Mounted on

udev                             2.9G     0  2.9G   0% /dev

tmpfs                            595M   24M  572M   4% /run

/dev/mapper/Expedition--vg-root  109G  106G     0 100% /

tmpfs                            3.0G     0  3.0G   0% /dev/shm

tmpfs                            5.0M     0  5.0M   0% /run/lock

tmpfs                            3.0G     0  3.0G   0% /sys/fs/cgroup

/dev/sdb                          99G   28G   67G  29% /hdd

/dev/sda1                        472M  109M  340M  25% /boot

tmpfs                            595M     0  595M   0% /run/user/1000

expedition@pan-expedition:/hdd/PaloLogs$

I think the "projects" live in the root so it must be large to be taking up that much room so what are some options here?

/dev/sbd is where the logs from the pan are getting exported.

[dgildelaig]

Could you please check if the folder /tmp/spark-warehouse is taking the space?

If so, how big did it get?

[stevenjwilliams83]

How do I check the size of the directory?

[dgildelaig]

du -hs /tmp/sparl-warehouse

I find it useful to see how much folders take by

for i in `ls`; do du -hs $i; done

[stevenjwilliams83]

expedition@pan-expedition:~$ du -hs /tmp/spark-warehouse

4.0K /tmp/spark-warehouse

expedition@pan-expedition:~$

[dgildelaig]

Try to locate the "big" folder with the script I mentioned above.

Start by / and trace the big folder down to the deep location.

I would appreciate this effort to see whether it is related to any task regarding Spark processes.

[stevenjwilliams83]

I am more than happy to help im just not a skilled linux guy so need a bit more detailed instructions on that script? Is it as easy as copy paste? I assume not. 

[dgildelaig]

You assumed right.

As easy as copy and paste.

It will let you know, from the different folders within your current path, the size of each of them.

So, start by

Step 1: cd /

Step 2: for i in `ls`; do du -hs $i; done

Step 3: Check which folder is the big one. Let's assume it is /tmp
Step 4: cd /tmp

Step 5: jump to step 2

[stevenjwilliams83]

Is there a way to sort? Also there is alot of permissions denied errors. 

[sjanita]

if there were many snapshots taken for each project this could grow the storage for the projects.

You can check the backup directories for each project.

Note: you will need su access to move into the project directory.

/home/userSpace/projects

within each project there will be a 'Backups' directory that can be checked for the snapshots.

[stevenjwilliams83]

expedition@pan-expedition:/home/userSpace$ ls -la

total 100

drwxr-xr-x 5 www-data www-data  4096 Apr 15 07:56 .

drwxr-xr-x 4 root     root      4096 Sep  1  2017 ..

drwxr-xr-x 3 www-data www-data  4096 Apr  5 10:01 devices

-rw-r--r-- 1 www-data www-data    82 Apr 15 07:56 environmentParameters.php

-rw-r--r-- 1 www-data www-data 64913 Apr 16 18:01 panReadOrders.log

drwxrw-rw- 3 www-data www-data  4096 Apr 12 13:38 projects

-rw-r--r-- 1 www-data www-data   598 May 20  2018 userDefinitions.php

drwxr-xr-x 3 root     root      4096 Apr 10 14:50 utils

-rw-r--r-- 1 www-data www-data    36 Aug 15  2018 .uuid4

expedition@pan-expedition:/home/userSpace$ su ?

No passwd entry for user '?'

expedition@pan-expedition:/home/userSpace$ su www-data projects

Password:

su: Authentication failure

expedition@pan-expedition:/home/userSpace$ projects

No command 'projects' found, did you mean:

Command 'projectx' from package 'project-x' (universe)

Command 'projectl' from package 'projectl' (universe)

Command 'project' from package 'turnin-ng' (universe)

projects: command not found

expedition@pan-expedition:/home/userSpace$ cd projects

-bash: cd: projects: Permission denied

expedition@pan-expedition:/home/userSpace$ su cd project

No passwd entry for user 'cd'

expedition@pan-expedition:/home/userSpace$ su cd projects

No passwd entry for user 'cd'

expedition@pan-expedition:/home/userSpace$

[sjanita]

from the cli you will need to issue the command:

expedition@Expedition:/home/userSpace$ sudo su

[sudo] password for expedition:

root@Expedition:/home/userSpace# whoami

root

root@Expedition:/home/userSpace#

Now that you are root you can cd into the projects directory and into each directory to check for backup files

root@Expedition:/home/userSpace# cd projects/

root@Expedition:/home/userSpace/projects# ls -al

total 60

drwxrw-rw- 15 www-data www-data 4096 Apr 18 11:41 .

drwxr-xr-x  5 www-data www-data 4096 Mar 25 15:52 ..

drwxr--r--  3 www-data www-data 4096 Mar 25 15:58 5250MLRE

drwxr--r--  4 www-data www-data 4096 Apr 17 01:02 850MachineLearning

root@Expedition:/home/userSpace/projects# cd 850MachineLearning/

root@Expedition:/home/userSpace/projects/850MachineLearning# ls -al

total 596

drwxr--r--  4 www-data www-data   4096 Apr 17 01:02 .

drwxrw-rw- 15 www-data www-data   4096 Apr 18 11:41 ..

drwxr--r--  2 www-data www-data   4096 Apr 16 11:32 Backups

-rw-r--r--  1 www-data www-data  53200 Apr  1 09:45 MT-850MachineLearning.xml.dat

-rw-r--r--  1 www-data www-data  20752 Apr  1 09:45 MT-850MachineLearning.zip.dat

-rw-r--r--  1 www-data www-data  83680 Mar 27 11:31 PA-850_011901002830.xml_2019.03.27_11h32m.dat

-rw-r--r--  1 www-data www-data  80912 Mar 25 15:58 PA-850_011901002830.xml.dat

-rw-r--r--  1 www-data www-data 129424 Mar 28 10:17 Panorama-QA_007307000481.xml_2019.03.28_10h17m.dat

-rw-r--r--  1 www-data www-data 129360 Mar 28 09:16 Panorama-QA_007307000481.xml.dat

-rw-r--r--  1 www-data www-data  84736 Mar 27 11:36 panos_8.xml.dat

drwxr-xr-x  2 www-data www-data   4096 Mar 28 10:17 toImport

root@Expedition:/home/userSpace/projects/850MachineLearning# cd Backups/

root@Expedition:/home/userSpace/projects/850MachineLearning/Backups# ls -al

total 93792

drwxr--r-- 2 www-data www-data    4096 Apr 16 11:32 .

drwxr--r-- 4 www-data www-data    4096 Apr 17 01:02 ..

-rw-r--r-- 1 www-data www-data 4593712 Mar 27 04:19 base-3-27_MT_2019-03-27_10-19-31.sql.dat

-rw-r--r-- 1 www-data www-data 6011680 Apr 11 02:11 current_MT_2019-04-11_02-11-51.sql.dat

-rw-r--r-- 1 www-data www-data 1252368 Mar 25 16:01 undo_MT_2019-03-25_22-01-27.sql.dat

-rw-r--r-- 1 www-data www-data 4237120 Mar 25 17:16 undo_MT_2019-03-25_23-16-57.sql.dat

-rw-r--r-- 1 www-data www-data 4429968 Mar 27 04:05 undo_MT_2019-03-27_04-05-24.sql.dat

-rw-r--r-- 1 www-data www-data 4361488 Mar 27 04:00 undo_MT_2019-03-27_10-00-00.sql.dat

-rw-r--r-- 1 www-data www-data 4593712 Mar 27 04:19 undo_MT_2019-03-27_10-19-32.sql.dat

-rw-r--r-- 1 www-data www-data 4978240 Apr  1 09:39 undo_MT_2019-04-01_09-39-19.sql.dat

-rw-r--r-- 1 www-data www-data 6014112 Apr 10 13:34 undo_MT_2019-04-10_11-34-19.sql.dat

-rw-r--r-- 1 www-data www-data 6014256 Apr 11 02:04 undo_MT_2019-04-11_00-04-53.sql.dat

-rw-r--r-- 1 www-data www-data 6011856 Apr 11 02:12 undo_MT_2019-04-11_00-12-25.sql.dat

-rw-r--r-- 1 www-data www-data 6013072 Apr 11 02:12 undo_MT_2019-04-11_00-12-51.sql.dat

-rw-r--r-- 1 www-data www-data 6013248 Apr 11 02:13 undo_MT_2019-04-11_00-12-59.sql.dat

-rw-r--r-- 1 www-data www-data 6011680 Apr 11 02:13 undo_MT_2019-04-11_00-13-27.sql.dat

-rw-r--r-- 1 www-data www-data 6011824 Apr 11 02:13 undo_MT_2019-04-11_00-13-33.sql.dat

-rw-r--r-- 1 www-data www-data 6011952 Apr 11 02:20 undo_MT_2019-04-11_00-20-38.sql.dat

-rw-r--r-- 1 www-data www-data 6718736 Apr 15 04:24 undo_MT_2019-04-15_04-24-20.sql.dat

-rw-r--r-- 1 www-data www-data 6719552 Apr 16 11:32 undo_MT_2019-04-16_11-32-55.sql.dat

root@Expedition:/home/userSpace/projects/850MachineLearning/Backups#

[stevenjwilliams83]

so what is removable?

root@pan-expedition:/home/userSpace/projects/CoreMigration/Backups# ls -la

total 107556

drwxr--r-- 2 www-data www-data     4096 Apr 16 18:04 .

drwxr--r-- 4 www-data www-data     4096 Apr 12 16:02 ..

-rw-r--r-- 1 www-data www-data  1312928 Apr 12 13:41 4-12-2019_MT_2019-04-12_13-41-27.sql.dat

-rw-r--r-- 1 www-data www-data 45896112 Apr 16 18:04 4-16-2019_MT_2019-04-16_18-04-24.sql.dat

-rw-r--r-- 1 www-data www-data  1312928 Apr 12 13:41 undo_MT_2019-04-12_13-41-27.sql.dat

-rw-r--r-- 1 www-data www-data  1454112 Apr 12 16:01 undo_MT_2019-04-12_16-01-21.sql.dat

-rw-r--r-- 1 www-data www-data  1454272 Apr 12 16:01 undo_MT_2019-04-12_16-01-31.sql.dat

-rw-r--r-- 1 www-data www-data 12779920 Apr 16 18:01 undo_MT_2019-04-16_18-01-33.sql.dat

-rw-r--r-- 1 www-data www-data 45896112 Apr 16 18:04 undo_MT_2019-04-16_18-04-25.sql.dat

root@pan-expedition:/home/userSpace/projects/CoreMigration/Backups#

[sjanita]

you will want to remove the newer files first. 

But looking at the size of the files, these specific files are MB in size so I don't know how much of an impact they will have to free up disk space, but if those files are not needed they can be deleted.

[sjanita]

it also look as if you expenaded the size of the default partiion and also added a second disk. 

were you priginally storing traffic logs in the /PALogs directory? If yes have you checked that directory for any traffic log files that can be deleted?