article Scheduled Log Processing in Version 1.1.21 in Expedition Articles

Scheduled Log Processing in Version 1.1.21

sjanita — Mon, 07 Oct 2019 15:43:36 GMT

This document describes how to enable and configure the new feature in update version 1.1.20 to enable the scheduling of log processing for the Machine Learning feature.

Re: New feature ver 1.1.20 - Scheduled Log Processing

Laimonas — Tue, 28 May 2019 07:53:49 GMT

It does not work. I don't see CPU spikes i usually see when log processing in progress.

Doesn't it have to have cron record or something ?

The schedule was sset to 10:05:00

Re: New feature ver 1.1.21 - Scheduled Log Processing

sjanita — Tue, 28 May 2019 09:33:06 GMT

Can you update to ver 1.1.21 and test again.

Also, how many log files are being processed? The best practice is to schedule the log processing during off hours.

Re: New feature ver 1.1.21 - Scheduled Log Processing

Laimonas — Tue, 28 May 2019 11:06:04 GMT

there were 5 files

we've successfully updated to 1.1.21

now we'll have to wait till tomorrow to have new file.

we'll let You know

Re: New feature ver 1.1.21 - Scheduled Log Processing

IvanMoore — Tue, 28 May 2019 21:19:23 GMT

upgraded to 1.1.21 and now I get a lot of odd issues and I get booted constantly. When I first log in, the Dashboard reloads many many times before finally setteling down. Then I go into one of my projects and click on say unused services. It takes a bit while it does its query then i get a white page and eventually logged out.

Re: New feature ver 1.1.21 - Scheduled Log Processing

Laimonas — Wed, 29 May 2019 07:19:55 GMT

after night, new file generated, but schedule did not work.

we see log message, but 1 file is still not processed.

Re: New feature ver 1.1.21 - Scheduled Log Processing

Poulsen — Thu, 30 May 2019 00:27:50 GMT

I am gettting the same behaviour as IvanMoore. I get booted constantly as well and when I log in, the Dashboard reloads many many times before finally settling down. This happens across IE/Firefox/Chrome/Safari as well as Windows/Mac.

Re: New feature ver 1.1.21 - Scheduled Log Processing

sjanita — Thu, 30 May 2019 23:07:29 GMT

i'm reviewing both the scheduled log export and teh UI refresh.

Re: New feature ver 1.1.21 - Scheduled Log Processing

sjanita — Thu, 30 May 2019 23:13:58 GMT

for the UI refresh issues can you check the remaining space in your root partition:

From the CLI of your Expedition server issue the below command.

$df -a

Check the available space of the vg-root directory. This seems to be causing the issue if the disk is 80% or higher capacity.

expedition@Expedition:~$ df -a

Filesystem 1K-blocks Used Available Use% Mounted on

sysfs 0 0 0 - /sys

proc 0 0 0 - /proc

udev 3002164 0 3002164 0% /dev

devpts 0 0 0 - /dev/pts

tmpfs 604736 8400 596336 2% /run

/dev/mapper/Expedition--vg-root 18502968 6977800 10562200 40% /

securityfs 0 0 0 - /sys/kernel/security

tmpfs 3023672 0 3023672 0% /dev/shm

tmpfs 5120 0 5120 0% /run/lock

tmpfs 3023672 0 3023672 0% /sys/fs/cgroup

cgroup 0 0 0 - /sys/fs/cgroup/systemd

pstore 0 0 0 - /sys/fs/pstore

cgroup 0 0 0 - /sys/fs/cgroup/blkio

cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct

cgroup 0 0 0 - /sys/fs/cgroup/devices

cgroup 0 0 0 - /sys/fs/cgroup/freezer

cgroup 0 0 0 - /sys/fs/cgroup/memory

cgroup 0 0 0 - /sys/fs/cgroup/net_cls,net_prio

cgroup 0 0 0 - /sys/fs/cgroup/perf_event

cgroup 0 0 0 - /sys/fs/cgroup/cpuset

cgroup 0 0 0 - /sys/fs/cgroup/pids

cgroup 0 0 0 - /sys/fs/cgroup/hugetlb

mqueue 0 0 0 - /dev/mqueue

hugetlbfs 0 0 0 - /dev/hugepages

systemd-1 0 0 0 - /proc/sys/fs/binfmt_misc

debugfs 0 0 0 - /sys/kernel/debug

fusectl 0 0 0 - /sys/fs/fuse/connections

/dev/sda1 482922 111167 346821 25% /boot

lxcfs 0 0 0 - /var/lib/lxcfs

tmpfs 604736 0 604736 0% /run/user/1000

Re: New feature ver 1.1.21 - Scheduled Log Processing

Laimonas — Fri, 31 May 2019 05:36:45 GMT

I've succeeded by checking the box "AutoProcess CSV log files".

I think guide forgot to mention this.

Re: New feature ver 1.1.21 - Scheduled Log Processing

Poulsen — Mon, 03 Jun 2019 01:23:23 GMT

For the UI refresh issue, I have plenty of disk space available... It didn't happen on v1.1.19, I only saw it in v1.1.21. But in v1.1.23 it seems to be no longer an issue, so I am happy again.

sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
udev 750684 0 750684 0% /dev
devpts 0 0 0 - /dev/pts
tmpfs 154440 4720 149720 4% /run
/dev/mapper/Expedition--vg-root 18502968 5114168 12425832 30% /
securityfs 0 0 0 - /sys/kernel/security
tmpfs 772192 0 772192 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 772192 0 772192 0% /sys/fs/cgroup
cgroup 0 0 0 - /sys/fs/cgroup/systemd
pstore 0 0 0 - /sys/fs/pstore
cgroup 0 0 0 - /sys/fs/cgroup/net_cls,net_prio
cgroup 0 0 0 - /sys/fs/cgroup/cpuset
cgroup 0 0 0 - /sys/fs/cgroup/blkio
cgroup 0 0 0 - /sys/fs/cgroup/hugetlb
cgroup 0 0 0 - /sys/fs/cgroup/pids
cgroup 0 0 0 - /sys/fs/cgroup/devices
cgroup 0 0 0 - /sys/fs/cgroup/memory
cgroup 0 0 0 - /sys/fs/cgroup/perf_event
cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct
cgroup 0 0 0 - /sys/fs/cgroup/freezer
systemd-1 0 0 0 - /proc/sys/fs/binfmt_misc
hugetlbfs 0 0 0 - /dev/hugepages
debugfs 0 0 0 - /sys/kernel/debug
mqueue 0 0 0 - /dev/mqueue
fusectl 0 0 0 - /sys/fs/fuse/connections
/dev/sda1 482922 160650 297338 36% /boot
lxcfs 0 0 0 - /var/lib/lxcfs
tmpfs 154440 0 154440 0% /run/user/1000