topic Re: Log Processing not working in Expedition Discussions

Log Processing not working

ehelm — Fri, 01 Nov 2019 12:45:27 GMT

Greetings! My instance of Expedition (1.1.43) has stopped processing all logs. The devices detail shows the new logs. I've clicked process logs from there and it appears a job is added but the job includes zero tasks and never completes or fails. I've tried clicking the little green "play" button and get "Error processing log files". /tmp/PAN_LogCoCo.log is empty. Any suggestions of things I can look at or do to kick start this log processing issue? Thanks!

Re: Log Processing not working

GNeyrinck — Fri, 01 Nov 2019 15:01:43 GMT

I have the exact same issue after the upgrade from 1.41 to this version.

I saw some errors in cli while upgrading.

Did you also update Expedition?

Regards

Re: Log Processing not working

dgildelaig — Mon, 04 Nov 2019 09:41:55 GMT

We are working on an issue related to CSV files processing.

I expect to release an update later today or tomorrow when this issue has been addressed.

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 10:56:54 GMT

I have now upgraded to the latest version 1.1.45 and it seems that the logs are being processed.

Even though I've only selected one log file to be processed, all pending (20) files seem to be processed in one time.

I'll check if the processing of logs ends ok now.

Regards,

Re: Log Processing not working

dgildelaig — Tue, 05 Nov 2019 11:04:06 GMT

When you say that "only selected one file" are you referring to having the other files set as Ignored?

If so, I will open a bug ticket to check the issue.

Otherwise, remember that the way to state which files are going to be processed is by defining which files are ignored. By default, all files are set to be processed.

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 11:11:19 GMT

Edit: ok, all files are going to be processed except the ones you set to ignore.

I meant to say that I'm only selecting one file to test if the processing would work again.

Since it's taking that long I'm assuming that all logs are being processed:

root@Expedition:/home/expedition# tail -f /tmp/error_logCoCo

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]

---- CREATING SPARK Session:

warehouseLocation:/datastore/spark-warehouse

+------------+--------+--------------------+----+------------+

+------------+--------+--------------------+----+------------+

|011901016881| 8.1.0|/PALogs/paloalto_...|4957| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4332| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4629| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|3861| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4629| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|3840| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|5264| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|3892| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|3646| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4783| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4772| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4619| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|5161| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4752| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4076| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4895| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|3789| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|4742| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|5059| Compress|

|011901016881| 8.1.0|/PALogs/paloalto_...|5407| Compress|

+------------+--------+--------------------+----+------------+

Memory: 6373m

LogCollector&Compacter called with the following parameters:

Parameters for execution

Master[processes]:............ local[3]

Available RAM (MB):........... 6525952

User:......................... admin

debug:........................ false

Parameters for Job Connections

Task ID:...................... 152

My IP:........................ 10.3.1.30

Expedition IP:................ 10.3.1.30:3306

Time Zone:.................... Europe/Helsinki

dbUser (dbPassword):.......... root (************)

projectName:.................. demo

Parameters for Data Sources

App Categories (source):........ (Expedition)

CSV Files Path:................./tmp/1572951103_traffic_files.csv

Parquet output path:.......... file:///PALogs/connections.parquet

Temporary folder:............. /datastore

---- AppID DB LOAD:

Application Categories loading...

Application Categories loaded

+------------+--------+--------------------+----+------------+--------+---+---------------+

+------------+--------+--------------------+----+------------+--------+---+---------------+

|011901016881| 8.1.0|/PALogs/paloalto_...|4957| Compress|grouping| 1| 4957.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4332| Compress|grouping| 2| 9289.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4629| Compress|grouping| 3| 13918.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|3861| Compress|grouping| 4| 17779.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4629| Compress|grouping| 5| 22408.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|3840| Compress|grouping| 6| 26248.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|5264| Compress|grouping| 7| 31512.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|3892| Compress|grouping| 8| 35404.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|3646| Compress|grouping| 9| 39050.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4783| Compress|grouping| 10| 43833.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4772| Compress|grouping| 11| 48605.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4619| Compress|grouping| 12| 53224.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|5161| Compress|grouping| 13| 58385.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4752| Compress|grouping| 14| 63137.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4076| Compress|grouping| 15| 67213.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4895| Compress|grouping| 16| 72108.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|3789| Compress|grouping| 17| 75897.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|4742| Compress|grouping| 18| 80639.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|5059| Compress|grouping| 19| 85698.0|

|011901016881| 8.1.0|/PALogs/paloalto_...|5407| Compress|grouping| 20| 91105.0|

+------------+--------+--------------------+----+------------+--------+---+---------------+

Selection criteria: 0 < accumulatedSize and accumulatedSize <= 6525952

Processing from lowLimit:0 to highLimit:6525952 with StepLine:6525952

Few logs can fit in this batch:20

8.1.0:/PALogs/paloalto_1_traffic_2019_10_25_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_01_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_02_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_31_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_19_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_17_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_03_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_28_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_05_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_27_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_30_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_22_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_04_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_18_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_26_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_21_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_29_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_20_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_23_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_24_last_calendar_day.csv

Logs of format 7.1.x NOT found

Logs of format 8.0.2 NOT found

Logs of format 8.1.0-beta17 NOT found

Logs of format 8.1.0 found

Logs of format 9.0.0 NOT found

Logs of format 9.1.0-beta NOT found

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 11:09:54 GMT

Another thing which isn't ok is the machine learning auto processing

It should be hard coded, but it's showing as empty in the gui

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 11:20:25 GMT

I meant to say that I've only select one file.

But you explained that all will be processed except the ones that you set to ignore.

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 12:17:34 GMT

All files have been processed now.

The only thing I need to verify now is the automatic processing of the logs.

Re: Log Processing not working

dgildelaig — Tue, 05 Nov 2019 12:25:22 GMT

I am modifying the interface to avoid this confusion. You are not the first one, neither the second or the tenth that though only the check-boxed would be processed.

Re: Log Processing not working

ehelm — Tue, 05 Nov 2019 14:21:54 GMT

Unfortunately I'm not seeing the logs processed. I do not get an error now, and it appears that logs process as there are several java processes that spin up when I click the "process files" button. The jobs show failed but no clue as to why. Also a link to the error_logCoCo is included below. I haven't seen anything in there that indicates where the problem is. Any thoughts? Thanks!

error_logCoCo

Re: Log Processing not working

ehelm — Tue, 05 Nov 2019 21:10:28 GMT

UPDATE: I noticed a low disk space error in the log. Took care of that (Moved location of DATA ANALYSIS STRUCTURES FOLDERs). Re-processed the files and things worked as expected.

Re: Log Processing not working

GNeyrinck — Tue, 05 Nov 2019 21:19:20 GMT

I encountered this error also, due to the longer time the logs weren't processed the foldersize got bigger.

I've added some additional space and relaunched the processing and all went ok.

I'm still seeing the (reduced) logfiles even if you select the "delete logs after processing"